JP2021026406A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

To appropriately evaluate input data even when a classifier has not sufficiently learned characteristics for classifying the input data.SOLUTION: An information processing apparatus according to the present invention comprises: a likelihood acquisition unit that acquires the class likelihood for medical data added with a correct label by using a first classifier that classifies the medical data by class; a classification result evaluation unit that evaluates the degree of deviation based on the class likelihood acquired by the likelihood acquisition unit and a class corresponding to the correct label; a determination unit that determines whether the degree of deviation estimated by the classification result evaluation unit satisfies a predetermined standard; and a learning unit for a classifier that performs learning of a second classifier with medical data determined to satisfy the predetermined standard by the determination unit as training data.SELECTED DRAWING: Figure 6

Description

The present invention relates to an information processing device, an information processing method, and a program for learning a classifier different from the classifier based on the classification result of the classifier for medical data to which a correct answer label is given.

Computer Aided Diagnosis (CAD) systems are known that analyze medical images and present information to the doctor to assist in interpretation. In the CAD system that classifies and presents diagnostic names that are candidates for differential diagnosis from medical images, the classifier is made to perform machine learning using teacher data that pairs medical data and correct diagnostic names (correct answer labels). There is something that will be realized.

The CAD system disclosed in Patent Document 1 correlates a support result such as lesion detection by an abnormality detection processing system such as an abnormality shadow based on machine learning with a correction result after the support result is corrected by a doctor. Save and quantitatively evaluate the performance of support processing.

Japanese Patent No. 41004036

In the technique of Patent Document 1, the performance of a classifier can be evaluated based on the correction information for the classification result of a single classifier. On the other hand, it is not disclosed to learn a classifier different from the classifier based on the classification result of the classifier for the medical data to which the correct answer label is given.

The information processing device according to the present invention has the following configurations. That is,
The likelihood acquisition unit that acquires the class likelihood for the medical data with the correct answer label using the first classifier that classifies the medical data, the class likelihood acquired by the likelihood acquisition unit, and the correct answer label. A classification result evaluation unit that evaluates the degree of divergence based on the class corresponding to, a judgment unit that determines whether the degree of divergence by the classification result evaluation unit meets a predetermined criterion, and
It is provided with a learning unit of a classifier that learns a second classifier using medical data determined by the determination unit as teacher data to satisfy a predetermined criterion.

According to the present invention, it is possible to learn a classifier different from the classifier based on the classification result of the classifier for the medical data to which the correct answer label is given.

System configuration diagram of an information processing system including the information processing devices of the first to fourth embodiments Hardware configuration diagram of the information processing apparatus of the first to fourth embodiments A conceptual diagram showing the configuration of the medical image DB of the first to fourth embodiments. Flow chart of information processing device classifier creation process Medical data evaluation flow chart for classification of information processing equipment Functional block diagram of the information processing apparatus of the first embodiment Example of display screen of the information processing apparatus of the first embodiment Flow chart of processing of information processing apparatus of Embodiment 1 Example of display screen of the information processing apparatus of the first embodiment Functional block diagram of the information processing apparatus of the second embodiment Example of display screen of the information processing apparatus of the second embodiment Flow chart of processing of information processing apparatus of Embodiment 2 Classification data diagram of embodiment 3 Example of display screen of the information processing apparatus of the third embodiment Example of display screen of the information processing apparatus of the third embodiment Functional block diagram of the information treatment device of the fourth embodiment Flow chart of processing of information processing apparatus of Embodiment 4

Hereinafter, the details of the invention will be described with reference to the accompanying drawings, based on the embodiments of the present invention. Unless otherwise specified, the items described in other embodiments and the like shall be assigned the same number and the description thereof shall be omitted. Further, the configuration shown in the following embodiments is only an example, and the present invention is not limited to the illustrated configuration.

<Embodiment 1>
In the first embodiment, an information processing device which is a CAD system for classifying diagnostic names for lung nodule shadows on a chest X-ray CT (Computed Tomography) image will be described. The information processing device of the present embodiment evaluates the degree of deviation between the likelihood of classification by the classifier and the correct answer, and evaluates the similarity between the data determined based on the degree of deviation and the data to be processed. And notify the user of the result. The user can choose whether or not to continue the process based on the notification.

(System configuration)
FIG. 1 is a system configuration diagram of an information processing system including the information processing device of the present embodiment.

In FIG. 1, the information processing system is composed of a medical image database (hereinafter referred to as a medical image DB) 102, an information processing device 101, and a LAN (Local Area Network) 103.

The medical image DB 102 stores medical data including a medical image taken by a medical image imaging device such as a CT device and a diagnosis name of the medical image. It also provides a known database function for retrieving and retrieving medical data via LAN 103. The structure of the medical data stored in the medical image DB 102 will be described with reference to FIG.

(Hardware configuration)
FIG. 2 is a hardware configuration diagram of the information processing device 101 of the present embodiment.

In FIG. 2, the storage medium 201 is a storage medium such as an OS (Operating System), a processing program for performing various processes according to the present embodiment, and an HDD (Hard Disk Drive) for storing various information. The ROM (Read Only Memory) 202 stores a program for initializing the hardware and starting the OS, such as a BIOS (Basic Input Output System). The CPU (Central Processing Unit) 203 performs arithmetic processing when executing a BIOS, an OS, and a processing program. The RAM (Random Access Memory) 204 temporarily stores information when the CPU 203 executes a program. The LAN interface 205 corresponds to a standard such as IEEE (Institute of Electrical and Electronics Engineers) 802.3ab, and is an interface for communicating via the LAN 103. 207 is a display for displaying a display screen, and 206 is a display interface for converting screen information to be displayed on the display 207 into a signal and outputting the signal. Reference numeral 209 is a keyboard for key input, 210 is a mouse for designating coordinate positions on the screen and inputting button operations, and 208 is an input interface for receiving signals from the keyboard 209 and the mouse 210. Reference numeral 211 is an internal bus for each block to communicate.

(Composition of medical data)
FIG. 3 is a conceptual diagram showing the structure of medical data stored in the medical image DB 102.

In FIG. 3, the medical data stored in the medical image DB 102 is composed of a first medical data set 310 and a medical data set 320 to be classified. The first medical data set 310 is a medical data set used for verifying the classifier. The first medical data set 310 includes a plurality of first medical data 311-j (j = 1, ..., N1), and is composed of the first medical data 311-j (j = 1, ..., N1). , N1) are composed of information such as patient information 301, diagnosis name 302, and image 303, respectively. Here, the patient information 301 is information about the patient such as a patient ID, a patient's name, age, and gender. The diagnosis name 302 is a diagnosis name related to the image 303, and for example, in the present embodiment, there are three types of “primary”, “metastasis”, and “benign”. Here, "primary" refers to primary lung cancer, "metastasis" refers to metastatic lung cancer, and "benign" refers to a benign nodule. Image 303 is a three-dimensional partial region image including a lung nodule extracted from a CT image. The medical data set 320 to be classified is a medical data set that is classified by the information processing device 101. The medical data set 320 to be classified includes a plurality of medical data 321-j (j = 1, ..., N3) to be classified, and the medical data 321-j (j = 1, ..., N3) to be classified is included. , N3) is composed of patient information 301 and series image 304. Here, the series image 304 is an image obtained by a CT apparatus in one shooting, and is composed of an image having a plurality of cross sections (hereinafter, referred to as a slice image).

The diagnosis name 302 may be "malignant" or "benign", or may be a diagnosis name in which the primary, metastasis, and benign are further subdivided. Further, the image 303 may be a combination of a CT image and coordinate information indicating a three-dimensional partial region including a lung nodule. The first medical data 311-i and the third medical data 321-i may include information other than those described above.

4 and 5 are diagrams simply showing a configuration in which the information processing apparatus 101 evaluates the data (input data) to be classified and notifies the data. Here, first, using FIG. 4, a configuration for creating a classification data set for evaluating medical data to be classified based on the first medical data set 310 will be described. Next, a configuration (FIG. 5) for evaluating the medical data set to be classified based on the classification data set created in FIG. 5 will be described.

FIG. 4 shows a flow for creating a classification data set and a classifier for evaluating medical data to be classified. In generating the classification data set, it is assumed here that 1 is set as the start condition. Here, it is assumed that at least one classifier has been trained and exists. Alternatively, if there is no trained classifier, a classifier that trains the first medical data set is provided. The flow will be described under such preconditions.

First, the i-th medical data acquisition unit 401 acquires the i-th medical data from the medical DB 102. The i-th data is the first medical data set when i = 1. That is, the first medical data set 310 is acquired by the first medical data acquisition unit 401. The medical data acquired by the i-th medical data acquisition unit 401 is transmitted to the i-th classifier 402, and the classification of the diagnosis name is calculated by the i- (first) classifier with the likelihood. A description of the classifier will be given later.

Next, after receiving the classification result by the third classifier 402, the evaluation unit 403 of the classification result evaluates the degree of deviation between the classification result and the correct answer. The evaluation method will also be described later. It is determined whether or not the deviation from the correct answer is equal to or greater than a predetermined standard, and the i + 1 medical data is generated from the i-th medical data. In addition, the data whose dissociation does not satisfy a predetermined criterion is stored in the i-th classifier as the i-th classification data. The classification data is, in other words, a data group that can be accurately classified for the classifier in the third classifier, in which the degree of deviation between the classification and the correct answer is smaller than the predetermined standard (does not meet the predetermined standard).

On the other hand, the medical data of the i + 1 which is the medical data whose degree of dissociation is larger than the predetermined standard (satisfies the predetermined standard) is transmitted to the learning unit 404 of the classifier of the i + 1. Based on the medical data of the i + 1, the learning unit of the i + 1 performs learning with the diagnosis name 302 corresponding to the data of the i + 1 as a class (label). Then, i + 1 is substituted for i and the flow from 401 is executed again. With this configuration, data whose degree of divergence is greater than or equal to a predetermined standard is recursively evaluated, a classification data set is created, and the created classification data set is compared with the input data to explain the classification described in FIG. Evaluation is possible for the target data (input data).

Although the end condition is not specified in this flow, for example, the process may be terminated when the number of training data is below a certain level, when the accuracy is below a certain level, or for the model structure. This may be used as an end condition for ending this flow when it is determined that the learning data is insufficient. Further, the case where over-learning or unlearning is determined may be set as the end condition, or even if the execution is performed only a predetermined number of times determined by the user, the end condition is set according to the number of data in the medical data set and the distribution of the data. May be good.

Next, with reference to FIG. 5, the flow performed by the information processing apparatus 101 when the medical data set 320 to be classified is input will be described. First, the classification target medical data acquisition unit 501 acquires the classification target medical data set 320 from the medical DB 102. Then, the acquired medical data 320 to be classified is transmitted to the medical data evaluation unit 502 to be classified. The classification target medical data evaluation unit 502 evaluates the similarity between the input classification target medical data set 320 and the classification data set. That is, it is evaluated whether or not the similarity of the medical data set 320 to be classified is equal to or higher than a certain level with respect to the classification data set which is a set of data that can be classified accurately corresponding to each of the classifiers. Then, the evaluation result is transmitted to the notification unit 503, and the notification unit 503 notifies based on the evaluation result received. The method of evaluating the similarity with the classification data of the classifier will also be described later.

Hereinafter, a functional block diagram (FIG. 6) showing a function for carrying out the flow of FIGS. 4 and 5 will be used.

(Functional block)
FIG. 6 is a functional block diagram of the information processing device 101 of the present embodiment.

In FIG. 6, in addition to the functional blocks shown in FIGS. 4 and 5, the information processing apparatus 101 has a likelihood acquisition unit 601 that acquires the likelihood as a classification result by the classifier, and the degree of deviation exceeds a predetermined standard. Judgment unit 602 for determining medical data, medical data set 603 of the i + 1 which is data determined to exceed a predetermined standard by the determination unit 602, and medical data determined not to exceed a predetermined standard by the determination unit 602. It is composed of a classification data set 620 in the medical image DB 102 that is stored as classification data in the i-th classifier. The functions of each part will be described below.

Here, a flow for creating a classification data set 620 for evaluating the medical data set 320 to be classified according to the flow of FIGS. 4 and 5 and a medical data set for classification based on the created classification data set 620. The flow of evaluating the data set 320 will be described separately.

(Flow for creating classification data set 620)
The third medical data acquisition unit 401 acquires medical data from the medical image DB 102. In the case of i = 1, for example, the first medical data set 310 is acquired. Then, the acquired medical data set is transmitted to the likelihood acquisition unit 601.

In the case of i = 1, the first classifier 402 (first classifier) has a diagnosis name (class) when, for example, a partial region image of the lung nodule in the image 303 constituting the first medical data is input. ). The i-th classifier 402 (first classifier) outputs the class of the input image as the classification result of the diagnosis name with the likelihood. That is, as the classification result by the classifier 404, the likelihood of being "primary", the likelihood of being "transition", and the likelihood of being "benign" are output. Specifically, the classifier 401 is a CNN (Convolutional Neural Network) machine-learned using the i-th medical data set 310.

The likelihood acquisition unit 601 inputs the image 303 constituting the first medical data set 310 acquired from the i-th medical data acquisition unit into the i-th classifier 402 (first classifier), and the i-th The likelihood of classification, which is the classification result by the classifier 402 of the above, is acquired. Specifically, the classification result by the third classifier 402 is the output from the final layer of CNN, and the output value of the three nodes corresponding to "nuclear power", "transition", and "benign" is calculated as Softmax. Get the value given by.

When i = 1, the evaluation unit 403 of the classification result has the first medical data 311-j (j = 1, ...,) in the first medical data set 310 acquired by the likelihood acquisition unit 601. Evaluate the degree of divergence between the likelihood of classification into classes and the correct answer for N1). For example, the difference between the likelihood of the correct answer class, which is the class corresponding to the correct answer label, and a predetermined value is compared. Specifically, the evaluation unit 403 of the classification result calculates the absolute value of the difference between the likelihood to the correct answer class and 1.0 among the classification results of the classifier 401 for the class corresponding to the correct answer. Then, for the classification other than the correct answer class, the first evaluation unit 403 determines the absolute value of the difference between the likelihood of the class having the highest likelihood and 0.0 among the classes excluding the likelihood to the correct answer class. Calculate and use the sum of the likelihood difference to the correct answer class and the likelihood difference to other than the correct answer class as the evaluation value. For example, the correct answer of the class when the diagnosis name is "primary" is written as (1.0, 0.0, 0.0), and the likelihood of "primary" in the classification result is 0.8, " It is assumed that the likelihood of "transition" is 0.2 and the likelihood of "beneficial" is 0.0 (0.8, 0.2, 0.0). When the determination unit 602 evaluates the difference between the likelihood (0.8, 0.2, 0.0) and the correct answer (1.0, 0.0, 0.0) based on the classification result, | 0.8 -1.0 | + | 0.2-0.0 | = 0.4. Similarly, when the likelihood is (0.8, 0.1, 0.1), the divergence is 0.3. The likelihood (0.6, 0.4, 0.0) is 0.8, and the likelihood (0.6, 0.2, 0.2) is 0.6. Therefore, even if the classification result is the same as the correct answer for "nuclear power plant", the likelihood for "nuclear power plant" is low, and if the highest likelihood other than "nuclear power plant" is high, the divergence becomes large. Similarly, the likelihood (0.3, 0.7, 0.0) is 1.4, the likelihood (0.3, 0.35, 0.35) is 1.05, and the likelihood ( In the case of 0.1, 0.9, 0.0), it is 1.8, and in the case of likelihood (0.1, 0.45, 0.45), it is 1.35. Therefore, even if the classification result is different from the correct answer "nuclear power plant", the likelihood for "nuclear power plant" is high, and the deviation is small when the highest likelihood other than "nuclear power plant" is low. That is, the divergence evaluated in this embodiment is different from the mere correct answer rate and the likelihood of classifying as a correct answer, and is a total of the degree of correct answer and the degree of incorrect answer. The evaluation value by the evaluation unit 403 of the classification result is not limited to this embodiment. For example, the absolute value of the difference between the correct answer class by the i-th classifier 402 and 1.0 is a class other than the correct answer class. The value obtained by adding the difference between the likelihood of and 0.0 may be calculated as the evaluation value. That is, the evaluation value calculated by the evaluation unit 403 of the classification result does not matter in any form as long as the degree of discrepancy between the correct answer and the classification result by the third classifier 402 can be evaluated.

Based on the evaluation result of the evaluation unit 403 of the classification result, the determination unit 602 sets the first medical data 311-j (i = 1, ...,) of the first medical data set 310 (when i = 1). From N1), it is determined whether or not the predetermined criteria are satisfied, and the medical data set 603 of the i + 1 which is the medical data satisfying the predetermined criteria is acquired. In the case of i = 1, the medical data set determined to meet the predetermined criteria is the second medical data set. Here, the predetermined reference is a fixed value defined in advance, and for example, the value of the deviation is 1.2 or more. In this case, even if the correct answer is "primary" and the classification result is different, the likelihood is (0.3, 0.35, 0.35) or (0.25, 0.375, 0.375). Since the deviations are 1.05 and 1.125, respectively, the data is not determined to be the second medical data set 603, which is a data set satisfying a predetermined criterion. On the other hand, in the case of data with a likelihood of (0.3, 0.5, 0.2), the deviation is 1.2, and in the case of data with a likelihood of (0.25, 0.45, 0.3). Since the deviation is 1.2, the data is the data of the second medical data set 603, which is a medical data set satisfying a predetermined standard. A setting unit (not shown) for setting a predetermined reference may be separately provided, or the user may accept input of a predetermined reference value via a GUI or the like. Further, the data determined by the determination unit 602 that the predetermined criteria are not satisfied based on the evaluation result by the evaluation unit 403 of the classification result is used as the classification data 620 in the i-th classifier and corresponds to the storage unit (medical image DB 102). Attach and memorize. The medical data determined not to satisfy the predetermined criteria is, for example, data having a dissociation value of less than 1.2. The data in which the deviation between the classification result by the classifier and the correct answer is less than the predetermined standard by the determination unit 403 refers to the medical data determined not to satisfy the predetermined standard. The determination unit 403 determines whether or not the predetermined criteria are satisfied. The predetermined standard is, for example, a threshold value. Meet the predetermined standard here means medical data in which the degree of dissociation exceeds the threshold value, and here means that the degree of dissociation does not exceed the threshold value for medical data. Refers to data.

That is, it is characterized in that the medical data determined by the determination unit 403 not to satisfy the predetermined criteria is used as the classification data corresponding to the classifier that classifies the medical data determined not to satisfy the predetermined criteria. Here, the classification data and the classifier that classifies the medical data constituting the classification data are associated and stored in the medical image DB.

The learning unit 404 of the i + 1 classifier (the second classifier in the case of i = 1) uses the teacher data of the diagnosis name paired with the medical data set 603 of the i + 1 determined by the determination unit 602. The i + 1 classifier is learned. Similarly, the i + 1 classifier has a configuration in which the diagnosis name is calculated by the likelihood when the image to be classified is input. That is, the information processing device 101 uses the third classifier 402 for classifying the medical data, and the likelihood acquisition unit 601 for acquiring the class likelihood corresponding to the medical data to which the correct answer label is given, and the likelihood acquisition unit 601 and the likelihood. It has a classification result evaluation unit 403 that evaluates the degree of deviation based on the class likelihood acquired by the acquisition unit 601 and the class corresponding to the correct answer label. Further, the determination unit 602 that determines whether or not the deviation of the classification result by the evaluation unit 403 satisfies the predetermined criteria, and the i + 1 that uses the medical data determined by the determination unit 602 to satisfy the predetermined criteria as teacher data. It is characterized by having a learning unit of the i + 1 classifier for learning the classifier.

When the flow up to this point is completed, if the end determination is not made due to the above-mentioned end conditions or the like, the above flow is repeated by substituting i + 1 for i. That is, the information processing device 101 uses the i + 1 classifier learned from the teacher data as a classifier for classifying the medical data (the i-th classifier 402), and the medical data determined to satisfy a predetermined criterion. It is characterized by having a control unit (CPU 203) capable of repeatedly executing the processes of the likelihood acquisition unit 601, the classification result evaluation unit 403, the determination unit 602, and the learning unit 404. By the iterative processing, the information processing apparatus 101 stores the plurality of classifiers and the classification data corresponding to each of the plurality of classifiers in the storage unit (medical DB 102).

The repetition of the flow of creating the classifier and the flow of creating the classification data may be defined by any of the above-mentioned termination conditions. For example, the training data for learning the classifier may decrease due to the repetition of the flow. Since the decrease in teacher data causes a decrease in the accuracy of the classifier, when the number of teacher data is less than a predetermined number with respect to the model structure of the classifier, etc., it is used as the end condition of the repeating flow. Alternatively, it may be terminated when the classification accuracy of the classifier becomes less than a predetermined value. Alternatively, it may be used as a repeat termination condition when overfitting or unlearning is determined, which is caused by uneven distribution of teacher data in the classifier, number, and number of learnings. Of course, the flow may be executed repeatedly only the number of times specified by the user. That is, the information processing device 101 determines that the number of teacher data for learning the classifier is less than or equal to the predetermined number, determines that the classification accuracy of the classifier is less than or equal to the predetermined value, determines overfitting, determines unlearned, and exceeds the number of times specified by the user. It is characterized in that the repetition is terminated when any of the determination processes of the determination is performed.

Processing when the medical data set 320 to be classified is input based on the plurality of classifiers constituting the information processing device 101 created from the following and the classification data set 620 corresponding to each of the plurality of classifiers. Will be described.

(Flow for evaluating the medical data set 320 to be classified)
The medical data acquisition unit 501 to be classified acquires the medical data set 320 to be classified from the medical image DB 102. The classification target medical data acquisition unit 501 transmits the acquired classification target medical data set 320 to the classification target medical data evaluation unit 502.

The classification target medical data evaluation unit 502 evaluates the similarity between the classification data set 620 and the partial region image of the lung nodule extracted from the series image 304 of the classification target medical data set 320. That is, the medical data evaluation unit 502 to be classified is characterized in that it evaluates the similarity between the classification data 620 and the medical data set 320 to be classified.

The partial region image of the lung nodule is extracted based on the operation on the display screen described with reference to FIG. Similarity is obtained by assigning a classifier associated with the classification data as a class (label) to the classification data created in the above classification data creation flow, and using a machine-learned classifier (classifier for classifying medical data to be classified). ) To evaluate. A classifier that classifies medical data to be classified is, for example, CNN. The likelihood output from the CNN is defined as the similarity (hereinafter referred to as the similarity). That is, the information processing device 101 uses a classifier that has learned the classification data with each of the plurality of classifiers as labels as teacher data, and evaluates the medical data to be classified by the medical data evaluation unit 502 to be classified. Have. Further, the medical data evaluation unit 502 to be classified calculates the classification result by the likelihood.

The notification unit 503 notifies the information based on the evaluation result by the medical data evaluation unit 502 to be classified. Specifically, the degree of similarity with the classification data corresponding to each classifier is displayed on the display screen. The display screen will be described with reference to FIG.

(Display screen)
FIG. 7 is a diagram showing an example of a display screen of the information processing device 101 of the present embodiment.

In FIG. 7, the display screen 700 is a user interface screen displayed on the display 207. The display screen 700 is composed of a patient information display area 701, an image display area 702, and a diagnosis support button 704. Further, in FIG. 7, 703 is a lung nodule partial region, and notification region 705 is a notification region displayed by notification unit 503.

In the patient information display area 701, the information of the patient information 301 of the medical data 321-j (j = 1, ..., N3) to be classified is displayed. FIG. 7 shows a display example when the patient name is “Taro Shimomaruko”, the patient ID is “pat0123456”, the age is “75 years old”, and the gender is “male”.

In the image display area 702, a series image 304 of medical data 321-j (j = 1, ..., N3) to be classified is displayed. In the image display area 702, the display can be changed such as slicing the series image to be displayed and changing the gradation display condition called WL (Window Level) / WW (Window Width).

The designation of the lung nodule partial region 703 is performed, for example, by the user dragging the mouse in the image display region 702, and is released by clicking the mouse. The nodule portion region 703 is displayed on the slice image in accordance with the drag, and a three-dimensional region (cube) having the same depth is designated with the slice image being displayed as the center. The specification of the partial area is not limited to the one specified only by the operation of the user. For example, the partial area may be specified by other image processing means, or the partial area is extracted from the image area. It may be specified by a machine learning-based model designed as such.

The diagnosis support button 704 is a button for classifying the diagnosis name from the image of the lung nodule partial region 703. When the diagnosis support button 704 is clicked with the mouse, the information processing apparatus 101 extracts the image of the lung nodule partial region 703. Then, the diagnosis name is classified from the extracted image.

The notification area 705 displays information based on the similarity between the image of the lung nodule partial area 703 and the classification data set 620 associated with the classifier. Specifically, it is a window that is displayed in a pop-up, and includes a button for displaying the similarity and specifying whether to execute or cancel the process.

(Processing flow)
FIG. 8 is a processing flow chart of the information processing apparatus 101 of the present embodiment.

This process is executed based on an instruction from the user after the information processing device 101 is started. When instructing the execution of the process, the user specifies the medical data 321-j (j = 1, ..., N3) to be classified as the target of the process.

In step S801, the third medical data acquisition unit 401 reads out the i-th medical data set 310 from the medical image DB 102.

The likelihood acquisition unit 601 inputs the image 303 constituting the third medical data set 310 read in step S601 into the i-th classifier 402, and classifies the class (diagnosis name) as the classification result from the i-th classifier 402. ) To get the likelihood of classification.

In step S803, the evaluation unit 403 of the classification result determines the likelihood of classification into the class based on the likelihood of classification to the class acquired by the likelihood acquisition unit 601 and the diagnosis name 302 of the third medical data set 310. Evaluate the degree of divergence between correct answers.

In step S804, the determination unit 404 determines whether the degree of deviation evaluated by the first evaluation unit 403 satisfies a predetermined criterion, and determines whether the degree of deviation is satisfied with the predetermined criteria, and determines whether the degree of deviation is satisfied with the predetermined criteria, and the first medical data 311-j (j) of the first medical data set 310 It is determined whether or not = 1, ..., N1) is the data of the medical data set 603 of the i + 1. If the degree of dissociation exceeds a predetermined standard, it is assumed to be the medical data set 603 of the i + 1, and if it does not exceed the predetermined standard, it is stored in the medical image DB as the classification data set 620 in the i-th classifier 402. Remember. The degree of deviation between the correct label and the classification result exceeds a predetermined standard, here means a case where the degree of deviation between the two exceeds a predetermined threshold. Further, "not exceeding a predetermined standard" here means a case where the degree of deviation between the correct label and the classification result does not exceed the predetermined threshold value.

In step S805, the learning unit 404 of the i + 1 classifier performs machine learning of the i + 1 classifier with the diagnosis name 302 as the correct label (class). That is, a first i + 1 classifier that classifies the i + 1 medical data as learning data and the diagnosis name as a class is generated.

In step S806, i + 1 is assigned to i to update the value of i.

Step S807 determines the end of classifier generation. In addition to the above, the end determination may be completed when, for example, the number of training data falls below a certain level, or when the accuracy falls below a certain level, or when the training data is insufficient for the model structure. It may be a condition to end this flow when it is done. Further, the case where over-learning or unlearning is determined may be set as the end condition, or even if the execution is performed only a predetermined number of times determined by the user, the end condition is set according to the number of data in the medical data set and the distribution of the data. May be good. If the end condition is not satisfied, the learning flow of the classifier is repeated again from step S801. When the end condition is satisfied, the process proceeds to the next step S807. Up to this point corresponds to the classification data creation flow, which is the process of FIG. 4 described above. The steps corresponding to the flow (FIG. 5) for evaluating the medical data set 320 to be classified created from here will be described. The flow from step S801 to step S807 is a flow for learning and creating a classifier, and if there is a different information processing device or a classifier that has already been learned in the same information processing device, this flow is It may be omitted.

In step S808, the classification target medical data acquisition unit 501 acquires the classification target medical data set 320 from the medical image DB 102 and reads out the classification target data. The user interface control unit (CPU 203) reads out the medical data 321-j (j = 1, ..., N3) to be classified specified at the time of executing this process in step S808, and in step S809, FIG. 7 A display screen 700 showing an example is displayed. Further, in step S810, an image of the lung nodule partial region 703 designated based on the operation by the user is extracted.

In step S811, the medical data evaluation unit 502 to be classified evaluates the similarity between the image extracted in step S810 and the image of the classification data set 620. The medical data evaluation unit 502 to be classified assigns a classifier associated with the classification data as a class (label), and based on the classification result by the machine-learned classifier (classifier for classifying the medical data to be classified). Evaluate.

In step S812, the notification unit 503 displays the notification area 705 on the display screen 700 based on the evaluation result in step S811. The notification area 705 may be displayed by the notification unit 503 only when the evaluation result in step S811 exceeds a predetermined value.

As described above, according to the present embodiment, the information processing apparatus 101 evaluates the degree of deviation between the likelihood of classification by the classification device i and the correct answer in the evaluation unit 403 of the classification result, and is based on the deviation. The medical data evaluation unit 502 to be classified evaluates the similarity between the classification data determined by the determination unit 602 and the data to be classified, and notifies the user of the evaluation result via the notification unit 503. The present invention has a configuration in which a plurality of classifiers are provided for a classification target and comparison is made with a classification data set 620 corresponding to the plurality of classifiers. With this configuration, for example, even if one classifier cannot sufficiently learn the features for classifying the medical data from the training data due to the distribution of the input data, it is possible to provide an appropriate classifier by providing a plurality of classifiers. Evaluation becomes possible. Further, for example, data in which teacher data is erroneously labeled can be separated from teacher data and pooled as teacher data (classification data) in different classifiers or medical data. When it is not possible to confirm the similarity of the classification data corresponding to multiple classifiers to meet the predetermined criteria, the training data when learning the classifiers includes data including features for classifying the medical data to be classified. It can be evaluated that it is not included. Therefore, it is possible to appropriately evaluate the input data based on the classification result of the classifier, which is the subject of the present invention.

Further, in the present embodiment, the evaluation result by the medical data evaluation unit 502 to be classified can be recognized by the user via the notification unit 503. The notification by the notification unit 503 allows the user to determine whether or not to execute the classification process. Further, when the data to be classified has a low similarity to the classification data in the classifier provided in the information processing device 101, the user indicates that the classification result by the classifier is unreliable. It can be recognized in advance. Further, the user can select in advance to cancel the classification process in which the classification result is expected to be unreliable. Regardless of whether the plurality of classifiers are provided in the single information processing device 101 or in the plurality of information processing devices, the classification process is performed in the virtual environment constructed between the plurality of information processing devices. May be good.

(Modification 1-1)
In the flow of creating the classifier in the present embodiment, it is expected that the number of classification data, the number of data in the i + 1 medical data set, and the number of classes decrease as the number of flows is repeated. Therefore, even if the likelihood to the classification data set corresponding to the classifier created by the multiple flows is larger than the classification data set corresponding to the classifier having a smaller number of flows than the classifier, the same criteria are used. It may not be desirable to determine the divergence. In this case, for example, as the number of flows increases, the reference by the determination unit 602 may be set larger, or the threshold value for executing the classification process may be set higher. The gist of the present modification 1-1 is to make the user recognize the evaluation of the input data, and for example, it is carried out to create a classifier with high likelihood without changing the judgment criteria. The number of flows may be notified, or the number of data learned from the classifier may be notified. Alternatively, both may be combined.

(Modification 1-2)
The evaluation unit 403 of the classification result of the first embodiment determines the degree of deviation between the classification result by the third classifier 402 and the correct answer, and for the class corresponding to the correct answer class, the likelihood of classification into the correct answer class and 1 Calculate the absolute value of the difference of .0. Furthermore, for classification into classes other than the correct answer, the absolute value of the difference between the likelihood of the class with the highest likelihood and 0.0 among the classes other than the correct answer is calculated, and the likelihood difference to the correct answer class and the correct answer are obtained. It was evaluated by calculating the sum of the likelihood differences to other than the class. On the other hand, the evaluation unit 403 of the classification result corresponding to the modified example 1 of the first embodiment determines the likelihood of the class having the highest likelihood in the class other than the correct answer from the likelihood of classification into the class corresponding to the correct answer class. Evaluate the divergence (calculate the evaluation value) by reducing it. When this evaluation value calculation method is applied, among the evaluation values calculated by the evaluation unit 403 of the classification result, the smallest deviation value is 1.0, and the largest deviation value is -1.0. Become. Further, the determination unit 602 determines, for example, −0.2 or less as the data of the i + 1 medical data set as a predetermined reference.

The evaluation unit 403 of the classification result may be only the absolute value of the difference between the likelihood of the correct answer classification and 1.0. In this case, the predetermined criterion of the determination unit 602 is determined based on the number of classifications. Specifically, it is determined whether or not the data is of the i + 1 medical data set based on how much the dissociation value is lower than the reciprocal of the classification number. For example, in the case of 3 classifications, if it is 0.3135 or less, which is about 5% lower than 1/3 = 0.33 ..., it is determined that the data is from the i + 1 medical data set. In the case of this modified example, it is not possible to make an evaluation considering the degree of mistake in classification other than the correct classification. For example, even if the data whose correct answer is "nuclear power plant" is classified as (0.32,0.68,0.0) or (0.32,0.34,0.34), there is a divergence. The values of are the same, and it is determined that the data is not the data of the i + 1 medical data set.

(Modification 1-3)
The notification unit 503 of the first embodiment displays the notification area 705 before the start of the classification process and causes the user to select execution or stop of the process. However, as shown in FIG. 9A, the similarity is displayed together with the classification result after the classification process. May be displayed. Further, as shown in FIG. 9B, the medical data evaluation unit 502 to be classified controls not to execute the classification process by the i-th classifier 402 when the similarity exceeds a predetermined value, and the notification unit 503. However, the fact that the process was not executed may be displayed together with the similarity. That is, the information processing device 101 determines whether or not the medical data to be classified is input data to the i-th classifier based on the similarity by the medical data evaluation unit 502 to be classified.

The notification area 901 in the notification unit 503 of FIG. 9A is an example of the notification area of this modification. In the notification area 901, the likelihood of "primary" is 83%, the likelihood of "transition" is 12%, and the likelihood of "benign" is 5%, and the similarity with the third classification data. Indicates that the degree is 95%.

The notification area 902 of FIG. 9B is also an example of the notification area of this modification. The notification area 902 has a similarity with the third classification data of 95%, and displays that the process has not been executed. It also displays a user confirmation button. The window of the notification area may be automatically closed after being displayed for a certain period of time without displaying the confirmation button.

According to this modification, when the similarity with the classification data set corresponding to the classifier created by the multiple classifier creation flow exceeds a predetermined value, the user instructs to execute or stop the execution of the process. No operation is required.

(Modification 1-4)
In this modification, a case where a trained classifier exists and the classification process is executed using the trained classifier will be described. Here, the processing in the case where the trained classifier is singular and the teacher data can be acquired will be described. First, the teacher data used when creating the trained classifier is compared with the first medical data set, duplicate data is deleted, and both data are integrated, and step S801 is set as the first data set 310. You may execute the process of. With this configuration, when new teacher data is acquired after the classifier is created, or by using another trained classifier, the classification process can be performed. If the classification target of the trained model is different, or if the teacher data used when creating the trained classifier and the first medical data set are widely dispersed, it is added as the first medical data set. Even if the first classifier is created by fine tuning or transfer learning for the trained classifier using the first medical data set without adding to the first medical data set. Good. This configuration is expected to improve the accuracy and robustness of classification with respect to the number and quality of teacher data.

<Embodiment 2>
In the second embodiment, as in the first embodiment, an information processing device which is a CAD system for making a diagnostic inference about a lung nodule shadow on a chest X-ray CT image will be described.

In the first embodiment, the determination unit 602 evaluates the degree of deviation between the likelihood of the classifier and the correct answer, and the medical data evaluation unit 502 to be classified determines the data based on the degree of the deviation (classification data set 620). ) And the medical data set to be classified were evaluated, and the result was notified to the user. In the second embodiment, the classifier setting unit 1001 for setting the classifier is further provided based on the evaluation result of the medical data evaluation unit to be classified. That is, the information processing device 101 has a classifier setting unit 1001 that sets a classifier that classifies the medical data to be classified among the plurality of classifiers based on the evaluation result by the medical data evaluation unit 502 to be classified. ..

Since the system configuration, hardware configuration, and configuration of the medical image DB 102 of the information processing apparatus according to the present embodiment are the same as those of the first embodiment, the description thereof will be omitted.

FIG. 10 is a functional block diagram of the information processing device of the present embodiment. In FIG. 10, 1001 is a classifier setting unit.

The classifier setting unit 1001 sets a third classifier for classifying the diagnosis name with respect to the medical data set 320 to be classified based on the evaluation result by the medical data evaluation unit 502 to be classified. The medical data evaluation unit 502 to be classified calculates the likelihood of classifying a plurality of classifiers as a class. Therefore, for example, when the flow of creating a classifier is carried out three times, the classifier has a configuration in which the first to third classifiers exist. Here, the medical data evaluation unit 502 to be classified uses each classifier as a class (label) and learns the classifier using the teacher data paired with the classification data corresponding to each classifier. The classification result when the medical data set to be classified is input is calculated so as to be 1 when the Softmax calculation is performed and the values (likelihoods) assigned to each class are totaled. For example, when the classification of (first classifier, second classifier, third classifier, etc.) is performed, the classification result is (0.6, 0.2, 0.1, 0.1). And the likelihood, which is the classification data of each class, is shown. In this case, the medical data to be classified indicates that the classification data corresponding to the first classifier has the highest likelihood. In other words, it indicates that the data in the first medical data set whose degree of deviation from the classification result classified by the first classifier is smaller than a predetermined criterion has a high likelihood. That is, it is shown that the reliability is maintained with a likelihood of 0.6 with respect to the classification result when the medical data to be classified is classified by the classifier 1. The classifier to be input is selected in response to the classification result from the medical data evaluation unit 502 to be classified.

When the classifier setting unit 1001 determines whether or not to input the medical data to be classified into the classifier based on the classification result by the medical data evaluation unit 502 to be classified, and determines the input to the classifier, the relevant person is concerned. A classifier for inputting the medical data set 320 to be classified is set, and the set classifier is used to classify the diagnosis name. The classifier setting unit 1001 simply sets the classifier showing the highest likelihood among the classification results from the medical data evaluation unit 502 to be classified as a classifier for performing classification processing. Alternatively, the classifier setting unit 1001 may set a threshold value, and the classifier that exceeds the threshold value and has the highest likelihood may be set as the classifier. Alternatively, the classifier setting unit 1001 may set a classifier whose likelihood exceeds the threshold value as a classifier for classifying the medical data to be classified. The classifier setting unit 1001 may set a classifier having a likelihood exceeding a threshold value and having the highest likelihood as a classifier.

When the setting unit 1001 of the classifier does not allow the medical data set 320 to be classified to be input to the classifier, for example, the likelihood of the classification result of the medical data evaluation unit 502 to be classified is greater than the threshold value. It may be small or the difference between the likelihoods to the class may be small. Alternatively, as described in the modified example 1-1, the classifier created by the multiple times classifier creation flow has more training data and the classifier than the classifier created by the lesser number of flows than the classifier. It may be unreliable in terms of the number of classes. Therefore, even if the threshold is set so that the classification of the diagnosis name is permitted only for the classifier i or less among the i-th classifiers, the lower limit of the number of training data for the classifier and the diagnosis name constituting the training data It is not necessary to allow input by the lower limit of the number of training data having.

FIG. 11 is an example of a display screen of the information processing apparatus of this embodiment.

In FIG. 11, the notification area 1101 is an example of the notification area by the notification unit 503. In the notification area 1101 of the present embodiment, a notification that a third classifier (described as "classifier 3" in the display screen example) is used together with the classification result by the classifier of the medical data evaluation unit 502 to be classified is used. Is displayed. That is, the notification unit 503 is characterized in that it notifies the information indicating the classifier that classifies the medical data to be classified and the classification result by the classifier.

FIG. 12 is a flow chart of processing of the information processing apparatus of this embodiment.

In the process of this embodiment, step S1212 is executed following step S811. In step S1212, the setting unit 1001 of the classifier determines whether or not input to the classifier is possible based on the classification result calculated by executing the classification process by the medical data evaluation unit 502 to be classified. As described above, the end condition is set by the number of data, the likelihood, the number of classes, the number of the classifier (the number i indicating how many times the classifier was created), and the like. According to step S1212, the classifier setting unit 1001 executes step S1213 when the end condition is satisfied, and executes step S1214 when it is determined that the end condition is satisfied.

In step S1213, the notification unit 503 notifies that the end condition is satisfied, and then the user selects whether to execute the classification process again. If the user selects the classification, step S1214 is executed.

In step S1214, the classifier setting unit 1214 sets at least one classifier for inputting the medical data set S320 to be classified. The method of setting the classifier is determined by the above-mentioned likelihood, the number of the classifier, the number of data, the number of classes, and the like.

In step S1215, in step S1214, the diagnosis name is classified by the third classifier (single or plural) set by the classifier setting unit 1001.

In step S1216, the notification unit 503 displays the classification result of the i-th classifier (single or plural) and the classifier used for the classification.

As described above, according to the present embodiment, the similarity between the plurality of classifiers and the classification data associated with the classifiers is determined, and the classifier setting unit 1001 determines the similarity with the classifiers based on the similarity. Judges whether or not input to is possible. By comparing the classification data with a plurality of classifiers, the data in which the characteristics could not be sufficiently learned even though the medical data set S320 to be classified was included in the training data, and the data not included in the training data. The distinction can be made more clearly. Furthermore, by performing similarity judgment by a classifier based on the classification data and classifying the diagnosis name based on the i-th classifier that meets a predetermined criterion, the reliability of the classification result indicated by the classifier is improved. Further, the output result from the classifier can be recognized in advance in the form of similarity with the classification data.

(Modification 2-1)
In the second embodiment, the classifier setting unit 1001 sets the classifier when the similarity, the number of data, and the number of classes are equal to or higher than a predetermined standard based on the classification result by the classifier trained with the classification data. Was done. The notification unit 503 may have an input means that allows the user to set the classifier after notifying information for selecting the classifier, such as the degree of similarity, the number of data, and the number of classes. For example, it is conceivable to set the classifier via a pull-down or an input reception unit for a check box or the like. With this configuration, it is possible to refer to the results using a plurality of created classifiers, or to select a classifier including the class for which the diagnosis name is to be confirmed.

(Modification 2-2)
In the second embodiment, the condition that the medical data set 320 to be classified is not input to the classifier is described on the condition that the difference in the likelihood of each classifier is small. However, if all the classifiers having a small difference in likelihood exert their classification ability with respect to the medical data set 320 to be classified, it is considered that the difference in likelihood between the two classifiers becomes small. Be done. In this case, even if the likelihood difference is small, a highly reliable diagnostic name can be classified by classifying with one of the classifiers.

In other words, it is appropriate to judge that the small likelihood difference allocated between the classifiers is low in similarity to the classification data (medical data that does not meet the prescribed criteria) with a small discrepancy between the classification result and the correct answer in the classifier. That is why it may not be. Therefore, when the likelihood difference between classes (classifiers) in the classification result of the classifier is small, the likelihood difference with other labels not classified in the classification data corresponding to the classifier is compared. That is, when the likelihood difference between the classifiers is small and the likelihood difference from other classes is large, the classifier setting unit 1001 serves as a classifier for classifying a plurality of classifiers having a small likelihood difference. Make settings and classify diagnostic names using the set classifier. Then, the classification results of the diagnosis names by a plurality of classifiers are compared and used as the classification results. With this configuration, although the classifier has the classification ability, the possibility of being excluded from the input data to the classifier is reduced because the likelihood difference is small.

(Modification 2-3)
In the modified example 2-2, when the difference in likelihood between the classifiers (classes), which is the result of classification by the classifier, is small, for example, the class of the classifier having a small difference in likelihood with the other classes. The configuration in which the classifier setting unit 1001 sets as a classifier for classifying the medical data to be classified when the difference from the likelihood corresponding to the above is compared and the difference is larger than a predetermined standard has been described.

In the modification 2-3, the classifier setting unit 1001 sets a plurality of classifiers as classifiers, normalizes the results of the plurality of classifiers, and then compares the sum to obtain the classification result of the diagnosis name. Good.

Here, as an example, it is assumed that a classifier exists up to a third classifier and the diagnostic names are A, B, C, D (others). Based on the medical data acquired by the medical data acquisition unit 501 of the classification target, the classification data corresponding to each classifier is used as training data by the medical data evaluation unit 502 of the classification target, and the classifier name is given to the label. Make a classification. It is assumed that the likelihood in that case is classified as follows. (1st classifier, 2nd classifier, 3rd classifier) = (0.6, 0.2, 0.2). In this modification, the classifier setting unit 1001 sets each classifier as a classifier for classifying the medical data set to be classified, and executes the classification process. And assume that the result is as follows. With respect to classifier A, (diagnosis name A, diagnosis name B, diagnosis name C, diagnosis name D) = (0.6, 0.4, 0.0, 0.0). The classifier B has (diagnosis name A, diagnosis name B, diagnosis name C, diagnosis name D) = (0.9, 0.1, 0.0, 0.0). The classifier C has a diagnosis name A, a diagnosis name B, a diagnosis name C, and a diagnosis name D) = (0.5, 0.5, 0.0, 0.0). Here, it is assumed that the number of training data and the number of classes between the classifiers are the same, but if the number of training data and the number of classes are different, the coefficient of classification between the classifiers is likely. If the reliability is low, such as in the normalization process for eliminating the variation in the degree or when the number of training data is less than a predetermined standard, the likelihood of each classifier may be multiplied as a coefficient.

Then, the result of class classification using the classifier name as a label is multiplied by the result of classifying the diagnosis name by each classifier. That is, the classifier A = (0.6 × 0.6, 0.6 × 0.4, 0.6 × 0.0, 0.6 × 0.0), and the same processing is performed with other classifiers. Do. Then, the sum of the diagnosis names for each classifier is obtained. The sum of the diagnosis names = (0.64, 0.36, 0.0, 0.0). It may be a classification result by a classification process for classifying the diagnosis names based on the sum of the diagnosis names.

<Embodiment 3>
As one aspect of the present invention, data in which the degree of deviation between the classification result of the classifier and the correct answer does not meet the predetermined criteria is stored as classification data in the classifier, and data that is not good at satisfying the predetermined criteria is classified into another classification. Pooled as vessel teacher data or medical data. As a result, multiple classifiers are created, and the reliability of classification for the medical data to be classified is evaluated by evaluating the similarity between the classification data corresponding to each of the multiple classifiers and the medical data to be classified. By providing a plurality of classifiers that can be recognized by the user and that have learned different characteristics, it has become possible to recognize and select a classifier suitable for inputting medical data to be classified.

In the present embodiment, a flow for creating classification data and a classifier, and a process performed by the information processing apparatus 101 on the teacher data by repeating the flow will be described.

The number of data and the number of classes decrease as the number of times of the flow of creating classification data and classifier from the third medical data is repeated. On the other hand, it is conceivable that there is classification data with the same label among the plurality of classifiers. Here, for the sake of simplicity, it is assumed that the label is a diagnostic name and the first medical data set is labeled with A to E (diagnosis name), and the description will be made with reference to FIG. FIG. 13 shows the classification data corresponding to the first to Nth classifiers and the number of samples of the classification data for each label of the diagnosis name constituting the classification data. As explained above, the number of data and the class as the lower classification data is from the first classification data corresponding to the first classifier to the Nth classification data corresponding to the Nth classifier. It shows that the number of is decreasing. On the other hand, considering, for example, the first classification data and the second classification data, the first classification data is the first classifier, and the data that does not meet the predetermined criteria (difference from the classification result from the correct answer). The degree of the data is, for example, less than a predetermined threshold value), that is, the data can be classified accurately by the first classifier. By comparison, the second classification data was not accurately classified by the first classifier, but was accurately classified by the second classifier, and corresponds to each label in the first classification data. It is conceivable that there is a feature for separating the data and the data corresponding to each of the second labels. Here, the classifier is trained by using the diagnosis name A in the first classification data and the diagnosis name A in the second classification data as separate labels. By assigning labels to each classifier and learning the classification data corresponding to the classifiers having the same diagnosis name A, it is possible to realize the robustness to the diagnosis name A by a plurality of classifiers. Become. A plurality of classifiers having the diagnosis name A may be provided as labels, or the classifier may be learned using a plurality of diagnosis names and a plurality of classification data as labels.

By using the classifier created by this configuration as the classifier used by the medical data evaluation unit 502 to be classified, the classification data corresponding to each of the plurality of classifiers is classified as a class. Also, more detailed evaluation results can be obtained.

For example, when the medical data to be classified that constitutes the medical data set 320 to be classified has a similarity of 95% with the data to which the diagnosis name A of the second classifier is given, FIG. 14 (a) shows. Notification area 1405 notifies the number of the classifier, the diagnosis name, and the degree of similarity. Further, in the case of this configuration, the classifier in the evaluation unit 502 of the medical data to be classified is a classifier having a class including each class of the plurality of classifiers. Therefore, the evaluation result in the medical data evaluation unit 502 to be classified can be evaluated without going through the step of setting the classifier by the classifier setting unit 1001. That is, when the likelihood of the second classifier to the diagnosis name A is high, but the likelihood of the first classifier to the diagnosis name A is small, learning is performed with characteristics different from those of the first classifier. It means that it is possible to classify diagnostic names with high reliability. If the content of the notification includes any of these, it may be notified together with other information. For example, as shown in the notification area 1406 in FIG. 14B, a plurality of diagnostic names may be classified, and the classification data having high similarity to each diagnostic name may be different. In such a case, the similarity and the diagnosis name may be notified by the notification unit 503 for each of the diagnosis names. Further, as an example, as shown in the pie chart 1500 of FIG. 15, the ratio of the classification data corresponding to each classifier and the classification result by the classifier may be associated and notified.

(Modification 3-1) Diagnosis name that the user wants to classify Custom Modification 3-1 describes a configuration in which a specific diagnosis name selected by the user is evaluated. Suppose a user wants to execute a classification process for, for example, diagnosis name A and diagnosis name B. A case where learning is performed using the diagnosis names constituting the classification data in a plurality of classifiers as labels in the diagnosis name A and the diagnosis name B as in the configuration of the third embodiment will be described. The class in which a plurality of classifiers classify is, for example, (diagnosis name A of the first classifier, diagnosis name A of the second classifier ... diagnosis name A of the N-1 classifier, classification of the Nth classifier. Vessel diagnostic name A, first classifier diagnostic name B, second classifier diagnostic name B ... N-1 classifier diagnostic name B, Nth classifier diagnostic name B, Others). The user may input the diagnosis name to be diagnosed or specify the class configuration via the input interface 208. Here, the classifier that classifies the diagnosis name is learned using the classification data corresponding to the created class. With this configuration, the class likelihood is output only for the diagnosis name desired by the user. If the classifier can be learned using the classification data corresponding to the created class, a plurality of classifications may be further created by the above-mentioned flow of creating the classifier.

<Embodiment 4>
The number and quality of teacher data is one issue for improving the performance of classifiers. Regarding the quality of teacher data, the quality is judged, for example, by whether or not annotations (pointing to labels) are properly attached. Some teacher data may be erroneously annotated, or the same annotation may be added to data that cannot be properly classified by the learned features.

In the present embodiment, a mode in which the teacher data is annotated again or the new medical data is annotated (hereinafter referred to as relabeling) will be described based on the plurality of classifiers and the flow of creating the classification data described above. do. Here, relabeling can be encouraged if the medical data described above is not properly annotated, or if it is appropriate to label the model structure of the classification differently. As shown in FIG. 13, it is assumed that there is medical data having a duplicate diagnosis name among the classification data corresponding to the plurality of classifiers, and the classifier corresponding to the diagnosis name is created. For example, it is effective when a plurality of samples exist for a plurality of classification data as shown in the diagnosis name A in FIG. The classifier created here classifies the first classification data of the diagnosis name A and the second classification data of the diagnosis name A into two classes. Of course, the number of classes can be multi-valued or any number. As the classifier, for example, a classifier called Gradient-weighted Class Activation Mapping (hereinafter referred to as Grad-CAM) is used. Grade-CAM is a technology that can display a heat map and likelihood of image parts that are greatly affected by each class. That is, it is possible to make the user recognize the characteristics when classifying the classification data corresponding to each of the first classifier class and the second classifier class into the two classes for the diagnosis name A together with the likelihood. it can. Further, the user can relabel the classification data based on the heat map by Grad-CAM, and by adding the relabeling data to the teacher data and training the classifier, the classification result by the classifier can be obtained. Reliability and robustness can be ensured. Hereinafter, the functional block of the information processing apparatus 101 will be described with reference to FIG. The information processing device 101 may have other configurations described in other embodiments, may have only the functional blocks described below separately, or may be configured by a plurality of information processing devices. May be done.

In FIG. 16, the information processing apparatus 101 does not satisfy a predetermined criterion for the degree of deviation between the correct answer label and the classification result among the classification results by each of the plurality of classifiers for the medical data having the correct answer label ( It has a classification data acquisition unit 1601 for acquiring classification data which is medical data (the degree of deviation is less than a predetermined threshold). Then, the classification data acquisition unit 1601 sends the classification data corresponding to different classifiers among the acquired classification data to the learning unit 1602 of the classifier as the teacher data with the same correct answer label. Send. The learning unit 1602 of the classifier learns the classifier based on the transmitted classification data. Then, the medical data acquisition unit 1603 acquires the medical data from the medical image DB 102 and transmits it to the label evaluation unit 1604 of the medical data. The label evaluation unit 1604 of the medical data transmits the acquired medical data to the learning unit 1602 of the classifier, and causes the classifier learned by the learning unit 1602 of the classifier to execute the classification process. Then, the medical data on which the classification process has been executed and the classification result are transmitted to the correct label setting unit 1605. The correct answer label setting unit 1605, which has acquired the medical data and the classification result, determines whether the medical data is labeled, and if the medical data is not labeled, sets a new label. On the other hand, if the label is hidden in the medical data, the label is replaced. The label evaluation unit 1604 of the medical data transmits the classification result by the classifier to the notification unit 1606. The notification unit 1606 notifies the classification result. That is, in the present embodiment, the information processing device 101 is an information processing device having a plurality of classifiers for classifying medical data, and the classification result of each of the plurality of classifiers for the medical data to which the correct answer label is given. Among them, the classification data acquisition unit 1601 for acquiring classification data in which the degree of deviation between the correct answer label and the classification result does not satisfy a predetermined criterion is provided. Further, among the acquired classification data, there is a classifier learning unit 1602 that learns the classifier using the classification data corresponding to different classifiers and having the same correct answer label as teacher data.

Further, it may have a correct answer label setting unit 1605 that sets a correct answer label of medical data based on the classification result of the learned classifier. Further, it has a notification unit 1606 for notifying the classification result.

FIG. 17 is a processing flow of the present embodiment. In step S1701, the classification data acquisition unit 1601 acquires a classification data set having the same label as the classification data corresponding to different classifiers among the classification data sets 620 in the medical image DB 102. In step S1702, the learning unit 1602 of the classifier learns the acquired classification data with a learning device based on, for example, Grad-CAM. In step S1703, the classification data acquisition unit 1601 determines whether or not there is classification data that further requires learning (generation) of the classifier, and if it is determined that generation of the classifier is necessary, the process returns to step S1701. And execute further processing. When the classification data acquisition unit 1601 determines that the learning (generation) of the classifier has been completed, the process proceeds to the subsequent step. Step S1704 is a step of acquiring medical data by the medical data acquisition unit 1603. The medical data acquired by the medical data acquisition unit 1603 may be data with a correct label or data without the correct label. For example, data having a small difference in class likelihood between classifiers using the classifier described in the above embodiment as a class, data requiring a correct label as new teacher data, and the like can be considered as targets. In step S1705, the label of the medical data is evaluated. The label evaluation unit 1604 of the medical data causes the learning device created by the learning unit 1602 of the classifier to execute the classification process. Then, as the classification result, the likelihood to each class and the heat map by Grad-CAM are acquired, and the classification result is notified via the notification unit 1606. That is, it is characterized in that the classification result by the learned classifier is the likelihood. Further, the classifier is a classifier based on Grad-CAM. The notification unit 1606 is characterized in that the heat map by Grad-CAM is notified. Further, as described above in FIG. 15, the notification unit 1606 may notify at least one of the number and the ratio of the training data corresponding to each classifier. In step S1706, the correct answer label setting unit 1605 determines the presence or absence of the current label, and if the label is attached, determines the consistency with the label. In the correct answer label setting unit 1605, the highest class among the classifiers by the classifier may be the correct answer label, or the class exceeding the threshold value may be the correct answer class. The correct label may be set by combining both. Label replacement is performed in step S1707. If the label is not attached, the label is attached based on the classification result (step S1707). In step S1706, if the label is attached and the label is highly reliable, the process is terminated. That is, the correct answer label setting unit 1605 in the information processing apparatus 101 is characterized in that it replaces the correct answer label of the medical data to which the correct answer label is given.

(Modification 4-1)
The fourth embodiment described above describes a method of relabeling classification data that cannot be classified by features that have been incorrectly annotated or learned by the classifier. In the modified example 4-1 when the correct answer label setting unit 1605 newly annotates the image data, labeling is promoted by using the Grad-CAM described in the fourth embodiment. That is, when data requiring new labeling is input to a classifier based on Grad-CAM, for example, an image region to be noted in the case of the diagnosis name A of the first classifier and a second classifier In the case of the diagnostic name A of the classifier, it is possible to acquire each notable image region. The user determines which label should be attached to the new image data based on the region of interest in the diagnostic name A of the first classifier and the region of interest in the diagnostic name A of the second classifier. be able to. It should be noted that the information processing apparatus 101 performs labeling based on the likelihood of classification by the classifiers using the diagnostic names of a plurality of classifiers as labels, regardless of the form in which the user is labeled based on the heat map by Grad-CAM. You may. Further, the data labeled by the information processing apparatus 101 may be used in a flow for creating a classifier as medical data in the medical image DB 102. That is, the setting of the correct answer label by the correct answer label setting unit is characterized in that the correct answer label is given to the medical data to which the correct answer label is not given.

101 Information processing device 102 Medical image DB
103 LAN
310 First i medical data set 320 Classification target medical data set 401 First i medical data acquisition unit 402 Second i classifier 403 Classification result evaluation unit 404 First i + 1 classifier learning unit 501 Acquisition of classification target medical data Department 502 Medical data evaluation unit to be classified 503 Notification unit 601 Probability acquisition unit 602 Judgment unit 603 Medical data set of the third i + 1

Claims (16)

A likelihood acquisition unit that acquires the class likelihood for medical data with a correct label using the first classifier that classifies medical data.
An evaluation unit for classification results that evaluates the degree of dissociation based on the class likelihood acquired by the likelihood acquisition unit and the class corresponding to the correct label.
A determination unit that determines whether or not the degree of deviation by the evaluation unit of the classification result satisfies a predetermined criterion, and
A learning unit of a classifier that learns a second classifier using medical data determined to meet a predetermined criterion by the determination unit as teacher data,
An information processing device characterized by having. The claim is characterized in that the medical data determined by the determination unit not to meet the predetermined criteria is used as the classification data corresponding to the classifier that classifies the medical data determined not to meet the predetermined criteria. The information processing apparatus according to 1. The second classifier learned from the teacher data is used as a classifier for classifying the medical data, and the medical data determined to meet the predetermined criteria is used as the medical data, and the likelihood acquisition unit and classification are used. The information processing apparatus according to claim 1 or 2, further comprising a control unit capable of repeatedly executing the processes of the result evaluation unit, the determination unit, and the learning unit. The information processing apparatus according to claim 3, wherein the information processing apparatus has a plurality of classifiers and a plurality of classification data corresponding to each of the plurality of classifiers by the repetition. A claim characterized by having a classification target medical data evaluation unit that evaluates classification target medical data using a classifier that has learned classification data with each of the plurality of classifiers as labels as teacher data. The information processing apparatus according to 4. The information processing apparatus according to claim 5, further comprising a notification unit for notifying an evaluation result by the medical data evaluation unit to be classified. The information processing apparatus according to claim 5, wherein the classifier in the medical data evaluation unit to be classified is a classifier that calculates a classification result into a class corresponding to the plurality of classifiers by likelihood. The claim is characterized by having a classifier setting unit for setting a classifier that classifies the medical data to be classified among the plurality of classifiers based on the evaluation result by the medical data evaluation unit to be classified. The information processing apparatus according to 5 or 7. The information processing device according to claim 8, wherein the setting unit of the classifier sets the classifier having the highest likelihood as a classifier for classifying the medical data to be classified. The information processing apparatus according to claim 8 or 9, wherein the setting unit of the classifier sets the classifier whose likelihood exceeds the threshold value as a classifier for classifying the medical data to be classified. .. The information processing according to any one of claims 8 to 10, wherein the information processing indicating the classifier set by the setting unit of the classifier and the notification unit for notifying the classification result by the classifier are provided. apparatus. In the repetition, it is determined that the number of teacher data for learning the classifier is less than or equal to the predetermined number, the classification accuracy of the classifier is determined to be less than or equal to the predetermined value, the overfitting is determined, the unlearned is determined, and the number of times specified by the user is exceeded. The information processing apparatus according to claim 3 or 4, wherein the repetition is terminated when any of the determination processes is performed. The likelihood acquisition step of acquiring the class likelihood for the medical data with the correct label using the first classifier that classifies the medical data, and
An evaluation step of a classification result for evaluating the degree of dissociation based on the class likelihood acquired by the likelihood acquisition unit and the class corresponding to the correct label, and
A determination step for determining whether or not the deviation by the evaluation unit of the classification result satisfies a predetermined criterion, and
A learning step of a classifier for learning a second classifier using medical data determined to meet a predetermined criterion by the determination unit as teacher data, and
An information processing method characterized by having. 13. The thirteenth aspect of the present invention is characterized in that the data determined not to meet the predetermined criteria by the determination step is used as the classification data corresponding to the classifier that classifies the data determined not to meet the predetermined criteria. The information processing method described. The second classifier learned from the teacher data is used as a classifier for classifying the medical data, and the medical data determined to meet the predetermined criteria is used as the medical data, and the likelihood acquisition step and classification are performed. The information processing method according to claim 13 or 14, further comprising a control step capable of repeatedly executing the processing of the result evaluation step, the determination step, and the learning step. A program for causing a computer to execute the information processing method according to claim 15.

Images