KR102543013B1 - Method for evaluating diagnostic ablility of an expert, artificial intelligence based method for assisting diagnosis and device using the same - Google Patents

Abstract

The present invention is a diagnosis assisting method implemented by a processor, comprising the steps of receiving medical data for an object, using a machine learning model configured to output a diagnosis evaluation result by taking the medical data as an input, and performing diagnosis based on the medical data. Determining the evaluation result, receiving the diagnostic evaluation result determined from the medical staff for the medical data, determining the first classification probability based on the diagnostic evaluation result for the individual machine learning model, the diagnostic evaluation result determined from the medical staff A method for assisting diagnosis, including determining a second diagnosis classification probability based on the first classification probability and determining a third classification probability for an individual based on the first classification probability and the second classification probability, and for assisting diagnosis using the same It provides a method for optimizing the device and furthermore the machine learning model.

Description

Expert's diagnostic ability evaluation method, artificial intelligence-based diagnostic assistance method, and device using the same

The present invention relates to an expert's diagnosis ability evaluation method, an artificial intelligence-based diagnosis assistance method, and a diagnosis assistance device, and more specifically, to a machine learning model that presents optimal automatic diagnosis results to a human expert-human cooperative diagnosis assistance. It's about the system.

When medical institutions use medical data to diagnose diseases, establish treatment plans, or evaluate treatment progress, there are frequent cases in which the size, range, and function of lesions must be quantitatively measured.

On the other hand, since medical data may give different clinical results depending on medical personnel, there may be many cases in which an evaluator fails to recognize a lesion in an image by mistake.

In particular, when an initial diagnosis of a disease is made using medical data, the presence or absence of the disease and the degree of progression are identified by detecting lesions in the medical data. At this time, in the process of detecting a lesion from a large amount of medical data obtained from a number of patients, there may be a difference in accuracy depending on the level of skill or fatigue of medical personnel, and thus the existence of the lesion may be overlooked.

Therefore, for diagnosis based on medical data, there is a demand for the development of a new diagnostic assistance system that evaluates medical data with high accuracy.

The background description of the invention has been prepared to facilitate understanding of the present invention. It should not be construed as an admission that matters described in the background art of the invention exist as prior art.

As a way to solve the above problems, an image analysis system such as a computer aided diagnosis (CAD) system has appeared. In this case, the computer-assisted diagnosis system may be a technology that analyzes or determines medical data in advance by a computer and provides a medical staff with the presence or absence of a lesion and the location or change of the lesion.

The computer-assisted diagnosis system not only identifies the presence of lesions in medical data and finds and displays the location of lesions, but also automatically or semi-automatically divides lesions and calculates their sizes, thereby limiting the accuracy of manual measurement. was expected to overcome.

However, the computer-assisted diagnostic system is expensive and has a limited supply, and since it is shared by a number of users through a network, it may be cumbersome to use. In addition, in order to use the computer-aided diagnosis system, medical staff must download image data from a Picture Archiving and Communication System (PACS), copy it to a Computer Aided Diagnosis (CAD) system, and re-enter it in the CAD system. Then, a very complicated procedure may be performed in which the target function must be executed by manipulating the menu, waited until the execution is completed, and then inquired about the result and inputted the result to the PACS again.

Because of these inconveniences, it may be difficult to apply the computer-assisted diagnostic system to actual clinical practice.

In particular, as it is difficult to confirm a diagnosis result only with the result of a computer-assisted diagnosis system, there may be difficulties in using it for actual patient care even though a technology capable of providing more improved medical services has been developed.

The inventors of the present invention could expect that a new diagnosis assistance system based on human diagnostic ability evaluation would supplement the limitations and problems of the conventional diagnosis assistance system.

In particular, the inventors of the present invention build a final machine learning model by setting weights according to the machine learning model and the diagnostic ability of each medical staff, and when diagnosis is performed based on actual medical data, by reflecting the pre-estimated weight, could be expected to improve the accuracy of

More specifically, the inventors of the present invention propose an expert-machine learning model joint diagnosis assistance system, configured so that, for various types of medical data, an expert and an artificial intelligence-based predictive model read the same measurement data and make a diagnosis decision. wanted to provide. For example, an expert-machine learning model joint diagnosis assistant system is required to determine a heart disease based on an electrocardiogram, a brain disease based on an EEG signal, or a neuropsychiatric disease based on cognitive behavioral data. In this case, it may be configured to make a final diagnosis decision based on the judgment result of the medical staff using a predictive model trained on the basis of each medical data.

Furthermore, the inventors of the present invention could expect that by providing a final machine learning model optimized according to the diagnosis ability of medical staff, the accuracy of disease diagnosis could be improved compared to directly using a pre-learned machine learning model.

At this time, the inventors of the present invention, for the diagnosis assistance system, use the already trained machine learning model and the estimated weight according to the medical staff's diagnostic ability, and when new data comes in, the classification result of the medical staff and the classification result of the machine learning model It was designed to reflect the weight. Then, the class (disease) that produces the maximum value could be designed to be finally judged as the category of the data.

Furthermore, the inventors of the present invention, when presenting the final evaluation result based on the machine learning model for the diagnosis assistance system, could design to calculate and output the relative reliability compared to the diagnostic ability of the medical staff together with the classification result.

Therefore, the problem to be solved by the present invention is to determine the already trained machine learning model for the evaluation results of medical data for evaluation and the classification weight for each class of medical staff, respectively, and when new medical data are input, the machine learning model and An object of the present invention is to provide a diagnosis assisting method configured to classify and provide a final class by reflecting a weight determined on a medical staff evaluation result, and a device using the same.

Another problem to be solved by the present invention is to provide a method for setting weights based on evaluation results of a machine learning model for medical data for evaluation and evaluation results of medical staff, and optimizing final evaluation based thereon.

The tasks of the present invention are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, a diagnosis assisting method according to an embodiment of the present invention is provided. The present method, as a diagnosis assisting method implemented by a processor, includes the steps of receiving medical data for an individual, using an individual machine learning model configured to output a diagnostic evaluation result by taking the medical data as an input, based on the medical data. Step of determining the diagnostic evaluation result, step of receiving the determined diagnostic evaluation result from the medical staff about the medical data, based on the pre-assigned weight for each machine learning model and the diagnostic evaluation result of the individual machine learning model, the first classification probability Determining a second classification probability based on a pre-assigned weight to the medical staff and a diagnosis evaluation result determined from the medical staff, and based on the first classification probability and the second classification probability, a first classification probability for the individual is determined. 3 determining the classification probability. Steps 1 and 2 may be omitted, and in step 3, the final evaluation result may be determined by applying weights already assigned to the evaluation results of medical staff and the results of individual machine learning models.

According to a feature of the present invention, the pre-assigned weight may be a weight assigned to at least one disease among a plurality of diseases. At this time, the pre-assigned weights are the steps of receiving medical data for evaluation in which at least one disease is labeled as the correct disease, the step of determining the diagnostic evaluation result of the medical data for evaluation, and the medical staff's diagnosis of the medical data for evaluation. Based on the step of receiving evaluation results, the correct disease, the diagnostic evaluation results of individual machine learning models for medical data for evaluation, and the medical staff's diagnostic evaluation results for medical data for evaluation, the classification accuracy of each medical staff and machine learning model is determined. It may be determined through the step of determining, and the step of determining a classification weight of at least one disease for each of the machine learning model and medical staff based on each classification accuracy.

According to another feature of the present invention, the step of determining the diagnostic evaluation result for the medical data for evaluation may include determining or inputting a classification probability of the at least one disease based on the medical data for evaluation using a machine learning model. It may include the step of receiving. Furthermore, the step of receiving the diagnostic evaluation result of the medical staff for evaluation medical data includes the step of receiving input of the classification reliability for the classification of the disease selected from the medical staff, based on the classification reliability, the classification probability of the disease selected from the medical staff. It may include the step of determining, and the step of receiving an input of the classification probability of the selected disease from a medical staff. In addition, the step of determining the classification accuracy of the medical staff and the machine learning model, based on the correct disease, the classification probability of at least one disease for the machine learning model, and the classification probability of the selected disease from the medical staff, the medical staff and the machine learning model determining the accuracy of each classification.

According to another feature of the present invention, the step of determining at least one disease classification weight may include the classification probability of at least one disease for a machine learning model and medical staff so that the summed disease classification probabilities are classified as correct diseases. The method may include summing the classification probabilities of the selected diseases, and determining weights at which an error between the correct answer and an error of the summed disease classification probabilities is minimized or mutual information is maximized.

According to another feature of the present invention, after the step of receiving the classification reliability for the classification of the selected disease from the medical staff, the method, based on the classification reliability for the classification of the disease selected from the medical staff, the remaining diseases among the plurality of diseases. A step of determining reliability for may be further included.

According to another feature of the present invention, each diagnostic evaluation result includes a classification probability for each of a plurality of diseases determined from an individual machine learning model and each medical staff, and the first classification probability is a weighted machine learning model. It may include classification probabilities for each of a plurality of diseases determined by Also, the second classification probability may include classification probabilities for each of a plurality of diseases determined by a medical staff to which weights have been added. In this case, the step of determining the third classification probability is the maximum of the classification probability for each of the plurality of diseases determined by the weighted machine learning model and the classification probability for each of the plurality of diseases determined by the weighted medical staff. It may include determining the disease as having value.

According to another feature of the present invention, determining the third classification probability may further include determining the third classification probability based on the first classification probability.

According to another feature of the present invention, after the step of determining the third classification probability, based on the first classification probability and the second classification probability, calculating the relative reliability of the medical staff, and providing the relative reliability may further include.

According to another feature of the present invention, there are a plurality of machine learning models, the pre-assigned weights are weights assigned to each of a plurality of diseases, and the method uses a plurality of machine learning models based on medical data. determining a diagnostic evaluation result, assigning a diagnostic evaluation result for each of the plurality of machine learning models to each of the plurality of machine learning models to obtain a weighted first classification probability for each of the plurality of machine learning models Assigning weights to each, summing the weighted first classification probabilities for each of the plurality of machine learning models so as to obtain a summed diagnostic evaluation result for each of a plurality of diseases, and summative diagnostic evaluation of the maximum value among the plurality of diseases The method may further include determining a disease having a result as a disease with a high risk of developing the subject.

In order to solve the above problems, a weight optimization method of a machine learning model according to another embodiment of the present invention is provided.

The optimization method includes the step of receiving medical data for evaluation, the step of learning an individual machine learning model configured to output a diagnostic evaluation result by using the medical data as an input, using the medical data for evaluation, Evaluating diagnostic characteristics of medical staff, determining disease classification weights for individual machine learning models and medical staff, based on the learning results of individual machine learning models for evaluation medical data and diagnostic characteristics of medical staff, and and optimizing one or fewer machine learning models based on the disease classification weight for each. This process can already be performed independently of a personal medical practitioner.

In order to solve the above problems, a diagnosis assisting method according to another embodiment of the present invention is provided.

The diagnosis assisting method includes the step of receiving medical data for evaluation, the step of learning a machine learning model configured to output a diagnostic evaluation result by using the medical data as an input, the step of using the medical data for evaluation, Evaluating the diagnosis characteristics of medical staff, determining disease classification weights for each of the machine learning model and medical staff based on the learning result of the machine learning model for the medical data for evaluation and the diagnostic characteristics of the medical staff, and each disease Fitting a machine learning model based on classification weights, receiving additional medical data for the individual, determining a diagnostic evaluation result based on the additional medical data using the machine learning model, receiving a diagnostic evaluation result determined from a medical staff, determining a first classification probability based on a pre-assigned weight for the machine learning model and a diagnostic evaluation result of the machine learning model, a pre-assigned weight for the medical staff and determining a second classification probability based on the diagnosis evaluation result determined by the medical staff, and determining a third classification probability based on the first classification probability and the second classification probability.

In order to solve the above problems, a device for assisting diagnosis is provided according to another embodiment of the present invention.

The device includes a communication unit configured to receive medical data for an entity, and a processor configured to communicate with the communication unit.

At this time, the processor

By using a machine learning model configured to output diagnostic evaluation results with medical data as input, a diagnostic evaluation result is determined based on the medical data, the diagnostic evaluation result determined from the medical staff is input, and the machine learning model A first classification probability is determined based on a pre-assigned weight and a diagnostic evaluation result of the machine learning model, and a second classification probability is determined based on a pre-assigned weight for the medical staff and a diagnosis evaluation result determined from the medical staff. , determine a third classification probability for the individual based on the first classification probability and the second classification probability.

According to a feature of the present invention, the pre-assigned weight is a weight assigned to at least one disease among a plurality of diseases, and the communication unit is further configured to receive medical data for evaluation in which the at least one disease is labeled as an answer disease. can At this time, the processor determines the diagnostic evaluation result for the medical data for evaluation using a machine learning model, receives the diagnostic evaluation result of the medical staff for the medical data for evaluation, and determines the correct disease and the medical data for evaluation. Based on the diagnostic evaluation result of the machine learning model and the diagnostic evaluation result of the medical staff for evaluation, the classification accuracy of the medical staff and the machine learning model is determined, and based on each classification accuracy, the machine learning model and the medical staff It may be further configured to determine a classification weight of at least one disease for each.

According to a feature of the present invention, the processor determines the classification probability of the at least one disease for the machine learning model based on the medical data for evaluation using the machine learning model, and determines the classification probability of the disease selected from the medical staff. Receive classification reliability, determine the classification probability of a disease selected from medical staff based on the classification reliability, receive classification probability of a disease selected from medical staff, receive a correct disease, classification probability of at least one disease for a machine learning model, and It may be further configured to determine classification accuracy of each of the medical staff and the machine learning model based on the classification probability of the disease selected from the medical staff.

According to another feature of the present invention, the processor sums the classification probability of at least one disease for the machine learning model and the classification probability of a disease selected from medical staff to obtain a summed disease classification probability, and the summed disease classification probability is It may be further configured to determine the weight that becomes the maximum value.

According to another feature of the present invention, the processor may be further configured to determine reliability for the remaining diseases among the plurality of diseases based on the classification reliability for the classification of the disease selected from the medical staff.

According to another feature of the present invention, each diagnostic evaluation result includes a classification probability for each of a plurality of diseases determined from a machine learning model and the medical staff, and the first classification probability is the weighted machine learning probability. Classification probabilities of the plurality of diseases determined by the model may be included, and the second classification probability may include classification probabilities of the plurality of diseases determined by the medical staff to which weights have been added. At this time, the processor determines the disease having the maximum value among the classification probabilities for each of the plurality of diseases determined by the weighted machine learning model and the classification probabilities for each of the plurality of diseases determined by the medical staff to which the weight is added. It may be further configured to determine a high risk of onset.

According to another feature of the invention, the processor may be further configured to determine a third classification probability based on the first classification probability.

Other embodiment specifics are included in the detailed description and drawings.

The present invention provides an ensemble classifier-based diagnostic assist system that reflects the diagnostic ability of medical staff and the diagnostic ability of a machine learning model based on an artificial intelligence algorithm, thereby supplementing the limitations and problems of conventional diagnostic assist systems.

In particular, the present invention provides a diagnostic assistance system capable of machine learning model-human cooperation based on human diagnostic ability evaluation, so that even when the diagnostic ability of the machine learning model is equal to or higher than that of an expert, it is not 100% reliable. can overcome the limitations and problems of the diagnosis assist system. In particular, it is possible to hold the medical staff on the issue of responsibility for diagnosis.

In addition, the present invention establishes a machine learning model by setting weights according to the diagnosis ability of each medical team and a machine learning model, and provides a diagnosis assistance system configured to reflect weights when diagnosis is performed based on actual medical data, Diagnosis results with improved accuracy and reliability can be provided.

In addition, the present invention can provide a diagnosis assistance system designed to calculate and output a relative reliability compared to the diagnostic ability of medical staff together with a classification result when an evaluation result based on a machine learning model is presented. Accordingly, the medical staff can easily recognize a weak diagnosis range by themselves. In addition, although the machine learning model discriminates with low accuracy, the incidence of diseases that medical staff can discriminate more accurately than the machine learning model can be judged with higher accuracy.

That is, the present invention can present diagnosis results by pre-determining the weaknesses of medical staff and giving weights to the judgment results of the machine learning model, emphasizing areas to be emphasized, and presenting diagnosis results with relatively low reliability in other areas. there is.

Furthermore, the present invention sets a classification weight for a machine learning model and medical staff based on the evaluation result of medical data for evaluation, and based on this, provides a machine learning model having optimal diagnostic performance according to the individual medical staff's ability. can do.

In other words, the machine learning model pre-trained by the learning data set can present classification results to medical staff with relative weights based on the ability of medical staff to use them, and can induce medical staff than machine learning models that do not. Diagnostic results can be provided.

The present invention regarding the process of presenting classification results based on relative weights according to individual skills may provide an educational effect for improving the skills of medical staff.

Effects according to the present invention are not limited by the contents exemplified above, and more various effects are included in the present specification.

1A exemplarily illustrates a diagnosis assistance system based on a diagnosis assistance device according to an embodiment of the present invention.
1B exemplarily illustrates the configuration of a device for assisting diagnosis according to an embodiment of the present invention.
1C illustrates a configuration of a medical staff device that receives and outputs information about a disease from a device for assisting diagnosis according to an embodiment of the present invention.
2 illustrates a procedure of a diagnosis assisting method according to an embodiment of the present invention.
3a and 3b show the present invention A procedure for determining a diagnostic evaluation result for a disease according to a diagnosis assisting method according to an embodiment is illustrated as an example.
4 and 5 exemplarily illustrate a procedure for determining weights in a diagnosis assisting method according to another embodiment of the present invention.
6A to 6B illustrate evaluation results according to a diagnosis assisting method according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various forms different from each other, only these embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention pertains. It is provided to completely inform the person who has the scope of the invention, and the present invention is only defined by the scope of the claims. In connection with the description of the drawings, like reference numerals may be used for like elements.

In this document, expressions such as "has," "may have," "includes," or "may include" indicate the existence of a corresponding feature (eg, numerical value, function, operation, or component such as a part). , which does not preclude the existence of additional features.

In this document, expressions such as “A or B,” “at least one of A and/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” (1) includes at least one A, (2) includes at least one B, Or (3) may refer to all cases including at least one A and at least one B.

Expressions such as “first,” “second,” “first,” or “second,” used in this document may modify various elements, regardless of order and/or importance, and refer to one element as It is used only to distinguish it from other components and does not limit the corresponding components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, without departing from the scope of rights described in this document, a first element may be named a second element, and similarly, the second element may also be renamed to the first element.

A component (e.g., a first component) is "(operatively or communicatively) coupled with/to" another component (e.g., a second component); When referred to as "connected to", it should be understood that the certain component may be directly connected to the other component or connected through another component (eg, a third component). On the other hand, when an element (eg, a first element) is referred to as being “directly connected” or “directly connected” to another element (eg, a second element), the element and the above It may be understood that other components (eg, third components) do not exist between the other components.

As used in this document, the expression "configured to" means "suitable for," "having the capacity to," depending on the circumstances. ," "designed to," "adapted to," "made to," or "capable of." The term "configured (or set) to" may not necessarily mean only "specifically designed to" hardware. Instead, in some contexts, the expression "a device configured to" may mean that the device "is capable of" in conjunction with other devices or components. For example, the phrase "a processor configured (or configured) to perform A, B, and C" may include a dedicated processor (eg, an embedded processor) to perform those operations, or by executing one or more software programs stored on a memory device. , may mean a general-purpose processor (eg, CPU or application processor) capable of performing corresponding operations.

Terms used in this document are only used to describe a specific embodiment, and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the art described in this document. Among the terms used in this document, terms defined in a general dictionary may be interpreted as having the same or similar meaning as the meaning in the context of the related art, and unless explicitly defined in this document, an ideal or excessively formal meaning. not be interpreted as In some cases, even terms defined in this document cannot be interpreted to exclude the embodiments of this document.

Each feature of the various embodiments of the present invention can be partially or entirely combined or combined with each other, and as those skilled in the art can fully understand, various interlocking and driving operations are possible, and each embodiment can be implemented independently of each other. It may be possible to implement together in an association relationship.

For clarity of interpretation of this specification, terms used in this specification will be defined below.

As used herein, the term "subject" may mean any subject for which a disease is to be predicted. In this case, the subject may be a subject suspected of a specific disease. For example, pancreatic cancer, urinary cancer, bladder cancer, colorectal cancer, colon cancer, breast cancer, prostate cancer, kidney cancer, hepatocellular cancer, thyroid cancer, gallbladder cancer, lung cancer, non-small cell lung cancer, small-cell lung cancer, ovarian cancer, cervical cancer, stomach cancer , endometrial cancer, esophageal cancer, head and neck cancer, melanoma, neuroendocrine cancer, CNS cancer, brain tumor, glioma, anaplastic oligodendroglioma, adult glioblastoma multiforme, adult anaplastic astrocytoma, bone cancer, soft tissue sarcoma, retina Blastoma, neuroblastoma, peritoneal effusion, malignant pleural effusion, mesothelioma, Wilms' tumor, trophoblast neoplasia, hemangiopericytoma, Kaposi's sarcoma, mucinous carcinoma, round cell carcinoma, squamous cell carcinoma, esophageal squamous cell carcinoma, oral carcinoma, adrenal gland It may be an object suspected of a disease such as cortical cancer, ACTH-producing tumor, ear disease, etc., but is not limited thereto.

Preferably, the disease disclosed in the present specification is a disease that can be diagnosed by medical data obtained from a medical data providing device such as CT, X-ray, MRI, F-MRI, ultrasound, endoscope and ultrasound endoscope, and further electrocardiogram and brain wave signal. can cover

On the other hand, the subject disclosed in this specification may be any mammal except for humans, but may include humans without being limited thereto.

As used herein, the term "medical data" is data obtained from an imaging diagnosis device, and may include all medical data capable of determining a diagnosis of a disease. For example, medical data may include, but are not limited to, CT images, X-ray images, MRI images, F-MRI images, ultrasound images, endoscopic images, and endoscopic ultrasound images of suspected lesions.

Meanwhile, medical data may be provided as a medical video consisting of a plurality of frames. According to the diagnosis assisting method according to an embodiment of the present invention, a diagnostic evaluation result may be determined for each frame of a medical video. As a result, the present invention can provide a streaming service by predicting a lesion simultaneously with receiving a medical image from an image diagnosis device, and can provide diagnosis information of a disease in real time. In this case, the medical image may be a 2D image or a 3D image.

As used herein, the term "machine learning model" may be a model configured to output an evaluation result related to a disease by taking medical data as an input.

More specifically, the machine learning model may be a model learned to classify the medical data into a class corresponding to a specific disease when medical data is input. At this time, the machine learning model may be configured to probabilistically calculate the risk of disease onset and then classify into a specific class based on this.

In particular, in the diagnostic assist system according to various embodiments of the present invention, a decision on a disease with a high risk of onset may be made based on a machine learning model and evaluation results of medical staff.

For example, when a diagnosis assistant system of expert-prediction model joint judgment is necessary to determine heart disease based on electrocardiogram, brain disease based on EEG signal, or neuropsychiatric disease based on cognitive behavioral data. , It can be configured to make a final diagnosis decision based on the prediction model learned on the basis of each medical data and the result of the decision of the medical staff.

In this case, the "diagnostic evaluation result" may refer to a scale associated with the risk of developing a disease, and may include the probability of developing a specific disease, the probability of belonging to a class corresponding to a specific disease, and the reliability of classification. Moreover, the diagnostic evaluation result may exist as a probability vector.

Meanwhile, within the present specification, “diagnostic evaluation result” may be used interchangeably with “diagnostic characteristic” and further “classification probability”.

On the other hand, "first classification probability" may refer to disease classification probability of the machine learning model in which weights previously assigned to the machine learning model are reflected.

Furthermore, the "second classification probability" may refer to a disease classification probability of the medical staff, in which weights pre-assigned to the medical staff are reflected.

In this case, the disease classification probability may be calculated based on the evaluation result of the medical staff for the evaluation data and the reliability of the selection.

In addition, the "third classification probability" may refer to a final classification probability in which a weight is applied to the classification result of the machine learning model and medical staff. In this case, the third classification probability may include a classification probability calculated for the suspected disease, a probability of being classified into a class corresponding to the suspected disease, and a risk of developing an upper, middle, or lower specific disease.

Meanwhile, the machine learning model may be a pretrained model to classify a disease based on a training data set prepared according to a target disease. In this case, the machine learning model may be further optimized by assigning a relative weight based on the diagnosis ability of medical staff so as to classify the disease with high reliability. More specifically, medical staff may have differences in diagnostic ability among individuals, so when optimizing the machine learning model, when the diagnostic ability of the medical staff is reflected, the diagnostic performance of the machine learning model can be further improved.

At this time, the machine learning model of the present invention is a CNN (Convolutional Neural Network) based VGG net, R, DenseNet and FCN (Fully Convolutional Network) having an encoder-decoder structure, SegNet, DeconvNet, DeepLAB V3+, U-net, etc. At least two models selected from deep neural network (DNN), SqueezeNet, Alexnet, ResNet18, MobileNet-v2, GoogLeNet, Resnet50, Resnet101, and Inception-v3 may form an ensemble. However, it is not limited to deep learning-based models and can be any model that provides diagnostic results.

As used herein, the term "medical staff" may mean a user using a machine learning model. In this specification, medical staff may be used interchangeably with “expert”.

As used herein, the term "weight" may mean a weight for classifying (evaluating) a disease by a machine learning model or by a medical staff. For example, if the ability of medical staff to classify disease A is superior to that of the machine learning model, a weight greater than 1 may be applied to classifying disease A. Furthermore, if the accuracy of the machine learning model for classifying disease B is lower than that of medical staff, a weight of less than 1 may be applied to the classification of disease B by the machine learning model.

Meanwhile, within the present specification, weights may be used in the same meaning as classification weights and disease classification weights.

That is, for each of the classifiers composed of the machine learning model and the medical staff, weights that provide optimal diagnosis performance for each class may be set.

More specifically, the classification result of the medical staff for the evaluation medical data for which the correct answer is determined for the diagnostic ability of the medical staff is obtained, and the classification probability value of the machine learning model for the same evaluation medical data (eg, the diagnostic evaluation result) this is obtained Next, a weight that best matches the correct disease can be estimated by adding the classification probability value by the model and the result value of the medical staff.

Hereinafter, a diagnosis assistance system based on a diagnosis assistance device according to an embodiment of the present invention will be described with reference to FIGS. 1A to 1C.

1A exemplarily illustrates a diagnosis assistance system based on a diagnosis assistance device according to an embodiment of the present invention. 1B exemplarily illustrates the configuration of a device for assisting diagnosis according to an embodiment of the present invention. 1C illustrates a configuration of a medical staff device that receives and outputs information about a disease from a device for assisting diagnosis according to an embodiment of the present invention.

First, referring to FIG. 1A , a diagnosis assistance system 1000 may be a system configured to provide disease-related information based on medical data on an individual. At this time, the diagnosis assistance system 1000 is configured to determine the risk of developing a disease for an individual based on medical data such as ear endoscope medical data, and receives information on medical data evaluation. It may be composed of a medical staff device 200 and a medical data providing device 300 providing medical data.

First, the device 100 for assisting diagnosis includes a general-purpose computer, laptop, and/or data server that performs various calculations to evaluate the risk of developing a disease based on the user's medical data provided from the medical data providing device 300. can include In this case, the medical staff device 200 may be a device for accessing a web server providing a web page about a disease or a mobile web server providing a mobile web site, but is not limited thereto. .

Specifically, the device for assisting diagnosis 100 may receive medical data from the medical data providing device 300, classify a disease with a high risk of onset from the received medical data, and provide information related to the risk of onset of the disease. there is. At this time, the medical data providing device 300 may provide a video or a frame within the video so as to provide various image information on one entity. Furthermore, the device 100 for assisting diagnosis may provide information associated with a risk of developing a disease based on a frame in a video.

The device 100 for assisting diagnosis may provide data associated with the onset of a disease for an individual to the medical device 200 .

In this way, data provided from the device 100 for assisting diagnosis may be provided as a web page through a web browser installed in the medical device 200, or may be provided in the form of an application or program. In various embodiments, this data may be provided in a form incorporated into the platform in a client-server environment.

Next, the medical staff device 200 is an electronic device that provides a user interface for requesting information on the onset of a disease for an individual and displaying evaluation result data, such as a smartphone, a tablet PC (Personal Computer), a laptop computer, and the like. / or PC, etc. may include at least one.

The medical device 200 may receive an evaluation result regarding the onset of a disease for an individual from the device 100 for assisting diagnosis, and display the received result through a display unit. Here, the evaluation result may include a disease name classified as having a high risk of onset, a risk of onset of the disease, and an incidence of the disease.

Next, with reference to FIG. 1B, components of the device 100 for assisting diagnosis according to the present invention will be described in detail.

Referring to FIG. 1B , the device 100 for assisting diagnosis includes a storage unit 110 , a communication unit 120 and a processor 130 .

First, the storage unit 110 may store various data generated while evaluating the risk of developing a disease for an individual. For example, the storage unit 110 may be configured to store the name of a disease classified as having a high risk of onset by a machine learning model, the risk of onset of a disease, and even the incidence of a disease. In various embodiments, the storage unit 110 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, magnetic memory , a magnetic disk, and an optical disk may include at least one type of storage medium.

The communication unit 120 connects the device for assisting diagnosis 100 to communicate with an external device. The communication unit 120 is connected to the medical staff device 200 and furthermore the medical data providing device 300 using wired/wireless communication to transmit/receive various data. Specifically, the communication unit 120 may receive medical data of an entity from the medical data providing device 300 . Also, the communication unit 120 may receive an evaluation result of a medical staff member for medical data or receive it through the medical team device 200 . Also, the communication unit 120 may transmit evaluation results to the medical device 200 .

The processor 130 is operatively connected to the storage unit 110 and the communication unit 120, and can execute various commands for analyzing medical data on an entity.

At this time, the processor 130 may be configured to be based on a machine learning model configured to classify into a class corresponding to a specific disease based on medical data, and furthermore, based on evaluation results of medical staff.

Furthermore, the processor 130 can make an ensemble diagnosis based on a plurality of machine learning models and the diagnosis capabilities of medical staff, thereby providing highly accurate and reliable evaluation results for diseases.

Meanwhile, the device 100 for assisting diagnosis is not limited to being designed in terms of hardware. For example, the processor 130 of the device 100 for assisting diagnosis may be implemented as software. Accordingly, the evaluation result for the disease may be displayed through the display unit of the medical data providing device 300 to which the software is connected.

Meanwhile, referring to FIG. 1C together, the medical device 200 includes a communication unit 210, a display unit 220, a storage unit 230, and a processor 240.

The communication unit 210 connects the medical device 200 to enable communication with an external device. The communication unit 210 may be connected to the diagnosis assisting device 100 using wired/wireless communication to transmit various data related to diagnosis of a disease. Specifically, the communication unit 210 transmits an evaluation result related to the diagnosis of a disease of an individual from the device 100 for assisting diagnosis, for example, the name of a disease classified as having a high risk of onset by a machine learning model, the risk of onset of the disease, and further A classification accuracy probability of a disease may be received. The above information may be displayed and provided for medical data, but is not limited thereto.

The display unit 220 may display various interface screens for displaying evaluation results related to the diagnosis of a disease of an individual.

In various embodiments, the display unit 220 may include a touch screen, and for example, a touch using an electronic pen or a part of the user's body, a gesture, a proximity, a drag, or a swipe (swipe) or hovering (hovering) input, etc. can be performed. For example, a medical staff may input evaluation results for medical data through the display unit 220 of the touch screen.

The storage unit 230 may store various data used to provide a user interface for displaying result data. In various embodiments, the storage unit 230 may be a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (for example, SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory) , a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium.

The processor 240 is operatively connected to the communication unit 210, the display unit 220, and the storage unit 230, and can perform various commands to provide a user interface for displaying result data.

Hereinafter, a diagnosis assisting method according to an embodiment of the present invention will be described in detail with reference to FIGS. 2, 3A, and 3D. 2 illustrates a procedure of a diagnosis assisting method according to an embodiment of the present invention. 3A and 3B exemplarily illustrate a procedure for determining a diagnostic evaluation result according to a diagnosis assisting method according to an embodiment of the present invention.

First, referring to FIG. 2 , the procedure of disease evaluation according to an embodiment of the present invention is as follows. First, medical data about an object is received (S210). Then, the diagnostic evaluation result for the medical data is determined by the individual machine learning model (S220). Next, the diagnostic evaluation result determined by the medical staff for the medical data is input (S230). Next, a first classification probability is determined based on the pre-assigned weights for the machine learning model and the diagnostic evaluation result of the machine learning model (S240), and based on the pre-assigned weights for the medical staff and the diagnostic evaluation results determined from the medical staff A second classification probability is determined (S250). Finally, based on the first classification probability and the second classification probability, a third classification probability for the object is determined (S260), and the result is provided (S270). The sequence of steps in FIG. 2 is exemplary, and a step of determining a first classification probability based on a diagnostic evaluation result using an individual machine learning model and a pre-assigned weight; The steps of determining the classification probabilities may be performed in parallel or independently of each other.

More specifically, in the step of receiving medical data (S210), various medical data related to the subject's suspected disease may be received.

According to an embodiment of the present invention, in the step of receiving medical data (S210), the medical data providing device may be driven to acquire medical data in real time.

Next, in the step of determining the diagnostic evaluation result (S220), medical data may be input to the machine learning model, and the risk of developing a specific disease may be determined.

According to a feature of the present invention, in the step of determining the diagnostic evaluation result (S220), the risk of developing a specific disease may be determined probabilistically. That is, in the step of determining the diagnostic evaluation result (S220), the probability that medical data belongs to a class corresponding to a specific disease may be calculated by a machine learning model.

Next, in the step of inputting the diagnostic evaluation result (S230), the classification result for the medical data for the medical staff may be input.

In this case, the classification result may include a binary number according to whether or not the medical data has a specific disease, a probability of belonging to a class corresponding to a specific disease, reliability of classification, and a probability vector.

Meanwhile, as described above, the order of determining the diagnostic evaluation result (S220) and inputting the diagnostic evaluation result (S230) is not limited to the above, and may be performed simultaneously.

Next, in the step of determining the first classification probability (S240), the first classification probability is determined based on the weight assigned to the machine learning model in advance and the diagnostic evaluation result of the machine learning model, and the second classification probability is determined. In step S250, a second classification probability is determined based on the diagnostic evaluation result of the machine learning model and the pre-assigned weight to the medical staff.

According to a feature of the present invention, in the step of determining the first classification probability (S240), classification probabilities for a plurality of diseases multiplied by a predetermined weight for the machine learning model are calculated. Next, in the step of determining the second classification probability (S250), classification probabilities for a plurality of diseases multiplied by a predetermined weight for medical staff may be calculated.

According to another feature of the present invention, in the step of determining the first classification probability (S240), a weight assigned to each of the plurality of machine learning models may be assigned to the diagnosis evaluation result for each of the plurality of machine learning models. . As a result, the weighted first classification probability for each of the plurality of machine learning models may be obtained. Weights will be described later with reference to FIG. 4 , for example.

Meanwhile, in the step of determining the second classification probability (S250), the second classification probability may be determined based on the medical staff's diagnosis evaluation result of the evaluation data and reliability thereof.

Next, in the step of determining the third classification probability ( S260 ), the third classification probability for the object may be determined based on the first classification probability and the second classification probability.

According to a feature of the present invention, in the step of determining the third classification probability (S260), the maximum probability among the classification probabilities of the plurality of diseases determined by the machine learning model and the classification probabilities of the plurality of diseases determined by the medical staff It can be determined that the risk of developing a disease having a high risk. In this case, the third classification probability may include a classification probability value calculated for the suspected disease for which the sum of the weights is maximized and a probability of being classified into a class corresponding to the suspected disease.

For example, referring to FIG. 3A together, in the step of determining the diagnostic evaluation result (S220), the first machine learning model 320 and the second machine learning model 330 for the input medical data 312 Classification probabilities 322 and 332 can be determined. In this case, the medical data 312 may be a medical image, but is not limited thereto. At this time, in step S230 of inputting the diagnostic evaluation result, the classification result 342 of the medical staff 340 for the medical data 312 may be input. At this time, the reliability (or certainty) of the classification of the disease by the medical staff 340 may be further input. Then, in the step of determining the first classification probability (S240), a disease classification predetermined for each of the first machine learning model 320 and the second machine learning model 330 with respect to the classification probabilities 322 and 332 Weight may be reflected. In addition, in the step of determining the second classification probability (S250), the classification probability is determined based on the classification result 342 and reliability of the medical staff 340, and a predetermined classification weight for the medical staff may be reflected. At this time, when the weight is reflected in the classification probabilities 322 and 332 and the classification result 342, a non-linear function may be used. Finally, in the step of determining the third classification probability (S260), among the first classification probability reflecting the classification weight of the machine learning models 320 and 330 and the second classification probability reflecting the classification weight of the medical staff 340, The classification probability 352 of the disease with the greatest risk of developing can be finally determined. More specifically, among classification probabilities for a plurality of diseases, a disease having a maximum probability may be determined to have a high risk of onset.

Referring back to FIG. 2 , according to another feature of the present invention, in step S260 of determining a third classification probability, a third classification probability may be determined based on the first classification probability. That is, the reading result for the machine learning model may be provided without the evaluation result for the medical staff.

At this time, after the step of determining the third classification probability (S260), the relative reliability of the medical staff may be further calculated.

For example, referring to FIG. 3B together, classification probabilities 322 and 332 of the first machine learning model 320 and the second machine learning model 330 for the input medical data 312 may be determined. Next, a predetermined disease classification weight for each of the first machine learning model 320 and the second machine learning model 330 may be reflected for the classification probabilities 322 and 332 . Then, after the first classification probabilities reflecting the classification weights of the machine learning models 320 and 330 are added for each disease, the classification probability 352 of the disease with the maximum risk of onset and the disease can be finally determined. More specifically, among classification probabilities for a plurality of diseases, a disease having a maximum probability may be determined to have a high risk of onset. At this time, the relative reliability of the machine learning models 320 and 330 compared to the medical staff may be further calculated. That is, the medical staff can finally determine the onset disease for the individual based on the relative reliability and classification probability 352 .

Referring back to FIG. 2 , finally, a classification result is provided (S270).

At this time, in the step of providing the classification result (S270), the evaluation result associated with the diagnosis of the disease of the individual, for example, the first classification probability, the second classification probability, and the third onset risk, furthermore, onset by the machine learning model Names of diseases classified as having a high risk, furthermore, classification probabilities of diseases, and the like may be provided.

According to the assisting diagnosis method according to the above various embodiments, an assisting diagnosis system with improved accuracy of disease diagnosis can be provided. Therefore, according to the present invention, by providing the diagnosis assistance system, it is possible to prevent a medical staff from misinterpreting a diagnosis of a disease due to inaccurate measurement, and to improve a medical staff's workflow in actual clinical practice. In addition, although the machine learning model discriminates with low accuracy, the incidence of diseases that medical staff can discriminate more accurately than the machine learning model can be judged with higher accuracy.

Hereinafter, with reference to FIGS. 4 and 5, a procedure for determining the weights for the medical staff and the machine learning model will be described in detail.

4 and 5 exemplarily illustrate a procedure for determining weights in a diagnosis assisting method according to another embodiment of the present invention.

First, referring to FIG. 4 , medical data for evaluation are received (S410), and a machine learning model is trained based on the medical data for evaluation (S420). Next, the diagnostic characteristics of the expert are evaluated (S430), and disease classification weights are determined (S440).

For example, referring to FIG. 5 , the plurality of evaluation medical data 412 for n lung tumors or lung tuberculosis received from the step of receiving evaluation medical data (S410) is used to train a machine learning model. In step S420, the first machine learning model 320 and the second machine learning model 330 are respectively input. In this case, the plurality of medical data 412 for evaluation may be images labeled with respect to a predetermined disease. As a result of this, the first machine learning model 320 and the second machine learning model 330 are evaluated, respectively, and classification probabilities 422 and 432 can be determined.

Then, a step of evaluating classification characteristics of experts (S430) is performed.

) 에 대해 의료진

(340) 는 각 평가용 의료 자료 (412) 를 평가하고, 그 결과 해당 의료진의 분류 행렬 (

) 을 산출할 수 있다. 동일 평가용 자료 (

) 에 대해 각 기계모델 Mj가 N개의 각 자료를 C개의 클래스 중 특정 질환 클래스로 진단한 확률로 구성된 분류 확률 행렬 (

) 이 산출될 수 있다.According to a feature of the present invention, in the step of evaluating the classification characteristics of the human expert (S430), a plurality of medical data 412 for evaluation (correct answers for N data with C classes as correct answers) (

) about the medical staff

340 evaluates each medical data 412 for evaluation, and as a result, the classification matrix of the corresponding medical staff (

) can be calculated. Data for the same evaluation (

), a classification probability matrix composed of the probability that each machine model Mj diagnoses each N data as a specific disease class among C classes (

) can be calculated.

) 를 특정 질환에 대응하는 클래스 (C), 예를 들어 '폐 종양' 또는, '폐 결핵'으로 분류한다. 즉, 의료진 (340) 은 복수의 전문가 평가용 의료 (412) 각각에 대하여 특정 질환인지를 평가한다. 그 다음, 의료진의 분류 결과 (442) 에 따라 의료진의 각 클래스에 대한 분류 신뢰도 (또는, 확실성) (443) 가 결정될 수 있다. More specifically, the medical staff 340 uses the medical data 412 for evaluation by multiple experts (

) is classified into a class (C) corresponding to a specific disease, for example, 'lung tumor' or 'pulmonary tuberculosis'. That is, the medical staff 340 evaluates each of the plural medical items 412 for evaluation by experts as to whether or not they are specific diseases. Then, classification reliability (or certainty) 443 for each class of medical staff may be determined according to the classification result 442 of medical staff.

) 에 의해 결정될 수도 있다. 보다 구체적으로, 의료진의 분류 결과 (442) 인 분류 행렬은 각 클래스에 대한 신뢰도 (443) 를 나타내는 조건 확률 벡터로서 변환될 수 있다. 이때, 해당 클래스 c 가 선택되었을 때 원래는 다른 클래스에 속한 것을 잘못 판단하여 선택될 확률을 고려하여, 의료진 (340) 의 평가 결과인 분류 행렬로부터 추출된 오차 행렬 (confusion matrix) 을 통해 조건부 확률 벡터

가 결정될 수 있다. 그 결과, 분류된 질환 클래스 이외의 다른 질환에서 개연적인 확률값이 그 의료진의 특성에 따라 각 분류 결과물에 할당되고, 복수의 질환 (모든 클래스) 에 대한 개별 신뢰도가 부여될 수 있으며, 이에 기초하여 분류된 질환에 대한 확률 벡터가 결정될 수 있다. 또한, 다양한 실시 예에서, 평가용 의료 자료 (412) 각각에 대한 신뢰도가 결정될 수 있다. 예를 들어, 기계학습모델이 특정 질환에 대해서 오류 또는 오판 가능성이 높으나, 의료진이 해당 질환에 대해서 더 높은 확률로 평가할 수 있는 경우, 해당 평가용 의료 자료 (412) 와 유사한 의료 자료에 대해서 신뢰도를 더 높게 설정할 수 있다.At this time, the classification reliability (Pc) 443 for each class of medical staff is a classification matrix for classification values (

) may be determined by More specifically, the classification matrix, which is the classification result 442 of the medical staff, can be converted into a conditional probability vector representing the reliability 443 for each class. At this time, when the corresponding class c is selected, the conditional probability vector is obtained through a confusion matrix extracted from the classification matrix as a result of evaluation by the medical staff 340, in consideration of the probability of being selected by wrongly judging that it originally belongs to another class.

can be determined. As a result, probable probability values in diseases other than the classified disease classes are assigned to each classification result according to the characteristics of the medical staff, and individual reliability for a plurality of diseases (all classes) can be assigned, based on which probability values are classified. Probabilistic vectors for the affected disease can be determined. Also, in various embodiments, reliability of each of the medical data 412 for evaluation may be determined. For example, if a machine learning model has a high probability of error or misjudgment for a specific disease, but the medical staff can evaluate the disease with a higher probability, reliability is established for medical data similar to the medical data for the evaluation (412). You can set it higher.

) 과 기계학습모델의 평가 신뢰도인 분류 확률 행렬

(보다 구체적으로, 기계학습모델의 소프트맥스 (soft max) 레이어) 에서 나온 각 클래스 별 분류 확률을 가중치 합하여 정답을 최적으로 맞추기 위해 개인별 최적의 가중치를 결정한다. Finally, in the step of determining the disease classification weight (S440), the conditional classification probability (reliability) of the individual corresponding to the currently read class (

) and the classification probability matrix, which is the evaluation reliability of the machine learning model

(More specifically, the weighted sum of the classification probabilities for each class from the softmax layer of the machine learning model) determines the optimal weight for each individual to optimally match the correct answer.

) 을 높은 확률로 맞추는 최적 가중치 (

) 가 결정될 수 있다. 즉, 질환 분류 가중치가 결정되는 단계 (S440) 의 결과로, 기계학습모델 (320 및 330) 과 의료진 (340) 의 복수의 질환, 예를 들어 폐 종양 또는 폐 결핵으로 질환을 분류하는 것에 대한 분류 가중치 (424, 434, 444) 가 각각 결정될 수 있다. At this time, the classification probabilities (422, 432) of the machine learning models (320 and 330) and the probability values by disease (by class) in the classification result (442) of the medical staff (340) are weighted to label the medical data for evaluation (

) with high probability, the optimal weight (

) can be determined. That is, as a result of the step of determining the disease classification weight (S440), the machine learning models 320 and 330 and the medical staff 340 classify the disease into a plurality of diseases, for example, lung tumor or lung tuberculosis. Weights 424, 434 and 444 can be determined respectively.

In this case, a linear function model of Equation 1 may be used as a method of applying the weight to the classification probability.

[Equation 1]

는 의료진을 의미하고, M_i=1:m은 m개의 기계학습모델에 대응할 수 있다.here,

denotes a medical staff, and M _{i = 1:m} may correspond to m machine learning models.

That is, the weight estimated by the above-described method may be applied to each evaluation result of a medical staff member and an already trained machine model for new medical data, and thus a highly reliable final evaluation result may be provided.

In this case, the weight determined for the machine learning model and the medical staff may have an optimal diagnosis capability according to the skill of the medical staff using the machine learning model.

Meanwhile, the determination of the weight is not limited thereto, and the accuracy of diagnosis for classifying a plurality of diseases of each model is evaluated based on the classification result of the plurality of models, and based on the diagnostic ability of each of the plurality of models. Thus, weights of a plurality of disease classifications may be determined.

Evaluation: evaluation of diagnostic assist system according to various embodiments of the present invention

Hereinafter, performance of the diagnostic assist system according to various embodiments of the present disclosure will be described with reference to FIGS. 6A and 6B.

6A to 6B illustrate evaluation results according to a diagnosis assisting method according to an embodiment of the present invention.

First, referring to (a) of FIG. 6A , classification results of medical staff consisting of 9 field experts and 4 general experts for 6 classes of diseases are shown. At this time, the classification accuracy rate for the 6 classes of diseases using the simple ensemble average value is between 0.78 and 0.83, and in particular, the average classification accuracy rate for general experts is lower than that of field experts.

In contrast, referring to (b) of FIG. 6A, when a customized diagnosis assistance method reflecting the skills of individual medical staff is applied, the accuracy of class classification is higher than that of the simple ensemble method ((a) of FIG. 6A). appears to be improved.

In particular, when referring to the increased classification accuracy of FIG. 6B together, the classification accuracy of general experts, who had a low classification accuracy in the classification according to the simple ensemble method, is remarkably marked in the classification according to the diagnosis assisting method according to various embodiments of the present invention. appears to be elevated.

In addition, as the number of samples used for evaluation by medical staff increases, the rate of increase in classification accuracy appears to increase.

The above results may mean that the accuracy and reliability of disease diagnosis are improved by the machine learning model and the diagnosis assistance system based on the machine learning model in which weights are set according to the diagnostic ability of each medical staff.

Accordingly, the present invention provides a diagnostic assistance system capable of machine learning model-human cooperation based on human diagnostic ability evaluation, so that even when the diagnostic ability of a machine learning model is equivalent to or higher than that of an expert, the conventional unreliable conventional can overcome the limitations and problems of the diagnosis assist system.

Although the embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and may be variously modified and implemented without departing from the technical spirit of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. The protection scope of the present invention should be construed according to the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

100: diagnostic aid device
110, 230: storage unit
120, 210: communication department
130, 240: processor
200: medical staff device
220: display unit
300: medical data providing device
312: medical data
320: first machine learning model
330: second machine learning model
322, 332, 352, 422, 432: classification probability
342, 442: classification result
340: medical staff
412 medical data for evaluation
443: classification confidence
424, 434, 444: classification weight

Claims (18)

As a diagnostic aid method implemented by a processor,
receiving medical data about the subject;
determining a diagnostic evaluation result based on the medical data using a machine learning model configured to output a diagnostic evaluation result by taking the medical data as an input;
receiving a diagnostic evaluation result determined from a medical staff for the medical data;
determining a first classification probability based on a pre-assigned weight for the machine learning model and a diagnostic evaluation result of the machine learning model;
determining a second classification probability based on a pre-assigned weight to the medical staff and a diagnostic evaluation result determined from the medical staff; and
And determining a classification probability of a disease having a maximum risk of occurrence among the first classification probability and the second classification probability as a third classification probability for the individual. According to claim 1,
The pre-assigned weight,
A weight given to at least one disease among a plurality of diseases,
receiving medical data for evaluation in which the at least one disease is labeled as a correct disease;
determining a diagnostic evaluation result for the medical data for evaluation using the machine learning model;
receiving a diagnostic evaluation result of the medical staff for the medical data for evaluation;
Determine the classification accuracy of the medical staff and the machine learning model, respectively, based on the diagnosis evaluation result of the machine learning model for the correct disease, the medical data for evaluation, and the medical staff's diagnosis evaluation result for the medical data for evaluation. step of doing, and
Based on the classification accuracy of each, determined through the step of determining the classification weight of the at least one disease for each of the machine learning model and the medical staff, the diagnosis assistance method. According to claim 2,
The step of determining the diagnostic evaluation result for the evaluation medical data,
Determining a classification probability of the at least one disease for the machine learning model based on the medical data for evaluation using the machine learning model;
The step of determining the classification accuracy of the medical staff for the evaluation medical data,
receiving an evaluation result for the classification of the selected disease from the medical staff;
Receiving an input of the evaluation reliability of the selected disease from the medical staff, and
Deriving a classification probability of a disease selected from the medical staff,
Determining the classification accuracy of each of the medical staff and the machine learning model,
Determining classification accuracy of the medical staff and the machine learning model, respectively, based on the correct disease, the classification probability of the at least one disease for the machine learning model, and the classification probability of the disease selected from the medical staff, diagnostic aid method. According to claim 3,
Determining the at least one disease classification weight,
summing the classification probability of the at least one disease for the machine learning model and the classification probability of a disease selected from the medical staff to obtain an aggregated disease classification probability; and
A method for assisting diagnosis, comprising determining a weight at which the summed disease classification probability has a maximum value. According to claim 3,
After the step of receiving the classification reliability for the classification of the selected disease from the medical staff,
Further comprising the step of determining the reliability of the remaining diseases among the plurality of diseases based on the classification reliability for the classification of the diseases selected from the medical staff. According to claim 1,
Each of the diagnostic evaluation results includes a classification probability for each of a plurality of diseases determined from the machine learning model and each of the medical staff,
The first classification probability includes a classification probability for each of a plurality of diseases determined by the weighted machine learning model,
The second classification probability includes classification probabilities for each of a plurality of diseases determined by the medical staff to which weights are added,
Determining the third classification probability,
Among the classification probabilities for each of the plurality of diseases determined by the machine learning model to which the weights are added and the classification probabilities of each of the plurality of diseases determined by the medical staff to which the weights are added, the risk of developing a disease having a maximum value A method for assisting diagnosis comprising determining that is high. delete According to claim 1,
After determining the third classification probability,
Calculating a relative reliability of the medical staff based on the first classification probability and the second classification probability; and
Further comprising the step of providing the relative reliability. According to claim 1,
The machine learning model is plural,
The pre-assigned weight,
It is a weight assigned to each of a plurality of diseases,
determining a diagnostic evaluation result based on the medical data using a plurality of machine learning models;
assigning a weight assigned to each of the plurality of machine learning models to a diagnostic evaluation result for each of the plurality of machine learning models to obtain a first classification probability to which a weight is applied to each of the plurality of machine learning models;
Summing weighted first classification probabilities for each of the plurality of machine learning models so as to obtain an aggregate diagnostic evaluation result for each of the plurality of diseases; and
The method for assisting diagnosis further comprising determining a disease having a maximum aggregate diagnostic evaluation result among the plurality of diseases as a disease having a high risk of developing the subject. As a method for optimizing a machine learning model implemented by a processor,
receiving medical data for evaluation;
learning a machine learning model configured to output a diagnostic evaluation result using medical data as an input using the medical data for evaluation;
Evaluating diagnostic characteristics of medical staff for the medical data for evaluation;
Determining disease classification weights for the machine learning model and each of the medical staff, based on the learning result of the machine learning model for the medical data for evaluation and the diagnostic characteristics of the medical staff; and
Optimizing the machine learning model based on the disease classification weight for each of the above, comprising the step of optimizing the machine learning model optimization method. As a diagnostic aid method implemented by a processor,
receiving medical data for evaluation;
learning a machine learning model configured to output a diagnostic evaluation result using medical data as an input using the medical data for evaluation;
Evaluating diagnostic characteristics of medical staff for the medical data for evaluation;
determining disease classification weights for the machine learning model and each of the medical staff, based on a learning result of the machine learning model for the medical data for evaluation and diagnosis characteristics of the medical staff;
optimizing the machine learning model based on the disease classification weight for each of the diseases;
Receiving actual medical data to be evaluated for the subject;
determining a diagnostic evaluation result based on additional medical data using the machine learning model;
receiving a diagnostic evaluation result determined from a medical staff for the additional medical data;
determining a first classification probability based on weights pre-assigned to the machine learning model and diagnostic evaluation results of the machine learning model;
determining a second classification probability based on a pre-assigned weight to the medical staff and a diagnostic evaluation result determined from the medical staff; and
and determining a classification probability of a disease having a maximum risk of occurrence among the first classification probability and the second classification probability as a third classification probability. a communications unit configured to receive medical data about the entity; and
A processor configured to communicate with the communication unit;
the processor,
determining a diagnostic evaluation result based on the medical data using a machine learning model configured to output a diagnostic evaluation result by taking medical data as an input;
Receive a diagnostic evaluation result determined from the medical staff for the medical data,
Determine a first classification probability based on a pre-assigned weight for the machine learning model and a diagnostic evaluation result of the machine learning model;
determining a second classification probability based on a pre-assigned weight to the medical staff and a diagnostic evaluation result determined from the medical staff;
A device for assisting diagnosis configured to determine a classification probability of a disease having a maximum risk of occurrence among the first classification probability and the second classification probability as a third classification probability for the individual. According to claim 12,
The pre-assigned weight,
A weight given to at least one disease among a plurality of diseases,
The communication department,
Further configured to receive medical data for evaluation in which the at least one disease is labeled as an answer disease,
the processor,
Using the machine learning model, determining a diagnostic evaluation result for the medical data for evaluation,
Receive a diagnostic evaluation result of the medical staff for the medical data for evaluation,
Determine classification accuracy of the medical staff and the machine learning model, respectively, based on the diagnosis evaluation result of the machine learning model for the correct disease, the medical data for evaluation, and the medical staff's diagnosis evaluation result for the medical data for evaluation. do,
Further configured to determine a classification weight of the at least one disease for each of the machine learning model and the medical staff, based on the respective classification accuracy. According to claim 13,
the processor,
Using the machine learning model, based on the medical data for evaluation, determining a classification probability of the at least one disease for the machine learning model;
Receive classification reliability for the classification of the selected disease from the medical staff,
Based on the classification reliability, determining a classification probability of a disease selected from the medical staff;
Receive a classification probability of a selected disease from the medical staff,
Further configured to determine the classification accuracy of the medical staff and the machine learning model, respectively, based on the correct disease, the classification probability of the at least one disease for the machine learning model, and the classification probability of the disease selected from the medical staff. device for. According to claim 14,
the processor,
summing the classification probability of the at least one disease for the machine learning model and the classification probability of a disease selected from the medical staff to obtain a summed disease classification probability;
A device for assisting diagnosis, further configured to determine a weight that maximizes accuracy for a correct answer based on the summed disease classification probability. According to claim 14,
the processor,
The device for assisting diagnosis is further configured to determine the reliability of the remaining diseases among the plurality of diseases based on the classification reliability for the classification of the disease selected from the medical staff. According to claim 12,
Each of the diagnostic evaluation results includes a classification probability for each of a plurality of diseases determined from the machine learning model and each of the medical staff,
The first classification probability includes classification probabilities for a plurality of diseases determined by the machine learning model to which weights are added,
The second classification probability includes classification probabilities for a plurality of diseases determined by the medical staff to which weights have been added,
the processor,
Among the classification probabilities for each of the plurality of diseases determined by the machine learning model to which the weights are added and the classification probabilities for each of the plurality of diseases determined by the medical staff to which the weights are added, for a disease having a maximum probability A device for assisting diagnosis, further configured to determine a high probability of onset. delete

Images