KR20230120992A - Method and apparatus for selecting medical data for annotation - Google Patents

Abstract

A method of operating a medical data screening device operated by at least one processor comprises the steps of: generating some medical data sampled from bulk medical data and annotation data of the partial medical data as training data; extracting annotation candidate data that is at least part of the mass medical data; and obtaining an inference result for the annotation candidate data from an artificial intelligence model trained using the training data, and selecting an annotation target for the next training of the artificial intelligence model from the annotation candidate data based on the inference result. Accordingly, the present invention may increase artificial intelligence model training performance compared to annotation cost.

Description

Method and apparatus for selecting medical data for annotation {METHOD AND APPARATUS FOR SELECTING MEDICAL DATA FOR ANNOTATION}

This disclosure relates to annotations.

An artificial intelligence model can be trained through a large amount of medical images and their annotated data. Annotation refers to a task of tagging required information or tagged annotation data in order to generate training data of an artificial intelligence model. In particular, in the case of medical data, since annotation is performed by a skilled expert, considerable cost and time are required.

Training performance can be improved as the amount of annotation data increases, but since the annotation budget is limited, it is necessary to select an annotation target from all medical data. Usually, some data randomly extracted from the entire medical data is selected as an annotation target, but it is difficult to see that this randomly extracted annotation target is the optimal data for training an artificial intelligence model.

The present disclosure provides a method and apparatus for screening medical data for annotation.

The present disclosure provides a method and apparatus for selecting medical data for annotation from a large amount of medical data by determining a medical data selection policy based on the training performance of a current artificial intelligence model and selecting an annotation target for next training according to the selection policy. provides

A method of operating a medical data sorting apparatus operated by at least one processor according to an embodiment, comprising: generating training data from partial medical data sampled from a large amount of medical data and annotation data of the partial medical data; Extracting annotation candidate data, which is at least a part of data, and acquiring an inference result for the annotation candidate data from an artificial intelligence model trained through the training data, and based on the inference result, the artificial intelligence among the annotation candidate data Selecting an annotation target for the next training of the model.

In the step of selecting an annotation target, a selection policy of medical data for the next training is determined based on the training performance of the artificial intelligence model, and an annotation target corresponding to the selection policy is selected from among the annotation candidate data based on the reasoning result. can be selected.

The selection policy may be a policy of extracting medical data of a specific class whose classification accuracy is lower than a standard at a higher rate than other classes.

The operation method may further include extracting verification data that is at least a part of the mass medical data, and evaluating training performance of the artificial intelligence model based on the verification data.

In the selecting of the annotation target, medical data randomly extracted from the mass medical data may be added to the annotation target.

In the step of selecting an annotation target, an inference result for the annotation candidate data is obtained from an artificial intelligence model trained through the training data, and medical data related to a specific class is selected from the annotation candidate data based on the inference result. can do.

The generating of the training data may include analyzing a reading report associated with the mass medical data, sampling the partial medical data including information related to training of the artificial intelligence model in the reading report, and annotating the partial medical data. The training data may be generated by obtaining data.

The mass medical data may include images acquired through at least one medical imaging device, pathology images, or patch images extracted from each medical image.

A method of operating an apparatus for selecting medical data operated by at least one processor according to an embodiment, determining a policy for selecting medical data based on training performance of a current artificial intelligence model, and determining the current artificial intelligence for annotation candidate data. Based on the inference result of the model, after selecting an annotation target corresponding to the selection policy from among the annotation candidate data for next training, training the current artificial intelligence model based on the annotation data obtained for the annotation target Repeating the process, and terminating the process when the next training of the current artificial intelligence model is not performed. The annotation candidate data may be data of at least a part of mass medical data.

The repeating step may extract verification data, which is at least a part of the mass medical data, and evaluate training performance of the current artificial intelligence model based on the verification data.

In the repeating step, medical data randomly extracted from the mass medical data may be added to the annotation target.

In the repeating step, the selection policy may be determined as before or differently according to the training performance of the current artificial intelligence model.

The current artificial intelligence model may be an initial model trained based on some medical data sampled from the mass medical data and annotation data of the partial medical data, or the initial model may be a model retrained through the above process.

The mass medical data may include images acquired through at least one medical imaging device, pathology images, or patch images extracted from each medical image.

An apparatus for selecting medical data operated by at least one processor according to an embodiment, comprising an artificial intelligence model trained to output a result inferred from an input based on training data, and annotation candidate data that is at least a part of mass medical data. and a selector that selects an annotation target for next training of the artificial intelligence model from among the annotation candidate data based on the inference result. Annotation data obtained for the annotation target for the next training may be used for the next training of the artificial intelligence model.

The selector may repeat the process of selecting the annotation target for the next training based on the reasoning result of the artificial intelligence model when the next training of the artificial intelligence model is required.

The selector may determine a selection policy of medical data for the next training based on the training performance of the artificial intelligence model, and select an annotation target corresponding to the selection policy from among the annotation candidate data based on the reasoning result. .

The selector may determine the same or different selection policy according to the training performance of the artificial intelligence model.

The selector may add medical data randomly extracted from the mass medical data to the annotation target.

The mass medical data may include images acquired through at least one medical imaging device, pathology images, or patch images extracted from each medical image.

According to the embodiment, it is possible to increase AI model training performance against annotation cost by intelligently selecting an annotation target from a large amount of medical data.

According to the embodiment, the training performance of the artificial intelligence model may be improved compared to training data in which some data randomly extracted from all medical data is annotated.

According to the embodiment, some patches requiring annotation may be selected from a large image, such as a pathology image through whole slide imaging (WSI).

1 is a diagram illustrating an apparatus for selecting medical data for annotation according to an exemplary embodiment.
2 is a diagram illustrating training through step-by-step annotation target selection according to an embodiment.
3 to 6 are flowcharts of a method for selecting medical data for annotation according to an embodiment.
7 and 8 are respective flowcharts of a method for selecting a patch image for annotation according to an exemplary embodiment.
9 is a hardware configuration diagram of an annotation screening device according to an embodiment.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly describe the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In the description, when a part is said to "include" a certain component, it means that it may further include other components without excluding other components unless otherwise stated. In addition, terms such as “… unit”, “… unit”, and “module” described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. there is.

In addition, terms including ordinal numbers such as “first” or “second” used in the description may be used to describe various elements, but the elements should not be limited by the terms. The terms may be used for the purpose of distinguishing one component from another.

An apparatus of the present disclosure is a computing device configured and connected so that at least one processor can perform the operations of the present disclosure by executing instructions. The computer program includes instructions described to cause a processor to execute the operations of the present disclosure, and may be stored in a non-transitory computer readable storage medium. The computer program may be downloaded through a network or sold in the form of a product.

Annotation may refer to a task such as displaying information or adding information to an annotation target. Alternatively, annotated data displayed or added to the annotation target may be referred to as annotation.

An expert who performs annotation work is called an annotator. Annotators need not necessarily be humans, but can be artificial intelligence models that have learned to annotate tasks.

An annotation target may be various medical data, and a medical image may be described as an example. Medical images are taken with various medical imaging equipment such as X-ray, magnetic resonance imaging (MRI), ultrasound (ultrasound), computed tomography (CT), mammography (MMG), and digital breast tomosynthesis (DBT). It may be an image or a pathological image. Pathological images may be acquired through Whole Slide Imaging (WSI). Also, at least some patches of the medical image may be an annotation target.

FIG. 1 is a diagram illustrating an apparatus for selecting medical data for annotation according to an embodiment, and FIG. 2 is a diagram illustrating training through step-by-step annotation target selection according to an embodiment.

Referring to FIG. 1 , an apparatus for selecting medical data for annotation (simply, a sorting apparatus) 10 is a computing device operated by at least one processor. The screening device 10 selects an annotation target on which an annotation job is to be performed for training of the artificial intelligence model 100 in the database 20 in which a large amount of medical data is stored. The selection device 10 determines a selection policy based on the training performance of the current AI model (Artificial Intelligence model, AI model) 100 in order to increase the AI model training performance against the annotation cost, and according to the determined selection policy By selecting annotation targets for the next training, annotation targets can be intelligently selected from large amounts of medical data. The selection policy may include selection conditions and may further include the number of annotation targets to be selected. The selection policy may be determined the same as before or differently depending on the training performance of the current AI model.

The selection device 10 includes a trainer 200 for training the artificial intelligence model 100, and a selector 300 for selecting an annotation target for the next training from annotation candidate data based on the inference result of the current artificial intelligence model 100. ) may be included. The screening device 10 may further include an annotation task processor 400 that requests an annotation task for an annotation target from an annotator and obtains annotation data for the annotation target. Annotation data for an annotation target is stored in the database 20 and can be used as training data for the next training.

The database 20 may include various types of medical data, and training data or annotation candidate data may be determined in the database 20 according to tasks of the artificial intelligence model 100 . The database 20 may include images obtained through at least one medical imaging device, pathology images, or patch images extracted from each medical image.

The artificial intelligence model 100 is a machine learning model that learns at least one task, and may be implemented as a computer program executed by a processor. The task to be learned by the artificial intelligence model may refer to a task to be solved through machine learning or a task to be performed through machine learning. The artificial intelligence model can be implemented as a computer program running on a computing device, downloaded through a network, or sold in the form of a product. Alternatively, the artificial intelligence model can work with various devices through a network.

The artificial intelligence model 100 is a model that outputs a result inferred from a new input by learning a relationship between an input and an output in a training process, for example, to receive a medical image and make medical inference included in the medical image It can be a trained model. Medical reasoning can be defined in various ways according to inputs and tasks.

The task of the artificial intelligence model 100 is, for example, a task of detecting or separating biological objects such as cells and tissues from medical images, or specific information (eg, immune phenotype (IP)) from medical images. , CPS (Combined positive score), TPS (Tumor Proportion Score), expression level for a specific dye, response to a specific treatment, etc.) may be a task of inferring.

The information that the artificial intelligence model 100 can infer from the pathology image may be diverse. For example, the artificial intelligence model 100 may detect a biological object included in a pathology image. The artificial intelligence model 100 may mark detected biological objects by drawing contours, bounding boxes, or dots in the center of cells of all types. The artificial intelligence model 100 may segment a biological object included in the pathology image. The artificial intelligence model 100 may recognize a region corresponding to a specific biological object in a pathology image in units of pixels. Here, the biological object may be cells such as cancer cells, normal cells, mitosis cells, immune cells, and fibroblasts, and all biological objects such as blood vessels, organs, nerves, cancer lesions, and tumor stroma. can include In addition, the artificial intelligence model 100 may infer image quality information (for example, whether there is blur, whether a contaminant is present, whether a tissue is folded, etc.). The artificial intelligence model 100 may infer whether the image includes a region of interest (eg, a cell region, a background region, a cancer region, etc.). The artificial intelligence model 100 may infer immune-related information (eg, immune phenotype classified into immune inflamed tumors, immune desert, immune excluded, etc.) from an image. The artificial intelligence model 100 may infer a score such as TPS or CPS of an image. The artificial intelligence model 100 may infer the expression level of a specific dye (eg, H&E, IHC, etc.) in an image. In order for the artificial intelligence model 100 to perform these various inferences, medical data annotated with correct answers to be inferred must be used as training data. To this end, the selector 300 selects an annotation target by determining which medical data to annotate will be helpful for training.

Similarly, information that can be inferred from images acquired by the artificial intelligence model 100 through medical imaging equipment such as X-ray, CT (Computed Tomography), MMG (Mammography), DBT (Digital Breast Tomosynthesis), etc. may vary. can For example, the artificial intelligence model 100 classifies an object in an image, draws a contour on a detected object, draws a bounding box on an object, or draws a dot in the center of an object. can be drawn, or classes can be classified in units of pixels. The artificial intelligence model 100 may detect a line such as an E-tube or an L-tube in an image, or calculate a probability corresponding to a class in units of pixels and display it as a heat map. Here, the class may be defined in various ways depending on the information to be inferred from the image. It may be an organ such as a carina or a diaphragm, or a medical device such as an E-tube or L-tube. For another example, the artificial intelligence model 100 may receive an image acquired through mammography such as MMT or DBT, classify a lesion in the breast as a target object, or calculate breast density or the possibility of breast cancer occurring in the future. . In order for the artificial intelligence model 100 to perform these various inferences, medical images annotated with classes corresponding to correct answers must be used as training data.

The trainer 200 trains the artificial intelligence model 100 based on training data. Training data may include medical data in which annotation data is mapped to ground truth. The current training data may include an annotation target selected for retraining of the artificial intelligence model 100 trained with previous training data and annotation data thereof. The artificial intelligence model 100 may learn to infer a result close to the annotation data, which is the correct answer, from an input based on current training data. In this case, some of the training data to which the annotation data is mapped may be used as validation data for evaluating training performance. Meanwhile, the artificial intelligence model 100 may perform supervised learning with medical data to which annotation data is mapped according to a learning method, or may learn by using medical data to which annotation data is not mapped as training data.

Meanwhile, the initial training data may include some data sampled from mass medical data and annotation data thereof. For example, for initial training data, the selector 300 may sample 1000 medical images from among 100,000 medical images and determine the 1000 medical images as annotation targets. Thereafter, the annotation task processor 400 obtains annotation data for 1000 medical images and stores them in the database 20, so that the trainer 200 can use the 1000 medical images to which the annotation data is mapped as initial training data. there is.

There may be various methods of sampling annotation targets for initial training data. For example, the selector 300 may randomly sample 1000 medical images from among 100,000 medical images. Alternatively, the sorter 300 analyzes a reading report associated with 100,000 medical images, and includes information related to training of the artificial intelligence model 100 (eg, a class name to be classified) in the reading report. Images can be sampled. In addition, sampling methods may vary, and if medical images to be used as initial training data are predetermined, annotation data for corresponding medical images may be obtained without sampling.

The artificial intelligence model 100 trained with training data may output various inference results according to inputs and tasks. For example, the inference result may include a class representing the cell type included in the image, a class representing the tissue type included in the image, a class representing the quality of the image or an artifact affecting the quality, and a class indicating whether the region in the image is a region of interest. It may be a class, a class representing immunity-related information, a class representing scores such as TPS or CPS, and a class representing the expression level of a specific dye. In addition, the inference result is a class representing the type of lesion (eg, pneumothorax, pulmonary nodule, lung cancer, tuberculosis, fracture, pneumonia, atelectasis, pulmonary consolidation, benign tumor in the breast, malignant tumor in the breast, breast cancer lesion, etc.), organ It may be a class representing a type (eg, lung, stomach, esophagus, carina, diaphragm, etc.), a class representing a type of medical device, a class representing breast density, or a class representing the possibility of developing breast cancer in the future. .

The selector 300 selects an annotation target for next training from annotation candidate data based on the training performance of the current artificial intelligence model 100 . The selector 300 may acquire training performance of the current artificial intelligence model 100 and determine an annotation target selection policy for next training data based on the training performance. At this time, the selector 300 may determine the current selection policy equal to or different from the previous selection policy according to the training performance of the current artificial intelligence model 100 . That is, since the selector 300 selects an annotation target for the next training data based on the selection policy determined according to the training performance of the current artificial intelligence model 100, the current artificial intelligence model 100 is used for training so far. The lack of learning in the training can be re-learned in the next training.

The training performance of the current artificial intelligence model 100 can be evaluated through validation data remaining among the current training data used for training the artificial intelligence model 100, or the correct answer value is annotated in mass medical data. It can be evaluated using the data as validation data. The selector 300 may evaluate the training performance of the current artificial intelligence model 100 by comparing the inference result of the current artificial intelligence model 100 on the verification data with the annotation data that is the correct answer. For example, if the current artificial intelligence model 100 is a classification model, the selector 300 may evaluate training performance based on classification accuracy for each class. In addition, when the classification accuracy of the specific class is below the standard, the selector 300 selects medical data related to the specific class so that the current AI model 100 can classify the specific class with poor classification accuracy in the next training. It is possible to determine the selection policy for extraction.

In order to extract medical data related to a specific class and obtain an annotation later, an output value obtained as a result of inference by the artificial intelligence model 100 may be used. For example, the sorter 300 sorts the medical data based on the output value of the artificial intelligence model 100 corresponding to the class A in order to extract the medical data related to the class A according to the determined selection policy. can do. The sorter 300 may select medical data having an output value equal to or greater than a specific cut-off from the sorted medical data. The selector 300 selects medical data having an output value equal to or greater than a specific cut-off, such that medical data having a relatively high probability that an object corresponding to class A actually exists or medical data having a high probability actually corresponding to class A can select them.

The sorter 300 may use variously determined sorting policies. For example, the sorter 300 may use a sorting policy 1 that extracts medical data of a specific class whose classification accuracy of the current artificial intelligence model 100 is below a standard at a higher rate than other classes. That is, in the sorter 300, the sorting policy 1 may include an optimal ratio of classes to be extracted to improve the classification accuracy of the current artificial intelligence model 100. The selector 300 may use the selection policy 2 for extracting medical data having at least one class score higher than or equal to a standard among annotation candidate data. For example, in the selection policy 2, a class may represent at least one of a cell, tissue, or structure to be annotated, and a class score may be a probability of being included in the corresponding class. The selector 300 may use a selection policy 3 for extracting medical data not including one or more classes from annotation candidate data. The sorter 300 may use the sorting policy 4 for extracting medical data in which all defined classes are evenly distributed among annotation candidate data. The selector 300 may use a selection policy 5 for extracting medical data having a predicted entropy of the artificial intelligence model 100 equal to or greater than a reference value from among annotation candidate data. When the artificial intelligence model 100 performs object detection or object separation, the sorter 300 may use the sorting policy 6 for extracting medical data whose ratio between classes satisfies the reference condition. The ratio between classes may be set differently according to the training performance and task of the current artificial intelligence model 100 . In the case of object detection, the ratio between classes may be set based on the number of objects (eg, cells, etc.) for each class, and in the case of object separation, based on the area of each class (eg, cancer region, etc.) A ratio between classes can be set. Also, the sorter 300 may use a new sorting policy that is a combination of the sorting policies.

Based on the selection policy, the selector 300 may select an annotation target for the next training from annotation candidate data. A selection policy may be determined according to training performance among various selection policies.

The selector 300 may obtain an inference result for annotation candidate data through the artificial intelligence model 100, and select medical data corresponding to a selection policy as an annotation target based on the inference result of the annotation candidate data.

The annotation target may be selected in units of images or in units of patches constituting an image. For example, when the annotation candidate data is pathology images, the selector 300 obtains inference results for the pathology images or a plurality of patches constituting each pathology image, and based on the inference results, the sorter 300 corresponding to the selection policy. Pathological images or patches may be determined as targets for annotation. For example, since a pathology image generated through whole slide imaging (WSI) is very large, the sorter 300 may extract patches corresponding to a region of interest from at least one pathology image and extract the patches as annotation candidate data. there is. In this case, the selector 300 may extract patches such that the region of interest overlaps, or may extract patches such that there is no overlapping region. The selector 300 may obtain inference results of patches through the artificial intelligence model 100 and select patches requiring annotation as annotation targets based on the inference results of the patches.

All data to which annotation data is not mapped in the database 20 may be annotation candidate data, or a certain amount of data among all data to which annotation data is not mapped in the database 20 may be annotation candidate data. For example, the annotation candidate data for secondary training data may be 99,000 medical images remaining after being used as primary training data among 100,000 medical images, or a certain amount sampled from among 99,000 (e.g., For example, 10,000 medical images). When the number of annotation targets to be screened is determined, the selector 300 randomly extracts K pieces of medical data corresponding to the selection policy or extracts the top K items satisfying a specific condition from among the medical data corresponding to the selection policy. can

For example, when the selector 300 determines to use screening policy 1 that extracts medical data of a specific class at a higher rate than other classes, medical data inferred as a specific class based on a result of reasoning on annotation candidate data. Data can be screened over inferred medical data into different classes.

The sorter 300 may request an annotation job for an annotation target to the annotation job processor 400 . Meanwhile, the selector 300 selects an annotation target based on a selection policy, but may include a predetermined number of randomly extracted medical data in the annotation target. Randomly extracted annotation objects have various distributions, which can be advantageous for model learning.

The annotation job processor 400 may request an annotation job for an annotation target from an annotator, obtain annotation data for the annotation target, and store the annotation data for the annotation target in the database 20 .

When annotation data for the annotation target selected for the next training is obtained, the trainer 200 retrains the artificial intelligence model 100 based on the new training data. The trainer 200 may retrain the artificial intelligence model 100 by using both new training data and existing training data.

Referring to FIG. 2 , the selection device 10 determines a selection policy of medical data based on the training performance of the current artificial intelligence model 100, and selects an annotation target for the next training according to the selection policy. It may be repeated until the training performance of the artificial intelligence model 100 exceeds the standard. The selection device 10 evaluates the training performance of the current artificial intelligence model 100, and if the next training is required, selects an annotation target based on the training performance, and then uses the annotation target and its annotation data to determine the current training performance. In retraining the artificial intelligence model 100, the total number of training times N is adjusted according to the training performance, and the number of annotation targets for each training may also be adjusted according to the training performance. The screening device 10 evaluates the training performance of the current artificial intelligence model 100, and may complete training of the artificial intelligence model 100 when the next training is unnecessary.

In this way, the screening device 10 does not select annotation targets (eg, 10,000 sheets out of 100,000 sheets) at once from a large amount of medical data, but artificially trained with primary training data (eg, 1,000 sheets out of 100,000 sheets). Based on the training performance of the intelligence model 100, a schedule of annotation targets to be used for secondary training (eg, 10,000 annotations out of 100,000 annotations) may be customized. And the screening device 10 determines whether tertiary training is necessary based on the training performance of the retrained artificial intelligence model 100 using the annotation determined to be used for the 2nd training, and determines the annotation schedule to be used for the 3rd training. The process of selecting an object (for example, 20,000 sheets out of 100,000 sheets) is performed step by step. The selection device 10 is configured to select a large amount of training data based on a result of training the artificial intelligence model 100 using a small amount of data primarily before performing training using a large amount of training data. selection policy can be determined. Therefore, the selection device 10 can efficiently select an annotation target required to improve the training performance of the current artificial intelligence model 100, and can increase the training performance of the artificial intelligence model compared to the annotation cost.

Each of FIGS. 3 to 6 is a flowchart of a method for selecting medical data for annotation according to an embodiment.

Referring to FIG. 3 , the screening device 10 generates some medical data sampled from a large amount of medical data and annotation data thereof as initial training data (S110). There may be various methods of sampling annotation targets for initial training data. For example, the selector 300 may randomly sample 1000 medical images from among 100,000 medical images. Alternatively, the sorter 300 analyzes a reading report associated with 100,000 medical images, and includes information related to training of the artificial intelligence model 100 (eg, a class name to be classified) in the reading report. Images can be sampled.

The screening device 10 trains the artificial intelligence model 100 to output a result inferred from an input through initial training data (S120). Annotation data may be added so that the artificial intelligence model 100 outputs a desired inference result from medical data. For example, the inference result may include a class representing the cell type included in the image, a class representing the tissue type included in the image, a class representing the quality of the image or an artifact affecting the quality, and a class indicating whether the region in the image is a region of interest. It may be a class, a class representing immunity-related information, a class representing scores such as TPS or CPS, and a class representing the expression level of a specific dye. In addition, the inference result is a class representing the type of lesion (eg, pneumothorax, pulmonary nodule, lung cancer, tuberculosis, fracture, pneumonia, atelectasis, pulmonary consolidation, benign tumor in the breast, malignant tumor in the breast, breast cancer lesion, etc.), organ It may be a class representing a type (eg, lung, stomach, esophagus, carina, diaphragm, etc.), a class representing a type of medical device, a class representing breast density, or a class representing the possibility of developing breast cancer in the future. .

The sorting device 10 extracts annotation candidate data, which is at least a part of mass medical data (S130).

The selection device 10 selects an annotation target for the next training of the artificial intelligence model from the annotation candidate data based on the inference result of the artificial intelligence model 100 on the annotation candidate data (S140). The selection device 10 determines a selection policy of medical data for the next training based on the training performance of the current artificial intelligence model 100, and selects an annotation target corresponding to the selection policy from among annotation candidate data based on the inference result. can be selected Meanwhile, the selection device 10 selects an annotation target based on a selection policy, but may include a predetermined number of randomly extracted medical data in the annotation target. At this time, the selection device 10 may determine the current selection policy to be the same as or different from the previous selection policy according to the training performance of the artificial intelligence model 100 .

The sorting device 10 acquires annotation data for an annotation target for the next training (S150).

The selection device 10 trains the artificial intelligence model 100 by using the annotation target to which the annotation data is mapped as training data (S160). If annotation data for the next training of the artificial intelligence model 100 is required, the selection device 10 may move to the annotation target selection step (S130) for the next training and repeat the annotation target selection.

Referring to FIG. 4 , the selection device 10 determines a selection policy of medical data based on the training performance of the current artificial intelligence model 100 (S210). The selection policy may be variously determined according to the training performance of the current artificial intelligence model 100 and tasks. For example, when the classification accuracy of a specific class is below the standard, the sorting device 10 is configured to perform classification related to the specific class so that the current artificial intelligence model 100 can classify the specific class with poor classification accuracy in the next training. A screening policy for extracting medical data can be determined. Meanwhile, the selection policy may include the number of annotation targets to be selected. At this time, the selection device 10 may determine the current selection policy to be the same as or different from the previous selection policy according to the training performance of the artificial intelligence model 100 .

The selection device 10 selects medical data corresponding to a selection policy from annotation candidate data as an annotation target for the next training, based on the inference result of the artificial intelligence model 100 on the annotation candidate data (S220). The screening device 10 may obtain an inference result for annotation candidate data through the artificial intelligence model 100, and select medical data corresponding to a selection policy as an annotation target based on the inference result of the annotation candidate data. . In the mass medical data, all data to which annotation data is not mapped may be annotation candidate data, or a certain amount of data among all data to which annotation data is not mapped may be annotation candidate data. Meanwhile, the selection device 10 selects an annotation target based on a selection policy, but may include a predetermined number of randomly extracted medical data in the annotation target.

The screening device 10 acquires annotation data for an annotation target, and trains the artificial intelligence model 100 based on new training data including an annotation target to which the annotation data is mapped (S230). The screening device 10 may retrain the artificial intelligence model 100 by using both new training data and existing training data. The artificial intelligence model 100 may perform supervised learning with medical data to which annotation data is mapped according to a learning method, or may perform unsupervised learning by using medical data to which annotation data is not mapped as training data.

The selection device 10 evaluates the training performance of the trained artificial intelligence model 100 based on the verification data (S240). The selection device 10 may obtain a result of the artificial intelligence model 100 reasoning on the verification data, and evaluate training performance by comparing the inference result with annotation data (correct answer) mapped to the verification data.

If the next training is not performed according to the evaluation result, the selection device 10 ends the annotation target selection procedure (S250). When the next training is performed according to the evaluation result, the selection device 10 may move to the annotation target selection step (S210) for the next training and repeat the annotation target selection and training. When the training performance of the artificial intelligence model 100 is less than the standard, the screening device 10 may determine that the next training is necessary.

Referring to FIG. 5 , the selection device 10 determines a selection policy of medical data based on the training performance of the current artificial intelligence model 100 (S310).

The selection device 10 selects an annotation target for the next training from annotation candidate data according to a selection policy based on the inference result of the artificial intelligence model 100 on the annotation candidate data (S320). Meanwhile, the selection device 10 selects an annotation target based on a selection policy, but may include a predetermined number of randomly extracted medical data in the annotation target.

The sorting device 10 requests an annotation work for a new annotation target (S330).

The selection device 10 obtains an inference result for verification data from the artificial intelligence model 100 trained on the basis of the result of the annotation task for the new annotation target (S340).

The screening device 10 evaluates training performance by comparing the inference result with the annotation data (correct answer) mapped to the verification data (S350).

According to the evaluation result, the selection device 10 ends the procedure for selecting an annotation target for the artificial intelligence model 100 or repeats the procedure for selecting an annotation target for the next training of the artificial intelligence model 100 (S360).

Referring to FIG. 6 , the selection device 10 generates some data sampled from a large amount of medical data and annotation data thereof as initial training data (S410). There may be various methods of sampling annotation targets for initial training data. For example, the selector 300 may randomly sample 1000 medical images from among 100,000 medical images. Alternatively, the sorter 300 analyzes a reading report associated with 100,000 medical images, and includes information related to training of the artificial intelligence model 100 (eg, a class name to be classified) in the reading report. Images can be sampled.

The screening device 10 trains the artificial intelligence model 100 to output an inference result from an input through initial training data (S420). Annotation data may be added so that the artificial intelligence model 100 outputs a desired inference result from medical data. For example, the inference result may include a class representing the cell type included in the image, a class representing the tissue type included in the image, a class representing the quality of the image or an artifact affecting the quality, and a class indicating whether the region in the image is a region of interest. It may be a class, a class representing immunity-related information, a class representing scores such as TPS or CPS, and a class representing the expression level of a specific dye. In addition, the inference result is a class representing the type of lesion (eg, pneumothorax, pulmonary nodule, lung cancer, tuberculosis, fracture, pneumonia, atelectasis, pulmonary consolidation, benign tumor in the breast, malignant tumor in the breast, breast cancer lesion, etc.), organ It may be a class representing a type (eg, lung, stomach, esophagus, carina, diaphragm, etc.), a class representing a type of medical device, a class representing breast density, or a class representing the possibility of developing breast cancer in the future. .

The selection device 10 evaluates the training performance of the trained artificial intelligence model 100 based on the verification data (S430). The verification data may be data generated to evaluate training performance of the artificial intelligence model 100 . Verification data may be data extracted from mass medical data. Verification data may be included in some data sampled in step S410. The selection device 10 may obtain a result of the artificial intelligence model 100 reasoning on the verification data, and evaluate training performance by comparing the inference result with annotation data (correct answer) mapped to the verification data.

The selection device 10 determines a selection policy of medical data for the next training based on the training performance of the current artificial intelligence model 100 (S440).

The selection device 10 selects an annotation target for the next training from annotation candidate data according to a selection policy based on the inference result of the artificial intelligence model 100 for annotation candidate data (S450). Meanwhile, the selection device 10 selects an annotation target based on a selection policy, but may include a predetermined number of randomly extracted medical data in the annotation target.

The screening device 10 acquires annotation data for an annotation target for the next training, and trains the artificial intelligence model 100 based on new training data including an annotation target to which the annotation data is mapped (S460).

The selection device 10 evaluates the training performance of the trained artificial intelligence model 100 (S470). The selection device 10 may obtain a result of the artificial intelligence model 100 reasoning on the verification data, and evaluate training performance by comparing the inference result with annotation data (correct answer) mapped to the verification data.

If the next training is not performed according to the evaluation result, the selection device 10 ends the annotation target selection procedure (S480). When the next training is performed according to the evaluation result, the selection device 10 may move to the annotation target selection step (S440) for the next training and repeat the annotation target selection and training.

7 and 8 are respective flowcharts of a method for selecting a patch image for annotation according to an exemplary embodiment.

Referring to FIG. 7 , the screening device 10 extracts patches from at least one medical image (S510). The screening device 10 may extract patches so that the region of interest overlaps, or may extract patches so that the region of interest does not overlap. The medical image may be, for example, a pathology image generated through WSI, but is not limited thereto.

The selection device 10 selects some patches corresponding to the selection policy among patches as annotation targets based on the inference result of the artificial intelligence model 100 for the patches (S520). Meanwhile, the selection device 10 selects an annotation target based on a selection policy, but may include randomly extracted patches in the annotation target.

The screening device 10 acquires annotation data for the selected patches and trains the artificial intelligence model 100 based on the patches to which the annotation data is mapped (S530).

The selection device 10 determines whether re-learning of the trained artificial intelligence model 100 is necessary (S540).

If re-learning is unnecessary, the selection device 10 ends the procedure for selecting an annotation target for the artificial intelligence model, and repeats the procedure for selecting an annotation target for the next training of the artificial intelligence model if re-learning is required (S550).

Referring to FIG. 8 , the screening device 10 obtains an inference result for at least one medical image by using an artificial intelligence model trained to output a result inferred from an input image (S610). The medical image may be, for example, a pathology image generated through WSI, but is not limited thereto.

The screening device 10 extracts patches from each medical image (S620). The screening device 10 may extract patches so that the region of interest overlaps, or may extract patches so that the region of interest does not overlap.

The selection device 10 maps an inference result corresponding to each patch based on the inference result for each medical image, and selects some patches corresponding to a selection policy among patches as annotation targets based on the inference result of each patch. Do (S630). Meanwhile, the selection device 10 selects an annotation target based on a selection policy, but may include randomly extracted patches in the annotation target.

In this case, the screening device 10 may train an artificial intelligence model that outputs an inference result from the patch image using the inference result mapped to each patch. Also, the selection device 10 may obtain annotation data for the selected patches and retrain the artificial intelligence model based on the patches to which the annotation data is mapped. As described above, the screening device 10 selects an annotation target for the next training based on the inference result of the artificial intelligence model for patches when re-learning the artificial intelligence model is required, and performs a procedure of training based on this. can be repeated

9 is a hardware configuration diagram of an annotation screening device according to an embodiment.

Referring to FIG. 9 , the sorting device 10 is a computing device operated by one or more processors 11, and includes a processor 11, a memory 13 for loading a computer program executed by the processor 11, and a computer. It may include a storage 15 for storing programs and various data, a communication interface 17, and a bus 19 connecting them. In addition, the sorting device 10 may further include various components.

The computer program may include instructions that, when loaded into memory 13, cause processor 11 to perform methods/operations in accordance with various embodiments of the present disclosure. That is, the processor 11 may perform methods/operations according to various embodiments of the present disclosure by executing instructions. A computer program consists of a series of computer readable instructions grouped together on a functional basis and is executed by a processor. A computer program may include instructions for performing the operations described in this disclosure.

The processor 11 controls the overall operation of each component of the annotation screening device 10 . The processor 11 may include at least one of a Central Processing Unit (CPU), a Micro Processor Unit (MPU), a Micro Controller Unit (MCU), a Graphic Processing Unit (GPU), or any type of processor well known in the art of the present disclosure. can be configured to include Also, the processor 11 may perform an operation for at least one application or computer program for executing a method/operation according to various embodiments of the present disclosure.

The memory 13 stores various data, commands and/or information. Memory 13 may load one or more computer programs from storage 15 to execute methods/operations according to various embodiments of the present disclosure. The memory 13 may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

Storage 15 may non-temporarily store computer programs. The storage 15 may be a non-volatile memory such as read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or the like, a hard disk, a removable disk, or a device well known in the art. It may be configured to include any known type of computer-readable recording medium.

The communication interface 17 supports wired and wireless Internet communication of the sorting device 10 . Also, the communication interface 17 may support various communication methods other than internet communication. To this end, the communication interface 17 may include a communication module well known in the art of the present disclosure.

Bus 19 provides communication between the components of sorting device 10 . The bus 19 may be implemented in various types of buses such as an address bus, a data bus, and a control bus.

The embodiments of the present disclosure described above are not implemented only through devices and methods, and may be implemented through a program that realizes functions corresponding to the configuration of the embodiments of the present disclosure or a recording medium on which the program is recorded.

Although the embodiments of the present disclosure have been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present disclosure defined in the following claims are also included in the present disclosure. that fall within the scope of the right.

Claims (20)

A method of operating a medical data selection device operated by at least one processor,
generating some medical data sampled from a large amount of medical data and annotation data of the partial medical data as training data;
Extracting annotation candidate data that is at least a part of the mass medical data; and
Obtaining an inference result for the annotation candidate data from an artificial intelligence model trained through the training data, and selecting an annotation target for next training of the artificial intelligence model from among the annotation candidate data based on the inference result.
Operation method including. In paragraph 1,
The step of selecting the annotation target is
Determining a selection policy of medical data for next training based on the training performance of the artificial intelligence model, and selecting an annotation target corresponding to the selection policy from among the annotation candidate data based on the reasoning result. In paragraph 2,
The selection policy
An operation method, which is a policy of extracting medical data of a specific class whose classification accuracy is below the standard at a higher rate than other classes. In paragraph 2,
extracting verification data that is at least a part of the mass medical data; and
Evaluating training performance of the artificial intelligence model based on the verification data
Operation method further comprising. In paragraph 1,
The step of selecting the annotation target is
Adding medical data randomly extracted from the mass medical data to the annotation target. In paragraph 1,
The step of selecting the annotation target is
Obtaining an inference result for the annotation candidate data from an artificial intelligence model trained through the training data, and selecting medical data related to a specific class from the annotation candidate data based on the inference result. In paragraph 1,
The step of generating the training data is
Analyzing a reading report associated with the mass medical data, sampling the partial medical data including information related to training of the artificial intelligence model in the reading report, obtaining annotation data of the partial medical data, and obtaining the training data generating, how it works. In paragraph 1,
The bulk of medical data is
An operating method comprising at least one image acquired through a medical imaging device, pathology images, or patch images extracted from each medical image. A method of operating a medical data selection device operated by at least one processor,
A selection policy for medical data is determined based on the training performance of the current artificial intelligence model, and an annotation target corresponding to the selection policy is selected from among the annotation candidate data based on the inference result of the current artificial intelligence model for annotation candidate data. After selecting for next training, repeating the process of training the current artificial intelligence model based on the annotation data obtained for the annotation target; and
When the next training of the current artificial intelligence model is not performed, terminating the process,
The method of operation, wherein the annotation candidate data is data of at least a part of mass medical data. In paragraph 9,
The repeating steps
Extracting verification data, which is at least a part of the mass medical data, and evaluating training performance of the current artificial intelligence model based on the verification data. In paragraph 9,
The repeating steps
Adding medical data randomly extracted from the mass medical data to the annotation target. In paragraph 9,
The repeating steps
The operating method of determining the selection policy as before or differently according to the training performance of the current artificial intelligence model. In paragraph 9,
The current AI model is
An initial model trained on the basis of some medical data sampled from the mass medical data and annotation data of the partial medical data, or the initial model is a model retrained through the process. In paragraph 9,
The bulk of medical data is
An operating method comprising at least one image acquired through a medical imaging device, pathology images, or patch images extracted from each medical image. A medical data selection device operated by at least one processor,
An artificial intelligence model trained to output inferred results from inputs based on training data, and
Extracting annotation candidate data, which is at least a part of mass medical data, obtaining an inference result for the annotation candidate data from the artificial intelligence model, and performing a next training of the artificial intelligence model among the annotation candidate data based on the inference result. Includes a selector for selecting an annotation target;
The annotation data obtained for the annotation target for the next training is used for the next training of the artificial intelligence model. In paragraph 15,
The selector
When the next training of the artificial intelligence model is required, the medical data selection device repeats the process of selecting the annotation target for the next training based on the reasoning result of the artificial intelligence model. In paragraph 15,
The selector
A medical data selection device that determines a medical data selection policy for the next training based on the training performance of the artificial intelligence model and selects an annotation target corresponding to the selection policy from among the annotation candidate data based on the reasoning result. . In paragraph 17,
The selector
A medical data selection device that determines the selection policy as before or differently according to the training performance of the artificial intelligence model. In paragraph 15,
The selector
Medical data selection apparatus for adding medical data randomly extracted from the mass medical data to the annotation target. In paragraph 15,
The bulk of medical data is
A medical data screening apparatus comprising at least one patch image extracted from images obtained through at least one medical imaging device, pathology images, or each medical image.