KR102209382B1 - Method for providing information of lesion diagnosis and device for providing information of lesion diagnosis using the same - Google Patents

Abstract

The present invention provides the steps of: receiving a medical image for a region suspected of developing a disease of the individual, predicting a lesion on the individual using a classifier configured to predict a lesion based on the medical image, and determining the lesion in the medical image. Among the predicting processing, a method for providing information on lesion diagnosis, including the step of generating a lesion expression suspicious image indicating a degree of interest in predicting a lesion of a classifier, and providing a predicted lesion and a suspected lesion expression image, and the same It provides a device for providing information about the used lesion diagnosis.

Description

A method for providing information on lesion diagnosis and a device for providing information on lesion diagnosis using the same {METHOD FOR PROVIDING INFORMATION OF LESION DIAGNOSIS AND DEVICE FOR PROVIDING INFORMATION OF LESION DIAGNOSIS USING THE SAME}

The present invention relates to a method for providing information on lesion diagnosis and a device for providing information on lesion diagnosis using the same, and more specifically, to a lesion diagnosis configured to classify a lesion related to a disease based on a medical image and provide information thereon. The present invention relates to a method for providing information about, and a device for providing information on lesion diagnosis using the same.

The medical image may refer to an image obtained by using a characteristic having different degrees of incidence, reflection, and transmission according to a density and a property of a human body by administering constant energy through a medical imaging device. These medical images include X-ray images, ultrasonography images, computed tomography (CT) images, magnetic resonance imaging (MRI) images, and positron emission tomography. There may be Tomography, PET) images, and even an endoscopy image.

Medical staff can provide early diagnosis of diseases and treatment according to lesions by reading such medical images. However, in the auxiliary diagnosis system based on such a medical image, there may be deviations in diagnosis according to the skill level of a medical staff. Diagnosis deviations may also be related to medical accidents such as misdiagnosis, delay in treatment timing, and inappropriate treatment.

For example, in the case of the thyroid gland, ultrasound images may be used in diagnosing a lesion for a tumor or hyperfunction. More specifically, the ultrasound examination of the thyroid gland is performed while scanning the neck area with a probe in a position where the subject is lying down. At this time, the thyroid gland is a diagnostic site that is difficult to distinguish from surrounding tissues, and if a curved surface is formed by a lesion such as a tumor in the thyroid gland, a space may be formed between the probe and the object while the probe is moving. There may be difficulties with diagnosis. As a result, the diagnosis of a lesion may be different depending on the skill level of the medical staff, and the reliability of diagnosis may be degraded.

That is, a conventional auxiliary diagnosis system based on medical images has difficulty in accurately predicting lesions associated with a disease, and thus a probability of occurrence of a medical accident may be high.

As the accuracy of diagnosis is more demanded for the prevention of medical accidents and improvement of medical services, the development of a new auxiliary diagnosis system is continuously required for the onset of disease and further accurate lesion diagnosis.

The technology that is the background of the present invention has been prepared to facilitate understanding of the present invention. It should not be understood as an admission that the matters described in the technology behind the invention exist as prior art.
Prior Document: Registered Patent Publication No. 10-1824691 (20180202)

As a new diagnostic system for the onset of a specific disease and further disease-related lesions, a molecular biological diagnosis method based on the level of a disease or a biomarker associated with a lesion has been proposed.

However, the molecular biological diagnosis method can diagnose the onset of a disease, but may have limitations in accurately distinguishing and diagnosing characteristics of lesions such as cell types and progression degrees of lesions having different biological behaviors.

On the other hand, the inventors of the present invention have noted that limitations and problems of the conventional auxiliary diagnosis system based on medical images can be supplemented by introducing an auxiliary diagnosis system based on an artificial intelligence algorithm. More specifically, the inventors of the present invention have paid attention to a computer aided diagnosis (CADx) system that provides accurate lesion diagnosis by supplementing diagnostic findings using artificial intelligence in addition to reading medical images by medical staff.

At this time, the inventors of the present invention were able to recognize that the prediction model learned by the data of the medical image in relation to the diagnosis based on the medical image can provide information related to the diagnosis of the lesion.

In particular, the inventors of the present invention were able to recognize that when a plurality of models are used, diagnostic information with high reliability can be provided according to improvement in accuracy and precision of lesion diagnosis.

More specifically, the inventors of the present invention provide a fine-tuning feature extraction model to extract features in the process of predicting lesions for medical images, and to classify lesions based on the extracted features. The constructed classification model was intended to be applied to the auxiliary diagnosis system.

As a result, the inventors of the present invention extract features for a medical image using a plurality of predictive models, and probabilistically predict a lesion of a disease based on the extracted features. It has come to develop an auxiliary diagnostic system based on a new medical image that can provide accurate information.

At this time, the inventors of the present invention combine the features extracted by the plurality of models configured to extract the features from the medical image, and calculate the result value by the plurality of classification models configured to classify the lesion based on the combined features. In combination, the ensemble method, which finally provides information on lesion diagnosis, could be further applied.

As a result, the inventors of the present invention were able to confirm that the classification result for the lesion can be provided with higher accuracy, precision, and specificity than when using a single model.

On the other hand, the inventors of the present invention could further recognize that there is a limitation in improving the diagnostic performance of the model due to the lack of training data in relation to the prediction model used for diagnosis based on medical images.

As a result, the inventors of the present invention could expect to improve the diagnostic performance of the predictive model of the present invention by configuring the predictive model applied to the auxiliary diagnostic system to selectively learn more effective training data for lesion prediction.

More specifically, the inventors of the present invention are to determine a selective learning medical image whose variance according to the result of feature extraction (or classification) from the learning medical image used for learning is greater than or equal to a predetermined level, and reflect this in the retraining of the models. I did.

As a result, the inventors of the present invention were able to confirm improvement in diagnostic ability, such as predicting whether a disease has occurred or not, and further predicting a lesion, by configuring models to learn medical images for selective learning.

Further, the inventors of the present invention confirmed that the thyroid ultrasound image was applied to the auxiliary diagnostic system as described above, and the features were extracted from the image and classified as benign or malignant, and thus information on lesion diagnosis can be provided. Could

Accordingly, the problem to be solved by the present invention is to classify lesions using a plurality of feature extraction models configured to extract features based on the received medical image and a plurality of classifiers configured to classify lesions based on the features. It is to provide a method of providing information about diagnosis.

Another problem to be solved by the present invention is information on lesion diagnosis, including a receiving unit configured to receive a medical image for a target site and a processor configured to extract features for a medical image and classify a lesion for a target site It is to provide a device for provision.

The problems of the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above-described problems, a method for providing information on lesion diagnosis according to an embodiment of the present invention is provided. The method includes the steps of: receiving a medical image for a target site, extracting a feature for a medical image using a plurality of feature extraction models configured to extract features extracted during a process of predicting a lesion within the medical image, And classifying the lesion for the target site in the medical image using a plurality of classifiers configured to classify the lesion based on the extracted features.

According to a feature of the present invention, each of the plurality of feature extraction models may be configured to extract at least one feature different from each other for each feature extraction model. Furthermore, the method may further include combining at least one feature each extracted by the plurality of feature extraction models to obtain a combined feature after the step of extracting the feature. In addition, the step of classifying the lesion for the target site may include classifying the lesion for the target site based on the binding feature using a plurality of classifiers.

According to another feature of the invention, at least one feature may be at least two features. In addition, the combining of the at least one feature includes first combining the at least one feature for each model to obtain a combination feature for each model, and a second combining the combination feature for each model to obtain a final combination feature. Can include. Furthermore, the step of classifying the lesion for the target site may further include classifying the lesion for the target site based on the final binding feature using a plurality of classifiers.

According to another feature of the present invention, the step of classifying a lesion within a medical image includes calculating a lesion probability for a feature using a plurality of classifiers, and classifying a lesion for a target site based on the lesion probability. It may include steps.

According to another feature of the invention, the lesion may comprise a plurality of predetermined lesions. In this case, the step of calculating the lesion probability includes calculating the probability of each of the plurality of lesions for the feature using a plurality of classifiers, and the plurality of lesions based on the probability of each of the plurality of lesions calculated by the plurality of classifiers. It may include calculating an average lesion probability for each lesion. In addition, classifying the lesion for the target site based on the lesion probability may include determining to have one of the plurality of lesions for the target site based on the average lesion probability.

According to another feature of the present invention, the features may include a plurality of features different from each other, and the plurality of classifiers may include a plurality of ensemble classifiers composed of a plurality of classifiers. In this case, the plurality of ensemble classifiers may be configured to classify lesions for features different from each other for each ensemble classifier. In addition, the step of classifying the lesion within the medical image may include calculating a lesion probability for each of a plurality of different features, using a plurality of ensemble classifiers, and a lesion probability for each of a plurality of different features. , It may include the step of classifying the lesion for the target site.

According to another feature of the invention, the lesion may comprise a plurality of predetermined lesions. In this case, the step of calculating the lesion probability includes calculating a probability of each of a plurality of lesions for each of a plurality of different features, and a plurality of lesions for each of a plurality of different features, using a plurality of ensemble classifiers. It may include calculating an average lesion probability for each of the plurality of lesions based on the probability of. In addition, classifying the lesion for the target site based on the lesion probability may include determining to have one of the plurality of lesions for the target site based on the average lesion probability.

According to another feature of the present invention, the feature extraction model may be a model modified to extract features before final prediction for a lesion.

According to another feature of the present invention, the feature extraction model may be a pre-trained convolutional neural network (CNN) model composed of a plurality of layers. In this case, a selected one of the plurality of layers may be a model transformed into an output layer.

According to another feature of the present invention, the feature extraction model includes: receiving a medical image for training, and learning to extract a feature based on the medical image for training with respect to a feature extraction model having a modified output layer. It may be a fine-tuned feature extraction model.

According to another feature of the present invention, the feature extraction model includes the steps of determining a medical image for selective learning whose variance is equal to or greater than a predetermined level among the medical images for learning, and features based on the medical image for selective learning for the finely adjusted feature extraction model. It may be a feature extraction model in which retraining is performed through the step of learning to extract.

According to another feature of the present invention, the classifier includes receiving a feature extracted by a feature extraction model with respect to a medical image for learning including a predetermined lesion, and based on the features extracted for the medical image for learning, in advance. It may be a classifier learned through the step of classifying the determined lesion.

According to another feature of the present invention, the classifier comprises the steps of determining a selected medical image for learning that has a variance of at least a predetermined level among the medical images for training, and a lesion determined in advance based on the features for the selected medical image for learning with respect to the learned classifier. It may be a classifier re-learned through the step of classifying.

According to another feature of the present invention, medical images include ultrasound images, X-ray images, computed tomography (CT) images, magnetic resonance imaging (MRI) images, endoscopic images, and positron emission tomography (PET). It may be at least one of the images.

According to another feature of the present invention, the plurality of feature extraction models may be at least two deep learning models among AlexNet, OverFeat, VGG, VGG-verydeep, ResNet, GoogleNet, and pseudoweight.

According to another feature of the present invention, a plurality of classifiers are, SVM (Support Vector Machine), RF (Random Forests), LDA (linear discriminant analysis), QDA (quadratic discriminant analysis), decision tree (decision tree), XG Boost (extreme gradient boosting), logistic regression (logistic regression), logistic regression, NN (nearest neighbor), and GMM (Gaussian mixture model) may be at least two classification models.

In order to solve the above-described problems, a device for providing information on lesion diagnosis according to an embodiment of the present invention is provided. The device includes a receiving unit configured to receive a medical image for a target portion, and a processor configured to communicate with the receiving unit. In this case, the processor is configured to extract features for the medical image using a plurality of feature extraction models configured to extract features extracted during the processing of predicting lesions in the medical image, and classify the lesions based on the extracted features. It is configured to classify lesions for a target site in the medical image using a plurality of classifiers.

According to a feature of the present invention, each of the plurality of feature extraction models may be configured to extract at least one feature different from each other for each feature extraction model. In this case, the processor is further configured to combine at least one feature, each extracted by a plurality of feature extraction models, to obtain a coupling feature, and classify a lesion for a target site based on the coupling feature using a plurality of classifiers. Can be configured.

According to another feature of the invention, at least one feature may be at least two features. In this case, the processor first combines at least one feature for each model to obtain a combination feature for each model, second combines the combination feature for each model to obtain a final combination feature, and determines the final combination feature using a plurality of classifiers. It may be further configured to classify the lesion for the site of interest on the basis of.

According to another feature of the present invention, the processor may be further configured to calculate a lesion probability for a feature using a plurality of classifiers, and to classify a lesion for a target site based on the lesion probability.

According to another feature of the invention, the lesion may comprise a plurality of predetermined lesions. At this time, the processor calculates the probability of each of a plurality of lesions for a feature using a plurality of classifiers, and calculates an average lesion probability for each of the plurality of lesions based on the probability of each of the plurality of lesions calculated by the plurality of classifiers. , May be further configured to determine to have one of the plurality of lesions for the target site based on the average lesion probability.

According to another feature of the present invention, the features may include a plurality of features different from each other. In this case, the plurality of classifiers may include a plurality of ensemble classifiers composed of a plurality of classifiers, and the plurality of ensemble classifiers may be configured to classify lesions for different features for each ensemble classifier. Further, the processor is further configured to calculate a lesion probability for each of a plurality of different features using a plurality of ensemble classifiers, and to classify the lesion for the target site based on the lesion probability for each of the plurality of different features. Can be.

According to another feature of the invention, the lesion may comprise a plurality of predetermined lesions. At this time, the processor uses a plurality of ensemble classifiers to calculate the probability of each of the plurality of lesions for each of a plurality of different features, and the plurality of lesions based on the probability of each of the plurality of lesions for each of the plurality of different features. It may be further configured to calculate an average lesion probability for each and determine to have one of a plurality of lesions for the target site based on the average lesion probability.

In order to solve the above-described problem, a device for providing information on lesion diagnosis according to another embodiment of the present invention is provided. The present device includes a medical image receiving unit configured to receive a medical image for a target site, and a feature for a medical image using a plurality of feature extraction models configured to extract features extracted during processing for predicting lesions in the medical image. And a lesion classifying unit configured to classify a lesion for a target site in the medical image using a feature extracting unit configured to extract and a plurality of classifiers configured to classify the lesion based on the extracted features.

The present invention provides a method for providing information on lesion diagnosis using a plurality of models configured to extract features for a medical image and predict the onset of a disease and further predict a lesion based on this, thereby providing a device using the same. There is an effect that can provide diagnostic information.

Accordingly, the present invention has an effect of overcoming a limitation of a conventional auxiliary diagnosis system based on a medical image, which may reduce the reliability of diagnosis because the lesion diagnosis result may be different depending on the skill level of the medical staff.

For example, the present invention can provide information on not only the onset of thyroid cancer, but also malignant or benign lesions based on the thyroid ultrasound image, and an accurate and effective treatment tailored to the individual's condition according to the prediction result. Can provide. Thus, the present invention has an effect that can contribute to providing a good prognosis for early disease treatment.

In particular, the present invention provides a new auxiliary diagnosis system using a plurality of feature extraction models finely tuned to extract features in the process of predicting lesions for medical images and a plurality of classifiers configured to classify lesions based on the extracted features. can do. Accordingly, the present invention has an effect of providing a classification result for a lesion with higher accuracy, precision and specificity than when using a single model.

In addition, according to the present invention, the predictive model applied to the auxiliary diagnosis system is configured to selectively learn more effective learning data for predicting results, thereby improving diagnostic performance.

That is, the present invention has an effect of overcoming a limitation of performance improvement due to insufficient training data of a pre-trained model.

More specifically, the present invention selects a medical image for training whose variance according to a prediction result is equal to or greater than a predetermined level, and reflects the models learned based on it, thereby providing an auxiliary diagnosis system based on a medical image with improved diagnostic performance. There is.

The effects according to the present invention are not limited by the contents exemplified above, and more various effects are included in the present specification.

1 is an exemplary diagram illustrating a configuration of a device for providing information on lesion diagnosis according to an embodiment of the present invention.
2A is an exemplary diagram illustrating a configuration of a device for providing information on lesion diagnosis according to another embodiment of the present invention.
2B is an exemplary diagram illustrating a feature extraction unit of a device for providing information about lesion diagnosis according to another embodiment of the present invention.
2C is an exemplary diagram illustrating a lesion classification unit of a device for providing information on lesion diagnosis according to another embodiment of the present invention.
3A is a diagram illustrating a procedure of a method for providing information on lesion diagnosis according to an embodiment of the present invention.
3B and 3C exemplarily illustrate a feature extraction procedure according to a method for providing information on lesion diagnosis according to an embodiment of the present invention.
3D and 3E exemplarily illustrate a procedure for classifying a lesion according to a method for providing information on lesion diagnosis according to an embodiment of the present invention.
4A is a diagram illustrating a feature extraction model used in various embodiments of the present invention and fine adjustments performed thereon.
4B illustrates evaluation results according to whether or not a feature extraction model used in various embodiments of the present invention is finely adjusted.
5A to 5E illustrate evaluation results according to combining a plurality of feature extraction models used in various embodiments of the present invention.
6 is a diagram illustrating evaluation results according to a combination of a plurality of classifiers used in various embodiments of the present invention.

Advantages of the invention and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various forms different from each other, and only these embodiments make the disclosure of the present invention complete, and common knowledge in the technical field to which the present invention pertains. It is provided to completely inform the scope of the invention to those who have it, and the invention is only defined by the scope of the claims.

The shapes, sizes, ratios, angles, numbers, etc. disclosed in the drawings for explaining the embodiments of the present invention are exemplary, and the present invention is not limited to the illustrated matters. In addition, in describing the present invention, when it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. When'include','have','consists of' and the like mentioned in the present specification are used, other parts may be added unless'only' is used. In the case of expressing the constituent elements in the singular, it includes the case of including the plural unless specifically stated otherwise.

In interpreting the components, it is interpreted as including an error range even if there is no explicit description.

Each of the features of the various embodiments of the present invention can be partially or entirely combined or combined with each other, and as a person skilled in the art can fully understand, technically various interlocking and driving are possible, and each of the embodiments may be independently implemented with respect to each other. It may be possible to do it together in a related relationship

For clarity of interpretation of the present specification, terms used in the present specification will be defined below.

As used herein, the term “medical image” may refer to all medical images captured by an imaging apparatus for a region suspected of a disease. Meanwhile, the medical image may include an X-ray image, an ultrasound image, a computed tomography image, a magnetic resonance image, a positron electron tomography image, and an endoscope image of a region suspected of a disease.

Preferably, the medical image disclosed in the present specification may be an ultrasound image of a region suspected of a disease, but is not limited thereto.

Furthermore, the medical image may be a 2D image, a 3D image, a still image of one cut, or a moving image composed of a plurality of cuts. For example, when the medical image is a video composed of a plurality of cuts, the disease occurrence and lesion for each of the plurality of medical images may be predicted according to the method for providing information on lesion diagnosis according to an embodiment of the present invention. . As a result, the present invention can provide diagnostic information on a lesion site predicted in real time by performing prediction of a lesion at the same time as receiving a medical image from an imaging apparatus.

As used herein, the term "target site" may be a specific body part of a subject who wants to observe a condition such as the presence or absence of a disease. For example, the target site may be the chest, spine, upper abdomen, lower abdomen, lungs, brain, liver, varicose veins, uterus, prostate, testis, musculoskeletal system, thyroid gland, or breast. However, the target portion is not limited thereto, and may be various portions as long as an image is acquired by the imaging apparatus.

As used herein, the term "lesion" may mean various clinical forms, features, or symptoms that appear for a specific disease. For example, in the case of a thyroid nodule, its lesions can be classified as benign or malignant. In addition, in the case of gastric cancer, its lesions can be classified into mucosal invasive gastric cancer and submucosal invasive gastric cancer. However, the lesion is not limited thereto. For example, the lesion may further include a'normal condition' clinically.

On the other hand, even if an individual has the same disease, lesions of biological characteristics may be different. At this time, in the treatment of the disease, the treatment applied may be different depending on the lesion. Accordingly, accurately classifying and diagnosing not only the onset of a disease, but also a lesion may be very important in providing an accurate and effective treatment tailored to an individual's condition.

In this case, in order to improve the accuracy of the diagnosis of the disease and the prediction of the lesion, a prediction model configured to predict the lesion based on the medical image may be used.

As used herein, the term "feature" may mean a feature extracted as meaningful information in a medical image.

For example, the feature may be a feature parameter capable of identifying (or recognizing) a target region present in a medical image.

As used herein, the term "feature extraction model" may be a model configured to recognize and extract features that are significant for a medical image.

According to a feature of the present invention, the feature extraction model may be a model modified to extract features before final prediction for a lesion. More specifically, the feature extraction model may be a pre-trained convolutional neural network (CNN) model composed of a plurality of layers, and may be a model in which one selected layer among the plurality of layers is transformed into an output layer.

According to another feature of the present invention, the feature extraction model may be a fine-tuned model in which a training medical image is received and learning is performed on a feature extraction model having a modified output layer based on a training medical image.

According to another feature of the present invention, the feature extraction model determines a selective learning medical image whose variance is equal to or greater than a predetermined level among the training medical images, and retraining is performed based on the selective learning medical image for the finely adjusted feature extraction model. It may be an performed feature extraction model. In this case, among the training medical images, a training medical image having a variance of 0.02 or 0.045 or more for a classification result by a finely adjusted feature extraction model may be determined as a selective training medical image.

For example, the variance ( A ) for the classification result may be calculated by Equation 1 below.

[Equation 1]

Here P ₀ and P ₁ may be the probability for two classes of lesions (eg, benign tumors or malignant tumors).

That is, a medical image for learning in which a difference between values of two lesion probabilities is equal to or greater than a predetermined level may be determined as a medical image for selective learning.

Meanwhile, the feature extraction model of the present invention may be at least one of AlexNet, OverFeat, VGG, VGG-verydeep, ResNet, GoogleNet, and Inception. According to a feature of the present invention, a combination of two or more selected models among the above models may be used.

For example, when a plurality of feature extraction models are applied in feature extraction, feature parameters extracted by each feature extraction model are combined, and the combined feature may be determined as a final feature.

However, the feature extraction model of the present invention is not limited to the above-described features.

As used herein, the term "classifier" may be a model trained to predict the onset of a specific disease and, further, a plurality of lesions for the same disease, based on features extracted from a medical image.

More specifically, the classifier of the present invention receives features extracted by a feature extraction model for a medical image for learning including a predetermined lesion, and classifies a predetermined lesion based on the extracted features for a medical image for training. It may be a classification model learned through the step of. For example, it may be a model configured to probabilistically predict (classify) two classes of malignant tumors or benign tumors based on features extracted for medical images of the thyroid gland.

On the other hand, the classifier of the present invention is a classifier relearned through the step of determining a selective learning medical image whose variance is equal to or greater than a predetermined level among the medical images for learning, and classifying a predetermined lesion based on a feature of the medical image for selective learning. I can. In this case, a medical image for learning having a variance of 0.02 or 0.045 or more for a classification result by the classifier among the medical images for learning may be determined as a medical image for selective learning.

For example, the variance ( B ) for the classification result may be calculated by Equation 2 below.

[Equation 2]

Here P ₀ and P ₁ may be the probability for two classes of lesions (eg, benign tumors or malignant tumors).

That is, a medical image for learning in which a difference between values of two lesion probabilities is equal to or greater than a predetermined level may be determined as a medical image for selective learning.

On the other hand, the classifier of the present invention is SVM (Support Vector Machine), RF (Random Forests), LDA (linear discriminant analysis), QDA (quadratic discriminant analysis), decision tree (decision tree), XG Boost (extreme gradient boosting), logistic Regression analysis (logistic regression), logistic regression, NN (nearest neighbor), and GMM (Gaussian mixture model) may be at least one model. According to a feature of the present invention, a combination of two or more selected models among the above models may be used.

For example, when a plurality of classifiers are applied in lesion classification, an average of the result values output by each classifier is calculated, and the lesion may be finally determined based on the average value.

However, the classifier of the present invention is not limited to the above-described features.

As used herein, the term "ensemble classifier" may mean a classifier having a configuration in which two or more classifiers selected from among the aforementioned classifiers are combined in series or in parallel.

According to a feature of the present invention, a plurality of ensemble classifiers can be used for lesion classification. In this case, the ensemble classifier may be configured to classify lesions for different features for each classifier.

For example, when a plurality of ensemble classifiers are applied in lesion classification, an average of result values output by each ensemble classifier may be calculated, and the lesion may be finally determined based on the average value.

Hereinafter, a device for providing information on lesion diagnosis according to an embodiment of the present invention will be described with reference to FIG. 1.

1 is a diagram illustrating a configuration of a device for providing information on lesion diagnosis according to an embodiment of the present invention.

Referring to FIG. 1, a device 100 for providing information on lesion diagnosis, a receiver 110, an input unit 120, an output unit 130, a storage unit 140, and a processor 150 are included.

In more detail, the receiving unit 110 may be configured to receive a medical image of a target portion of the subject from the image diagnosis apparatus. In this case, the receiving unit 110 may receive an X-ray image, an ultrasound image, a computed tomography image, a magnetic resonance image, a positron tomography image, and further a medical image of an endoscopic image for a region suspected of a disease. Meanwhile, a region in which a lesion is expressed may be included in the medical image acquired through the receiving unit 110.

The input unit 120 is not limited, such as a keyboard, a mouse, and a touch screen panel. The input unit 120 may set the device 100 for providing information on lesion diagnosis and instruct an operation thereof. For example, the medical practitioner may directly select a region in which a lesion is to be predicted for a medical image received by the receiving unit 110 through the input unit 120.

Meanwhile, the output unit 130 may visually display the medical image received by the receiving unit 110. Further, the output unit 130 is configured to output features extracted by the processor 150 to be described later, or to visually display information on lesions classified by a classifier, and provide information on lesion diagnosis to medical personnel. I can.

The storage unit 140 may be configured to store a medical image received through the receiving unit 110 and to store an instruction of the device 100 for providing information on lesion diagnosis set through the input unit 120. Furthermore, the storage unit 140 is configured to store information on features and classified lesions extracted by the processor 150 to be described later. However, it is not limited to the above, and the storage unit 140 may store various pieces of information determined by the processor 150.

The processor 150 may be a component for providing an accurate prediction result for the device 110 for providing information on lesion diagnosis. In this case, the processor 150 may use a predictive model configured to extract features from the medical image and classify lesions based on this. In particular, the processor 150 may be based on a plurality of predictive models that have been retrained based on a medical image for selective learning to provide more reliable information.

For example, the processor 150 may use a plurality of feature extraction models configured to extract features from a medical image and a plurality of classifiers configured to probabilistically classify a lesion based on the features.

Accordingly, based on the medical image received from the processor 150 receiving unit 110, a feature is extracted, a lesion is classified, and a lesion of an individual is predicted probabilistically, thereby providing lesion diagnosis information with high accuracy.

Meanwhile, the processor 150 uses at least two feature extraction models selected from AlexNet, OverFeat, VGG, VGG-verydeep, ResNet, GoogleNet, and Inception, and uses SVM (Support Vector Machine), RF (Random Forests), LDA (linear discriminant analysis), quadratic discriminant analysis (QDA), decision tree, XG Boost (extreme gradient boosting), logistic regression, logistic regression, NN (nearest neighbor), and GMM (Gaussian mixture model). It can be configured to use at least two classification models selected. However, the configuration of the processor 150 is not limited thereto.

Hereinafter, a device for providing information on lesion diagnosis according to another embodiment of the present invention will be described in detail with reference to FIGS. 2A to 2C.

2A is an exemplary diagram illustrating a device for providing information on lesion diagnosis according to another embodiment of the present invention. 2B is an exemplary diagram illustrating a feature extraction unit of a device for providing information about lesion diagnosis according to another embodiment of the present invention. 2C is an exemplary diagram illustrating a lesion classification unit of a device for providing information on lesion diagnosis according to another embodiment of the present invention.

First, referring to FIG. 2A, a device 200 for providing information on lesion diagnosis, a medical image receiving unit 210, a feature extracting unit 220, and a lesion classifying unit 230 are included.

More specifically, the medical image receiving unit 210 may be configured to receive a medical image for a target site. At this time, the medical image receiving unit 210 may receive an X-ray image, an ultrasound image, a computed tomography image, a magnetic resonance image, a positron tomography image, and further a medical image of an endoscope image for a region suspected of a disease.

Meanwhile, according to a feature of the present invention, the medical image receiving unit 210 may be an image diagnosis apparatus that provides a medical image for a target site. For example, the medical image receiving unit 210 may be an ultrasound probe or an endoscope probe.

In this case, a region in which the lesion is expressed may be included in the medical image acquired through the medical image receiving unit 210.

Next, the feature extraction unit 220 may be configured to receive a medical image through the medical image reception unit 210 and extract (output) a feature based on the medical image.

According to a feature of the present invention, the feature extraction unit 220 may be based on a feature extraction model configured to extract features from a medical image. In this case, the feature extraction model may be a model modified to extract features before final prediction of the lesion.

For example, referring to FIG. 2B, the feature extraction unit 220 may be configured with a pre-trained feature extraction model 222 configured with a plurality of layers. Furthermore, the feature extraction unit 220 uses a modified feature extraction model 224 to output one layer selected from a plurality of layers, for example, the last layer as the number of classes for the pre-learned feature extraction model 222. Can be configured. In addition, the feature extraction unit 220 may be configured with a finely adjusted feature extraction model 226 that has been trained to extract a feature based on a training medical image for the deformed feature extraction model 224. In addition, the feature extraction unit 220 is composed of a retrained feature extraction model 228 to extract a feature based on a selective learning medical image having a variance of at least a predetermined level with respect to the finely adjusted feature extraction model 226. I can.

In this case, a medical image for training with a variance of 0.02 or 0.045 or more for a classification result by a finely adjusted feature extraction model among the training medical images may be determined as a selective learning medical image.

For example, the variance ( A ) for the classification result may be calculated by Equation 1 below.

[Equation 1]

Here P ₀ and P ₁ may be the probability for two classes of lesions (eg, benign tumors or malignant tumors).

That is, a medical image for learning in which a difference between values of two lesion probabilities is equal to or greater than a predetermined level may be determined as a medical image for selective learning.

Meanwhile, the feature extraction model may be at least one of AlexNet, OverFeat, VGG, VGG-verydeep, ResNet, GoogleNet, and Inception.

According to another feature of the present invention, the feature extraction unit 220 may be configured as a combination of two or more selected models.

For example, when the feature extraction unit 220 is composed of a plurality of feature extraction models, feature parameters extracted by each feature extraction model are combined. The combined feature can then be determined as the final feature.

However, the configuration of the feature extraction unit 220 is not limited to that described above.

Next, the lesion classification unit 230 may be configured to receive a feature extracted by the feature extraction unit 220 and classify (output) a lesion based on the received feature.

According to a feature of the present invention, the lesion classification unit 230 may be based on a classifier learned to predict whether a specific disease has occurred, and further predicts a plurality of lesions for the same disease, based on features extracted from a medical image. .

For example, referring to FIG. 2C, the lesion classification unit 230 receives, for the classifier 232, an extracted feature for a medical image for learning including a predetermined lesion, and classifies a predetermined lesion based on this. It can be configured with a classifier 234 that has been learned to. Furthermore, the lesion classification unit 230 may be configured with a retrained classifier 236 to classify diseased diseases based on features of a medical image for selective learning having a variance of at least a predetermined level with respect to the learned classifier 234. .

In this case, a medical image for learning having a variance of 0.02 or 0.045 or more for a classification result by the classifier among the medical images for learning may be determined as a medical image for selective learning.

For example, the variance ( B ) for the classification result may be calculated by Equation 2 below.

[Equation 2]

Here P ₀ and P ₁ may be the probability for two classes of lesions (eg, benign tumors or malignant tumors).

That is, a medical image for learning in which a difference between values of two lesion probabilities is equal to or greater than a predetermined level may be determined as a medical image for selective learning.

Meanwhile, the classifier is SVM (Support Vector Machine), RF (Random Forests), LDA (linear discriminant analysis), QDA (quadratic discriminant analysis), decision tree, XG Boost (extreme gradient boosting), logistic regression analysis ( logistic regression), logistic regression, NN (nearest neighbor), and GMM (Gaussian mixture model).

According to a feature of the present invention, the lesion classification unit 230 may be configured as a combination of two or more selected models.

For example, when the lesion classifier 230 is configured with a plurality of classifiers, an average of the result values output by each classifier may be calculated, and the lesion may be finally determined based on the average value.

According to another feature of the present invention, the lesion classification unit 230 may be configured as an ensemble classifier in which two or more classifiers selected from among the aforementioned classifiers are combined in series or in parallel.

According to another feature of the present invention, the lesion classifier 230 may be configured with a plurality of ensemble classifiers configured to classify lesions for different features for each classifier.

For example, when the lesion classification unit 230 is configured with a plurality of ensemble classifiers, an average of the result values output by each ensemble classifier may be calculated, and the lesion may be finally determined based on the average value.

According to the configuration characteristics of the devices 100 and 200 for providing information on lesion diagnosis according to various embodiments of the present disclosure, the present invention can provide highly reliable information in lesion diagnosis. For example, the device 100, 200 for providing information on lesion diagnosis of the present invention is configured as a single predictive model according to configurational features using a plurality of feature extraction models and a plurality of classifiers, and thus lesions without feature extraction. It is possible to classify lesions with higher accuracy, precision and specificity than devices configured to classify. Further, the device for providing information on lesion diagnosis 100 and 200 according to the present invention is a conventional prediction model that is pre-trained according to a configuration feature using a feature extraction model and a classifier relearned based on the selected learning medical image. Lesions can be classified with higher accuracy, precision, and specificity than devices configured with.

Accordingly, the present invention provides devices 100 and 200 for providing information on lesion diagnosis with improved diagnostic capabilities, thereby providing information with higher reliability in lesion diagnosis than in a conventional auxiliary diagnosis system based on a single predictive model. I can.

Hereinafter, a method of providing information on lesion diagnosis according to various embodiments of the present disclosure will be described in detail with reference to FIGS. 3A to 3E. 3A is a diagram illustrating a procedure of a method for providing information on lesion diagnosis according to an embodiment of the present invention. 3B and 3C exemplarily illustrate a feature extraction procedure according to a method for providing information on lesion diagnosis according to an embodiment of the present invention. 3D and 3E exemplarily illustrate a procedure for classifying a lesion according to a method for providing information on lesion diagnosis according to an embodiment of the present invention.

Referring to FIG. 3A, the procedure of lesion classification according to an embodiment of the present invention is as follows. First, a medical image of a target site is received (S310). Then, during the process of predicting lesions for the medical image, features are extracted using a plurality of feature extraction models configured to output the extracted features (S320). Next, a lesion for a target site is classified using a plurality of classifiers configured to classify a lesion based on features (S330).

More specifically, in the step of receiving a medical image (S310), a medical image for a target site may be received from the imaging apparatus.

According to a feature of the present invention, in the step of receiving a medical image (S310), at least one of an ultrasound image, an X-ray image, a CT image, an MRI image, an endoscope image, and a PET (image) is received.

Next, in the feature extraction step S320, each of the plurality of feature extraction models may extract a feature with respect to the medical image received as a result of the step S310 of receiving the medical image.

According to a feature of the present invention, each of the plurality of feature extraction models in the feature extraction step (S320) may be configured to extract at least one feature different from each other for each feature extraction model.

For example, referring to FIG. 3B together, in the step of extracting a feature (S320), the medical image 312 received as a result of the step of receiving a medical image (S310) is a plurality of feature extraction models 322, respectively. Can be entered in Then, one feature 324 that is different from each other in each feature extraction model 322 may be extracted (output). Finally, the features extracted by each feature extraction model 322 are combined, so that a combined feature 226 can be obtained.

According to another feature of the present invention, each of the plurality of feature extraction models in the feature extraction step S320 may be configured to extract at least two different features for each feature extraction model.

For example, referring to FIG. 3C, a medical image 312 may be input to each of the plurality of feature extraction models 322. Then, two plurality of features 324 different from each other can be extracted (output) in each feature extraction model 322. Next, through a first combination of combining a plurality of features 324 for each of the feature extraction models 322, each of the combining features 326 for each model may be obtained. Finally, the final combination feature 328 may be obtained through the second combination of combining the plurality of model-specific combination features 326.

Finally, in the step of classifying the lesion (S330), a plurality of classifiers may classify the lesion with respect to the feature extracted as a result of the step of extracting the feature (S320).

According to a feature of the present invention, in the step of classifying a lesion (S330), a lesion probability for a feature is calculated by a plurality of classifiers, and a lesion for a target site is classified based on the lesion probability.

According to another feature of the present invention, in the step of classifying the lesion (S330), each of the plurality of classifiers is configured to classify the lesion based on a combined feature in which features output by the plurality of feature extraction models are combined.

For example, referring to FIG. 3D, in the step of classifying the lesion (S330), one combining feature 326 obtained as a result of the step of extracting a feature (S320) is input to each of the plurality of classifiers 332 Can be. Then, a predetermined probability of a lesion, that is, a lesion probability 334 for the binding feature 326 by each of the plurality of classifiers 332 may be calculated. Then, an average lesion probability 335 for a plurality of lesion probabilities 334 calculated by each of the plurality of classifiers 332 is calculated. Finally, if the average lesion probability 335 for a lesion is equal to or higher than a predetermined level, it can be classified as having a predetermined lesion for the target site. That is, information on the classification lesion 336 may be provided as a result of the classification of the lesion (S330). At this time, the probability for the classification lesion 336 may be provided together as information about lesion diagnosis.

According to another feature of the present invention, in the step of classifying a lesion (S330), one of a plurality of predetermined lesions is determined by a plurality of classifiers.

More specifically, in the step of classifying the lesion (S330), a probability of each of a plurality of lesions for a feature is calculated by a plurality of classifiers, and a plurality of the plurality of lesions are calculated based on the probability of each of the plurality of lesions calculated by the plurality of classifiers. The average lesion probability for each lesion can be calculated. Then, based on the average lesion probability, the target site may be determined to have one of the plurality of lesions.

According to another feature of the present invention, in the step of classifying a lesion (S330), a lesion based on a plurality of features is determined by an ensemble classifier composed of a plurality of classifiers.

For example, referring to FIG. 3E, in the step of classifying the lesion (S330), a plurality of joining features 326 obtained as a result of the step of extracting features (S320) are respectively added to the plurality of ensemble classifiers 342. Can be entered. Then, a lesion probability 334 and an average lesion probability 335 are calculated by each of the plurality of ensemble classifiers 342, and classification lesions 336 for each ensemble classifier 342 may be determined, respectively. Then, an average classified lesion probability 337 for the classified lesion 336 for each ensemble classifier 342 may be calculated. Finally, if the mean classification lesion probability 337 for the lesion is above a predetermined level, it can be finally classified as having a predetermined lesion for the target site. That is, information on the final lesion 338 determined as a result of classifying the lesion (S330) may be provided. At this time, the probability for the final lesion 338 may be provided together as information about lesion diagnosis.

According to another feature of the present invention, in the step of classifying a lesion (S330), one of a plurality of predetermined lesions is determined by a plurality of ensemble classifiers.

More specifically, in the step of classifying the lesion (S330), the probability of each of the plurality of lesions for the feature is calculated by the plurality of ensemble classifiers, and based on the probability of each of the plurality of lesions calculated by the plurality of ensemble classifiers. The average lesion probability for each lesion may be calculated. Then, based on the average lesion probability, the target site may be determined to have one of the plurality of lesions.

According to the information method for lesion diagnosis according to various embodiments of the present invention, the present invention can provide highly reliable information in lesion diagnosis. For example, the present invention is configured to extract features and classify lesions using a plurality of feature extraction models and a plurality of classifiers, with higher accuracy and precision than a device based on a single model configured to classify lesions without a feature extraction procedure. And can classify lesions by specificity

Accordingly, the present invention can provide more reliable information in lesion diagnosis than in the conventional auxiliary diagnosis system based on a single predictive model.

Example 1: Evaluation of fine-tuned feature extraction models

Hereinafter, a finely adjusted feature extraction model used in various embodiments of the present invention will be described in detail with reference to FIGS. 4A and 4B. 4A is a diagram illustrating a feature extraction model used in various embodiments of the present invention and fine adjustments performed thereon. 4B illustrates evaluation results according to whether or not a feature extraction model used in various embodiments of the present invention is finely adjusted.

More specifically, in this evaluation, a feature is extracted for each of the fine-tuned feature extraction model of the present invention and a feature extraction model based on a conventional pre-trained model, and the extracted two types of features are applied to the classifier, and the lesion It was performed by measuring the accuracy, precision and specificity of the classification results.

In this case, AlexNet, OverFeat, VGG, VGG-verydeep, ResNet and Inception were used as feature extraction models, and SVM and RF were used as classifiers.

More specifically, the feature extraction models of AlexNet, OverFeat, VGG and VGG-verydeep were fine-tuned to extract features for one of layers fc1 and fc2, or two output layers of fc1 and fc2. At this time, when features are extracted for two layers fc1 and fc2, a self-coupled combined feature was used as the final feature.

Furthermore, ResNet and Inception's feature extraction models were fine-tuned to extract features from one selected layer among multiple layers.

In addition, SVM and RF in the classification process were implemented by MATLAB 2018a using the built-in functions fitcsvm and TreeBagger . Furthermore, the SVM was configured to select the penalty parameter C as 1 and calculate the kernel size by empirical procedure.

However, the feature extraction model and classification of the present invention are not limited to those described above.

Referring to Figure 4a, the fineness of language (Language), package (Package) and pre-trained weights (Pre-trained weights) applied to each of the feature extraction models of AlexNet, OverFeat, VGG, VGG-verydeep, ResNet and Inception. Adjustment information is shown. Furthermore, fine-tuning information of epochs, learning rates, and learning decays, which means the number of repetitive learning for the medical image for training, applied to each of the models are shown.

In this case, values for the epoch, the learning rate, and the learning decay may be values optimized through six cross tests.

Referring to FIG. 4B, in most feature extraction models, when the last layer is transformed and fine adjustments performed by re-learning as a selective learning medical image are reflected, the accuracy, precision and specificity of classification are improved.

In particular, as a result of extracting features from the fine-tuned AlexNet, OverFeat and VGG feature extraction models, and classifying lesions therefrom (With fine-tuning), it appears that the accuracy, precision and sensitivity are greatly improved.

Furthermore, when RF is applied as a classifier, it appears that the lesion classification ability is better than when using the SVM classifier.

In other words, these results show that in the case of the feature extraction models of AlexNet, OverFeat and VGG in which the last layer was transformed and retrained, the lesion classification ability was improved compared to the original AlexNet, OverFeat and VGG feature extraction models in the pretrained state. Can mean that.

According to the results of Example 1 above, the finely tuned feature extraction models used in various embodiments of the present invention, particularly the feature extraction models of AlexNet, OverFeat and VGG, in which fine adjustments have been performed, have improved lesion classification capability. Accordingly, according to the present invention, by using the finely adjusted feature extraction model as described above, it is possible to provide highly reliable information in lesion diagnosis.

Example 2: Evaluation for combining multiple feature extraction models

Hereinafter, an evaluation result according to whether a plurality of feature extraction models used in various embodiments of the present invention are combined will be described with reference to FIGS. 5A to 5E. 5A to 5E illustrate evaluation results according to combining a plurality of feature extraction models used in various embodiments of the present invention.

At this time, this evaluation was performed on six models having excellent lesion classification ability in Example 1 described above.

More specifically, referring to FIG. 5A, AlexNet fine-tuned to extract features for the fc2 layer ([A]), OverFeat fine-tuned to extract features for the fc2 layer ([O]), and for the fc1 layer. VGG ([V]) fine-tuned to extract features, VGG-verydeep ([Vv]) with an output layer of fc2, ResNet with one output layer ([R]) and Inception (with one output layer) Six models of [I]) were selected.

Features were extracted from the six selected models, and the extracted features were applied to a classifier, and the accuracy, precision and specificity of the lesion classification result were measured. At this time, SVM and RF were used as classifiers.

Referring to FIG. 5B, when a combination of two models selected among six models is used, it is shown that the accuracy, precision, and specificity for classification are generally improved.

More specifically, when the features extracted from each of the two models are combined and applied to SVM and RF to classify the lesion, it appears that the performance for lesion classification is improved compared to the case of using a single model.

In particular, the accuracy of lesion classification is single, except for [VvI], which combines features of VGG-verydeep ([Vv]) with an output layer of fc2 and Inception ([I]) with one output layer. It appears to be better than when using the model.

Referring to FIG. 5C, when a combination of three models selected from among six models is used, an improvement in accuracy, precision, and specificity similar to that when a combination of two models is used is shown.

More specifically, when the features extracted from each of the three models are combined and applied to SVM and RF to classify the lesion, it appears that the performance for lesion classification is improved compared to the case of using a single model.

In particular, the accuracy of lesion classification is as follows: AlexNet ([A]) fine-tuned to extract features for the fc2 layer, VGG ([V]) fine-tuned to extract features for the fc1 layer, and one output layer. In the case of [AVI] that combines the features of Inception ([I]), the specificity of the classification result by the RF model is 99%, which is significantly higher than that of the results using single or two models.

Furthermore, OverFeat ([O]) fine-tuned to extract features for the fc2 layer, VGG ([V]) fine-tuned to extract the features for the fc1 layer, and ResNet ([R]) with one output layer. In the case of [OVR] combining the features of the SVM model, the accuracy of the classification result by the SVM model (94.7%) and the accuracy of the classification result by the RF model (94.0%) are improved.

Referring to FIG. 5D, when a combination of four models selected from among six models is used, remarkable improvements in accuracy, precision and specificity appear.

More specifically, when the features extracted from each of the four models are combined and applied to SVM and RF to classify the lesion, it appears that the performance for lesion classification is improved compared to the case of using a single model.

In particular, the accuracy of the classification result according to all combinations of the four models disclosed in FIG. 5D is at least 90% and at most 94%, which appears to be improved compared to the case of using a single model. Furthermore, the specificity of the classification results according to all combinations of the four models is at least 91% and at most 99%, which appears to be improved compared to the case of using a single model. In addition, it appears that the level of improvement of the lesion classification ability is higher when the classifier of SVM is applied and the classifier of RF is applied.

Referring to FIG. 5D, when a combination of 5 and 6 models selected from among 6 models is used, the classification result according to the combination of 4 or less models described above is displayed at a similar level.

According to the results of Example 2 above, the feature extraction model used in various embodiments of the present invention shows that the performance of lesion classification is improved when a plurality of combinations are used. Accordingly, the present invention provides an auxiliary diagnosis system based on a plurality of feature extraction models, thereby providing a classification result for a lesion with higher accuracy, precision, and specificity than when a single model is used. That is, the present invention has an effect of providing highly reliable information in lesion diagnosis.

Example 3: Evaluation for combining multiple classifiers (evaluation for ensemble classifiers)

Hereinafter, an evaluation result according to the combination of a plurality of classifiers used in various embodiments of the present invention will be described with reference to FIG. 6. 6 is a diagram illustrating evaluation results according to a combination of a plurality of classifiers used in various embodiments of the present invention.

This evaluation is used in the above-described Example 2, AlexNet fine-tuned to extract features for the fc2 layer ([A]), OverFeat fine-tuned to extract features for the fc2 layer ([O]), fc1 VGG ([V]) fine-tuned to extract features for the layer, VGG-verydeep ([Vv]) with an output layer of fc2, ResNet ([R]) with one output layer and one output layer. Inception ([I]) was performed on six models.

More specifically, a feature (or combined feature) extracted from at least one model selected from among the six models was applied to an ensemble classifier in which SVM and RF were combined, from which accuracy, precision and specificity for the lesion classification result. Was evaluated.

At this time, in the classification of the lesion by the ensemble classifier, features are input to the SVM and RF, the probability of the lesion, that is, the lesion probability is calculated by each of the SVM and RF, and the lesion is changed based on the average lesion probability for the calculated lesion probability. It can be done by making a final decision (classification).

6, a lesion classification result of an ensemble classifier consisting of SVM and RF based on a feature extracted by a single model selected from six models or a feature extracted by a combination of two or more models selected from six models Is shown.

More specifically, for both a single feature extraction model or a plurality of feature extraction models, the lesion classification ability appears to be improved when the ensemble classifier is applied, compared to when a single classifier (SVM or RF) is applied.

In particular, the accuracy and specificity in the results of lesion classification by a combination of a plurality of feature extraction models and a plurality of classifiers appear to be increased compared to when a single feature extraction model and a single classifier are used. More specifically, OverFeat ([O]) fine-tuned to extract features for the fc2 layer, VGG ([V]) fine-tuned to extract features for the fc1 layer, and Inception ([I) with one output layer. ]) in the case of [O][V][I] with the combination of three feature extraction models and ensemble classifiers applied, the specificity is 98%, significantly more than when using a single feature extraction model and a single classifier. Appears to have increased. Furthermore, in the case of [Vv][R] with a combination of two feature extraction models of VGG-verydeep ([Vv]) with an output layer of fc2 and ResNet ([R]) with one output layer and an ensemble classifier However, the sensitivity was 92%, which was significantly increased compared to the case of using a single feature extraction model and a single classifier. Also, AlexNet fine-tuned to extract features for the fc2 layer ([A]), VGG fine-tuned to extract features for the fc1 layer, and ResNet ([R]) with one output layer. In the case of [A][V][R] with the combination of the three feature extraction models and the ensemble classifier applied, the accuracy was 94.7%, which was significantly increased compared to the case of using a single feature extraction model and a single classifier. .

According to the results of Example 3 above, the classifier used in various embodiments of the present invention shows that the performance of lesion classification is improved when used in a plurality of combinations. In particular, when a plurality of feature extraction models and a plurality of classifiers are used, the accuracy and specificity of classification are improved. Accordingly, the present invention provides an auxiliary diagnostic system based on a plurality of feature extraction models and a plurality of classifiers, so that lesions are classified with higher accuracy, precision, and specificity than when using a single feature extraction model and a single classifier. Can provide results. That is, the present invention has an effect of providing highly reliable information in lesion diagnosis.

According to the above embodiments, the present invention provides an assistive diagnosis system based on a plurality of models configured to extract features from a medical image and predict whether or not a disease has occurred or a lesion based on this. It has the effect of providing information.

Accordingly, the present invention can overcome a limitation of a conventional auxiliary diagnosis system based on a medical image, which may reduce the reliability of diagnosis since lesion diagnosis results may be different depending on the skill level of a medical staff.

Although the embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and various modifications may be made without departing from the spirit of the present invention. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain the technical idea, and the scope of the technical idea of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

100, 200: device for providing information on lesion diagnosis
110: receiver
120: input
130: output
140: storage unit
150: processor
210: medical image receiving unit
220: feature extraction unit
222: pre-trained feature extraction model
224: Deformed feature extraction model
226: Fine-tuned feature extraction model
228: Retrained feature extraction model
230: lesion classification unit
232: classifier
234: learned classifier
236: retrained classifier
312: Medical imaging
322: Feature extraction model
324: feature
326: Join feature
328: final join feature
332: classifier
334: Lesion probability
335: mean lesion probability
336: classification lesion
337: mean classification lesion probability
338: final lesion
342: ensemble classifier

Claims (24)

delete A method of providing information on lesion diagnosis implemented by a processor,
Receiving a medical image for a target site;
Extracting a feature for the medical image using a plurality of feature extraction models configured to extract a feature extracted during a process for predicting a lesion in the medical image;
Classifying a lesion for the target region in the medical image using a plurality of classifiers configured to classify a lesion based on the extracted features,
Each of the plurality of feature extraction models,
Configured to extract at least one feature different from each other for each of the feature extraction models,
After the step of extracting the feature,
Combining the at least one feature, each extracted by the plurality of feature extraction models, to obtain a combined feature,
Classifying the lesion for the target site,
And classifying the lesion for the target site based on the binding feature by using the plurality of classifiers. The method of claim 2,
The at least one feature,
Is at least two features,
The step of combining the at least one feature,
First combining the at least one feature for each model, so as to obtain a combination feature for each model, and
And second combining the combining features for each model to obtain a final combining feature,
Classifying the lesion for the target site,
Further comprising the step of classifying a lesion for the target site based on the final binding feature by using the plurality of classifiers. A method of providing information on lesion diagnosis implemented by a processor,
Receiving a medical image for a target site;
Extracting a feature for the medical image using a plurality of feature extraction models configured to extract a feature extracted during a process for predicting a lesion in the medical image;
Classifying a lesion for the target region in the medical image using a plurality of classifiers configured to classify a lesion based on the extracted features,
Classifying the lesion within the medical image,
Using a plurality of classifiers, calculating a lesion probability for the feature, and
A method for providing information on lesion diagnosis, comprising the step of classifying a lesion for the target site based on the lesion probability. The method of claim 4,
The lesion is,
Contains a plurality of predetermined lesions,
The step of calculating the lesion probability,
Using a plurality of classifiers, calculating a probability of each of a plurality of lesions for the feature, and
Comprising the step of calculating an average lesion probability for each of the plurality of lesions, based on the probability of each of the plurality of lesions calculated by the plurality of classifiers,
Classifying the lesion for the target site based on the lesion probability,
And determining that the target site has one of the plurality of lesions based on the average lesion probability. A method of providing information on lesion diagnosis implemented by a processor,
Receiving a medical image for a target site;
Extracting a feature for the medical image using a plurality of feature extraction models configured to extract a feature extracted during a process for predicting a lesion in the medical image;
Classifying a lesion for the target region in the medical image using a plurality of classifiers configured to classify a lesion based on the extracted features,
The feature includes a plurality of features different from each other,
The plurality of classifiers,
Including a plurality of ensemble classifiers composed of the plurality of classifiers,
The plurality of ensemble classifiers,
It is configured to classify lesions for different features for each of the ensemble classifiers,
Classifying the lesion within the medical image,
Calculating a lesion probability for each of the plurality of different features using a plurality of the ensemble classifiers, and
And classifying a lesion for the target site based on a lesion probability for each of the plurality of different features. The method of claim 6,
The lesion is,
Contains a plurality of predetermined lesions,
The step of calculating the lesion probability,
Calculating a probability of each of a plurality of lesions for each of the plurality of different features, using a plurality of the ensemble classifiers, and
Comprising the step of calculating the average lesion probability for each of the plurality of lesions, based on the probability of each of the plurality of lesions for each of the plurality of different features,
Classifying the lesion for the target site based on the lesion probability,
And determining that the target site has one of the plurality of lesions based on the average lesion probability. A method of providing information on lesion diagnosis implemented by a processor,
Receiving a medical image for a target site;
Extracting a feature for the medical image using a plurality of feature extraction models configured to extract a feature extracted during a process for predicting a lesion in the medical image;
Classifying a lesion for the target region in the medical image using a plurality of classifiers configured to classify a lesion based on the extracted features,
The feature extraction model,
A method of providing information on lesion diagnosis, which is a model modified to extract the feature before final prediction for the lesion. The method of claim 8,
The feature extraction model,
It is a pre-trained CNN (convolutional neural network) model composed of a plurality of layers,
A method of providing information on lesion diagnosis, wherein one of the plurality of layers is a model transformed into an output layer. The method of claim 8,
The feature extraction model,
Receiving a medical image for learning, and
A method for providing information on lesion diagnosis, which is a feature extraction model fine-tuned through the step of learning to extract features based on the training medical image for the feature extraction model having the transformed output layer. The method of claim 10,
The feature extraction model,
Determining a selective learning medical image having a variance of at least a predetermined level among the training medical images, and
A method for providing information on lesion diagnosis, which is a feature extraction model re-learned through the step of learning to extract features based on the medical image for selective learning with respect to the finely adjusted feature extraction model. A method of providing information on lesion diagnosis implemented by a processor,
Receiving a medical image for a target site;
Extracting a feature for the medical image using a plurality of feature extraction models configured to extract a feature extracted during a process for predicting a lesion in the medical image;
Classifying a lesion for the target region in the medical image using a plurality of classifiers configured to classify a lesion based on the extracted features,
The classifier,
Receiving a feature extracted by the feature extraction model for a medical image for learning including a predetermined lesion, and
A method for providing information on lesion diagnosis, which is a classifier learned through the step of classifying the predetermined lesion based on the features extracted for the training medical image. The method of claim 12,
The classifier,
Determining a selective learning medical image having a variance of at least a predetermined level among the training medical images, and
The method for providing information on lesion diagnosis, which is a classifier relearned through the step of classifying the predetermined lesion based on features of the medical image for selective learning with respect to the learned classifier. The method according to any one of claims 2 to 13,
The medical image,
A method of providing information on lesion diagnosis, which is at least one of ultrasound images, X-ray images, computed tomography (CT) images, magnetic resonance imaging (MRI) images, endoscopy images, and positron emission tomography (PET) images . The method according to any one of claims 2 to 13,
The plurality of feature extraction models are at least two deep learning models among AlexNet, OverFeat, VGG, VGG-verydeep, ResNet, GoogleNet, and Inception, a method for providing information on lesion diagnosis. The method according to any one of claims 2 to 13,
The plurality of classifiers are SVM (Support Vector Machine), RF (Random Forests), LDA (linear discriminant analysis), QDA (quadratic discriminant analysis), decision tree (decision tree), XG Boost (extreme gradient boosting), logistic regression Analysis (logistic regression), logistic regression, NN (nearest neighbor) and GMM (Gaussian mixture model) at least two classification models, a method of providing information on lesion diagnosis. delete A device for providing information on lesion diagnosis implemented by a processor,
A receiving unit configured to receive a medical image for a target site, and
And a processor configured to communicate with the receiving unit,
The processor,
Using a plurality of feature extraction models configured to extract features extracted during the process of predicting lesions in the medical image, extracting features for the medical image,
Configured to classify a lesion for the target site in the medical image using a plurality of classifiers configured to classify a lesion based on the extracted features,
Each of the plurality of feature extraction models,
Configured to extract at least one feature different from each other for each of the feature extraction models,
The processor,
Combine the at least one feature, each extracted by the plurality of feature extraction models, to obtain a binding feature, and classify a lesion for the target site based on the binding feature using the plurality of classifiers. Configured, a device for providing information on lesion diagnosis. The method of claim 18,
The at least one feature,
Is at least two features,
The processor,
First combining the at least one feature for each model to obtain a combination feature for each model,
For providing information on lesion diagnosis, further configured to secondly combine the binding characteristics for each model to obtain a final binding characteristic, and classify a lesion for the target site based on the final binding feature using the plurality of classifiers device. A device for providing information on lesion diagnosis implemented by a processor,
A receiving unit configured to receive a medical image for a target site, and
And a processor configured to communicate with the receiving unit,
The processor,
Using a plurality of feature extraction models configured to extract features extracted during the process of predicting lesions in the medical image, extracting features for the medical image,
Configured to classify a lesion for the target site in the medical image using a plurality of classifiers configured to classify a lesion based on the extracted features,
The processor,
A device for providing information on lesion diagnosis, further configured to calculate a lesion probability for the feature using a plurality of classifiers, and classify a lesion for the target site based on the lesion probability. The method of claim 20,
The lesion is,
Contains a plurality of predetermined lesions,
The processor,
A probability of each of a plurality of lesions for the feature is calculated using a plurality of classifiers, and an average lesion probability for each of the plurality of lesions is calculated based on the probability of each of the plurality of lesions calculated by the plurality of classifiers, The device for providing information on lesion diagnosis, further configured to determine to have one of the plurality of lesions for the target site based on the average lesion probability. A device for providing information on lesion diagnosis implemented by a processor,
A receiving unit configured to receive a medical image for a target site, and
And a processor configured to communicate with the receiving unit,
The processor,
Using a plurality of feature extraction models configured to extract features extracted during the process of predicting lesions in the medical image, extracting features for the medical image,
Configured to classify a lesion for the target site in the medical image using a plurality of classifiers configured to classify a lesion based on the extracted features,
The feature includes a plurality of features different from each other,
The plurality of classifiers,
Including a plurality of ensemble classifiers composed of the plurality of classifiers,
The plurality of ensemble classifiers,
It is configured to classify lesions for different features for each of the ensemble classifiers,
The processor,
Using a plurality of the ensemble classifier, further configured to calculate a lesion probability for each of the plurality of different features, and classify the lesion for the target site based on the lesion probability for each of the plurality of different features , Device for providing information on lesion diagnosis. The method of claim 22,
The lesion is,
Contains a plurality of predetermined lesions,
The processor,
Using a plurality of the ensemble classifier, calculating the probability of each of a plurality of lesions for each of a plurality of different features, and the plurality of lesions based on the probability of each of the plurality of lesions for each of the plurality of different features The device for providing information on lesion diagnosis, further configured to calculate an average lesion probability for each and determine that the target site has one of the plurality of lesions based on the average lesion probability. delete

Images