CN114332577A - Colorectal cancer image classification method and system combining deep learning and radiomics - Google Patents

Abstract

The present invention proposes a colorectal cancer image classification method and system combining deep learning and radiomics, aiming at the small sample problem of deep learning model training, making full use of existing data for data enhancement (rotation, translation, image transformation, etc.) In order to solve the problem of time-consuming and labor-intensive manual labeling by doctors, an automatic segmentation network model of deep learning is introduced to automatically label regions of interest from images. In view of the problem that the features extracted by the deep learning model are poorly interpretable and the feature information is not comprehensive enough, the combination of radiomics features, deep learning features and clinicopathological information is used to obtain more and more comprehensive feature information, and further improve the performance of radiomics. Classification accuracy and reliability.

Description

Colorectal cancer image classification method and system combining deep learning and radiomics

technical field

The invention belongs to the technical field of medical image processing and image classification, and in particular relates to a colorectal cancer image classification method and system combining deep learning and radiomics.

Background technique

1. Radiomics solution: Radiomics mainly uses computer software to extract a large number of high-dimensional quantitative image features from CT, MRI and PET images with high throughput; the process includes data collection, image segmentation, feature extraction, and feature selection. And model building, relying on deeper mining, prediction and analysis of massive image data information to assist physicians in making the most accurate image classification. In practical applications, radiomic lesions require manual marking by doctors, which is not only time-consuming but also subject to subjective bias. In addition, the lack of a standardized and standard process and quality control system for the calculation of target characteristics using quantitative methods limits the performance of such methods.

2. The scheme of deep learning: deep learning refers to a method of combining low-level features to form more abstract high-level features or categories, and then learning effective features from a large amount of input data, and using these features for classification, regression and information retrieval. technology. Models can automatically learn to extract and select image features and make predictions, thereby mining information in images more comprehensively and deeply. There are many types of models, and currently in the field of medical imaging, convolution neural network (CNN) is the most widely used. Using convolutional networks makes the training and inference process very computationally intensive. After completing model training, deep learning can realize fully automatic analysis of images, but this advantage is based on higher data acquisition costs. Deep learning requires more collection and labeling of more data, perhaps ten or more times that of radiomics.

As mentioned above, the shortcomings of the existing colorectal cancer medical image classification mainly include:

1. Radiomics requires doctors to manually label data, which is time-consuming and subject to subjective bias. Imaging equipment from different manufacturers still lacks uniform standards in terms of scanning parameters and reconstruction algorithms, and the extracted features are also phenotypic features, which are not deep enough.

2. The deep learning model requires a large amount of data for training, occupies a large amount of computing resources, requires high hardware equipment, and the extracted deep learning features are poorly interpretable.

3. The extracted features are not comprehensive enough. Only radiomics features or deep learning features are used, not a combination of the two.

SUMMARY OF THE INVENTION

In order to make up for the gaps and deficiencies of the prior art, the present invention proposes a colorectal cancer image classification method and system combining deep learning and radiomics. Its main designs include:

1. For the small sample problem of deep learning model training, make full use of existing data for data enhancement (rotation, translation, image transformation, etc.)

2. In order to solve the problem of time-consuming and labor-intensive manual labeling by doctors, an automatic segmentation network model of deep learning is introduced to automatically label regions of interest from images.

3. In view of the problem that the features extracted by the deep learning model are poorly interpretable and the feature information obtained is not comprehensive enough, the combination of radiomics features, deep learning features and clinicopathological information is used to obtain more and more comprehensive feature information to further improve the imaging group. classification accuracy and reliability.

The present invention specifically adopts the following technical solutions:

A colorectal cancer image classification method combining deep learning and radiomics, characterized in that it includes the following steps:

Step S1: Data preprocessing: using the data enhancement method to amplify the colorectal cancer image data; use the manually labeled data to train the segmentation network, and use the trained model to automatically segment the region of interest from the image to obtain more contains labeled data;

Step S2: Feature extraction: Extract relevant omics features from abdominal CT through the open source Python software package Pyradiomics; use the resnet training model to select a model with the best results, and use this model to extract deep learning features;

Step S3: Feature selection: remove the relevant features, calculate the Pearson correlation matrix to eliminate the highly correlated features (p>0.90); and use the method of recursive feature elimination to sort the remaining features according to the predictive ability;

Step S4: Feature fusion: using natural language processing technology to convert the clinicopathological information provided by the doctor; fuse the radiomics features, deep learning features and clinicopathological information to obtain more comprehensive feature information;

Step S5: Using ensemble learning, using classifiers including support vector machine SVM, Bayesian discriminator, logistic regression discriminator and Lasso regression, vote on the classification results of each classifier during prediction, and select the best model to use. to perform the final image classification.

Further, in step S1: first, perform data enhancement on the existing colorectal cancer image data to obtain more abdominal images, including inputting the amplified data into the depth through image rotation, image translation and image conversion. The learning network U-Net is trained. After the model is trained, it can be used to automatically segment the region of interest, and manually fine-tune the lesion region obtained by the segmentation model to obtain more labeled data.

Further, in step S3: feature selection first traverses all features, calculates Pearson correlation coefficients in pairs, when P>0.90, removes one of them randomly, so that the features after dimension reduction do not have high similarity; then use recursive features Elimination, the remaining features are sorted according to their predictive power.

Further, in step S4: clinical pathological information feature selection is performed by stepwise discriminant regression, and all features are introduced one by one for testing one by one; when the originally introduced feature variable becomes no longer significant due to the introduction of subsequent variables, it is eliminated; Repeat until neither significant variables are selected into the equation nor non-significant independent variables are eliminated from the regression equation.

And, a colorectal cancer image classification system combining deep learning and radiomics, characterized in that: based on a computer system, including:

The data preprocessing module adopts the data enhancement method to amplify the colorectal cancer image data; uses the manually labeled data to train the segmentation network, and uses the trained model to automatically segment the region of interest from the image to obtain more labeled data. The data;

The feature extraction module extracts relevant omics features from abdominal CT through the open source Python software package Pyradiomics; uses the resnet training model to select the model with the best results, and uses this model to extract deep learning features;

The feature selection module is used to remove relevant features, calculate the Pearson correlation matrix to eliminate high correlation features (p>0.90); and use the method of recursive feature elimination to sort the remaining features according to the predictive ability;

Feature fusion module, which uses natural language processing technology to convert clinical pathological information provided by doctors; integrates radiomics features, deep learning features and clinical pathological information to obtain more comprehensive feature information;

The integrated learning module uses classifiers including support vector machine SVM, Bayesian discriminator, logistic regression discriminator and Lasso regression to vote on the classification results of each classifier during prediction, and select the best model for final execution. image classification.

Further, in the data preprocessing module, the data enhancement adopts the methods of image rotation, image translation and image conversion, and the augmented data is input into the deep learning network U-Net for training.

Further, in the feature selection module, firstly traverse all the features, calculate the Pearson correlation coefficients in pairs, when P>0.90, remove one of them randomly, so that the features after dimension reduction do not have high similarity; then use recursion Feature elimination, sorting the remaining features according to their predictive power.

Further, in the feature fusion module, the selection processing of clinical pathological information features is to introduce all the features one by one through step-by-step discriminant regression and test them one by one; When it is significant again, it is eliminated; it is repeated until neither significant variables are selected into the equation nor non-significant independent variables are eliminated from the regression equation.

Compared with the prior art, the main design points of the present invention and its preferred solution include:

1. For the task of colorectal cancer medical image classification, a network that integrates radiomics features, deep learning features and clinical features is designed, which can realize the fusion of multiple features, not only interpretable radiomics features , which also incorporates deeper abstraction of deep learning features and information provided by doctors and patients.

2. Fully combines radiomics and deep learning. The application of deep learning in the data preprocessing stage, including data augmentation and automatic segmentation of regions of interest, can effectively alleviate the limitations of small data volume and time-consuming doctor labeling. Deep learning models are used to extract high-dimensional effective features, natural language processing techniques are used to analyze clinicopathological information, and radiomics can extract rich phenotypic features.

3. Integrated learning modeling and the use of multiple classifiers make the final result more comprehensive.

Its advantages over existing technologies include:

1. Existing methods use radiomics features or deep learning features alone, and the application of feature information is not comprehensive enough. Radiomics requires doctors to manually label feature regions that are time-consuming and labor-intensive, and the extracted features are not comprehensive enough for phenotypic features. Deep learning requires A large amount of data and its extracted features are deep abstract features, and the interpretability is not strong.

The present invention and its preferred solution introduce interpretable radiomics features, deep abstract features and clinical pathological features, which not only solves the drawbacks of incomplete feature extraction and poor interpretability, but also solves the problem of large quantity demand and manual labeling. Time-consuming and labor-intensive, etc.

2. Most of the clinical information currently added are some quantitative characteristics such as gender and age, which are relatively limited.

The present invention and its preferred solution introduce the method of natural language processing (NLP) to process clinical pathological information, which makes the imported information more comprehensive. NLP technology can process large amounts of text data, allowing machines to understand and utilize richer text information.

Description of drawings

1 is a schematic diagram of a classification network framework designed by an embodiment of the present invention;

2 is a schematic diagram of a U-Net network structure adopted in an embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a Resnet network adopted in an embodiment of the present invention.

Detailed ways

In order to make the features and advantages of this patent more obvious and easy to understand, the following specific examples are given and described in detail as follows:

As shown in Figure 1, the colorectal cancer image classification method and system combining deep learning and radiomics proposed in this embodiment specifically includes the following solutions:

(1) Data preprocessing: use data enhancement methods (rotation, translation, image transformation, etc.) to fully augment the existing data; use the existing manually labeled data to train the segmentation network, and use the trained model from Automatically segment regions of interest in images to obtain more labeled data.

(2) Feature extraction: Extract relevant omics features from abdominal CT through the open source Python software package Pyradiomics; the resnet training model selects the model with the best results, and uses this model to extract deep learning features.

(3) Feature fusion: natural language processing (NLP) technology is used to convert clinicopathological information provided by doctors. Integrate radiomic features, deep learning features and clinicopathological information to obtain more comprehensive feature information.

(4) Feature selection: Remove the relevant features, calculate the Pearson correlation matrix to eliminate the highly correlated features (p>0.90); and use the recursive feature elimination method to sort the remaining features according to the predictive ability.

(5) Classifier: Ensemble Learning is adopted, including support vector machine (SVM), Bayesian discriminator, logistic regression discriminator and Lasso regression and other classifiers. When predicting, the classification results of each classifier are voted to select the best model.

2. Detailed design

(1) Classification network based on the combination of deep learning and radiomics

The classification network framework designed in this embodiment is shown in Figure 1. First, data enhancement is performed on the original data to obtain more abdominal images, mainly through image rotation, image translation, and image conversion. The amplified data is input into the deep learning network U-Net for training. After the model is trained, it can be used to automatically segment the region of interest, and the lesion region obtained by the segmentation model can be manually fine-tuned by professional doctors to obtain more labeled data. This method greatly reduces the time for manual annotation by doctors.

Radiomics features were extracted from abdominal CT through an open-source Python software package Pyradiomics, and then feature selection was performed. The deep features extracted by deep learning network and the clinical features processed by NLP were modeled by ensemble learning. Using ensemble learning (Ensemble Learning), including support vector machine (SVM), Bayesian discriminator, logistic regression discriminator and Lasso regression and other classifiers. When predicting, the classification results of each classifier are voted to select the best model.

The number of extracted radiomic features may vary from hundreds to tens of thousands, and not every feature is associated with the clinical problem to be solved; on the other hand, in practice, due to the relatively large number of features, the If the number is small, it is easy to lead to the phenomenon of overfitting of the subsequent model, thus affecting the accuracy of the model. Feature selection first traverses all features, and calculates the Pearson correlation coefficient in pairs. When P>0.90, one of them is randomly removed. This method can make the features after dimension reduction not have high similarity; The remaining features are sorted.

The feature selection of clinicopathological information is through stepwise discriminant regression, and all features are introduced one by one and tested one by one. When the originally introduced feature variable becomes no longer significant due to the introduction of later variables, it is eliminated. Repeat until neither significant variables are selected into the equation nor non-significant independent variables are eliminated from the regression equation.

(2) Automatic segmentation network U-Net

The U-Net network consists of an encoding path that captures contextual information and a decoding path. As shown in Figure 2, the feature map of the encoder is spliced with the up-sampling feature map of the decoder at each stage through a skip-layer connection, so that form a U-shaped structure. The decoder learns the features lost due to the encoder pooling by making skip connections at each stage.

(3) Feature extraction by deep learning network

The Resnet-based architecture is used to train and extract high-dimensional deep learning features. Unlike traditional convolutional neural networks, each layer in the Resnet network structure accepts the output of the previous layer, making the network more accurate and efficient. The deep learning features are extracted before the output layer of the network, the output layer is removed, and the high-dimensional features obtained from the last layer of the hidden layer are used as the output deep learning features, as shown in Figure 3.

The above program design solution provided in this embodiment can be stored in a computer-readable storage medium in a coded form, and implemented in the form of a computer program, and the basic parameter information required for calculation is input through computer hardware, and the calculation result is output. .

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations of methods, apparatus (apparatus), and computer program products according to embodiments of the invention. It will be understood that each process in the flowchart, and combinations of processes in the flowchart, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce A device that implements the functions specified in one or more of the flow charts.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The device implements the functions specified in one or more of the flowcharts.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that Instructions provide steps for implementing the functions specified in a flow or flows of the flowchart.

The above are only preferred embodiments of the present invention, and are not intended to limit the present invention in other forms. Any person skilled in the art may use the technical content disclosed above to make changes or modifications to equivalent changes. Example. However, any simple modifications, equivalent changes and modifications made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solutions of the present invention still belong to the protection scope of the technical solutions of the present invention.

This patent is not limited to the above-mentioned best embodiment, anyone can come up with other various forms of colorectal cancer image classification methods and systems combining deep learning and radiomics under the inspiration of this patent. Equivalent changes and modifications made within the scope of the patent shall fall within the scope of this patent.

Claims (8)

1. a colorectal cancer image classification method combining deep learning and radiomics, is characterized in that, comprises the following steps: Step S1: Data preprocessing: using the data enhancement method to amplify the colorectal cancer image data; use the manually labeled data to train the segmentation network, and use the trained model to automatically segment the region of interest from the image to obtain more contains labeled data; Step S2: Feature extraction: Extract relevant omics features from abdominal CT through the open source Python software package Pyradiomics; use the resnet training model to select a model with the best results, and use this model to extract deep learning features; Step S3: Feature selection: remove the relevant features, calculate the Pearson correlation matrix to eliminate the highly correlated features (p>0.90); and use the method of recursive feature elimination to sort the remaining features according to the predictive ability; Step S4: Feature fusion: using natural language processing technology to convert the clinicopathological information provided by the doctor; fuse the radiomics features, deep learning features and clinicopathological information to obtain more comprehensive feature information; Step S5: Using ensemble learning, using classifiers including support vector machine SVM, Bayesian discriminator, logistic regression discriminator and Lasso regression, vote on the classification results of each classifier during prediction, and select the best model to use. to perform the final image classification. 2. The colorectal cancer image classification method combining deep learning and radiomics according to claim 1, characterized in that: in step S1: firstly, data enhancement is performed on the existing colorectal cancer image data to obtain more The abdominal image, including the way of image rotation, image translation and image transformation, input the augmented data into the deep learning network U-Net for training, after the model is trained, it can be used to automatically segment the region of interest, and manually fine-tune the segmentation The lesion area obtained by the model can obtain more labeled data. 3. The colorectal cancer image classification method combining deep learning and radiomics according to claim 1, characterized in that: in step S3: feature selection first traverses all features, and calculates Pearson correlation coefficients in pairs, when P > At 0.90, one of them is randomly removed so that the features after dimension reduction do not have high similarity; then recursive feature elimination is used to sort the remaining features according to their predictive ability. 4. The colorectal cancer image classification method combining deep learning and radiomics according to claim 1, characterized in that: in step S4: clinical pathological information feature selection is by stepwise discriminant regression, and all features are sequentially introduced to perform one by one. Test; when the originally introduced characteristic variable becomes no longer significant due to the introduction of later variables, it is eliminated; it is repeated until neither significant variables are selected into the equation nor insignificant independent variables are eliminated from the regression equation. 5. A colorectal cancer image classification system combining deep learning and radiomics, characterized in that: based on a computer system, comprising: The data preprocessing module adopts the data enhancement method to amplify the colorectal cancer image data; uses the manually labeled data to train the segmentation network, and uses the trained model to automatically segment the region of interest from the image to obtain more labeled data. The data; The feature extraction module extracts relevant omics features from abdominal CT through the open source Python software package Pyradiomics; uses the resnet training model to select the model with the best results, and uses this model to extract deep learning features; The feature selection module is used to remove relevant features, calculate the Pearson correlation matrix to eliminate high correlation features (p>0.90); and use the method of recursive feature elimination to sort the remaining features according to the predictive ability; Feature fusion module, which uses natural language processing technology to convert clinical pathological information provided by doctors; integrates radiomics features, deep learning features and clinical pathological information to obtain more comprehensive feature information; The integrated learning module uses classifiers including support vector machine SVM, Bayesian discriminator, logistic regression discriminator and Lasso regression to vote on the classification results of each classifier during prediction, and select the best model for final execution. image classification. 6. The colorectal cancer image classification method combining deep learning and radiomics according to claim 5, characterized in that: in the data preprocessing module, data enhancement adopts image rotation, image translation and image conversion. , and input the augmented data into the deep learning network U-Net for training. 7. The colorectal cancer image classification method combining deep learning and radiomics according to claim 5, characterized in that: in the feature selection module, all features are first traversed, and Pearson correlation coefficients are calculated pairwise, and when P When >0.90, one of them is randomly removed so that the features after dimension reduction do not have high similarity; then recursive feature elimination is used to sort the remaining features according to their predictive ability. 8. The colorectal cancer image classification method combining deep learning and radiomics according to claim 5, characterized in that: in the feature fusion module, the selection processing of clinical pathological information features is, by stepwise discriminant regression , all the features are introduced in turn and tested one by one; when the originally introduced feature variables become no longer significant due to the introduction of subsequent variables, they are eliminated; repeat until neither significant variables are selected into the equation nor insignificant independent variables removed from the regression equation.

Images