CN116013449A - Auxiliary prediction method for cardiomyopathy prognosis by fusing clinical information and magnetic resonance image - Google Patents

Abstract

The invention relates to a cardiomyopathy prognosis auxiliary prediction method fusing clinical information and magnetic resonance images, which comprises the steps of firstly adopting a Relief feature selection algorithm to screen clinical indexes, then carrying out feature fusion on the screened clinical indexes and cardiac MRI images, and constructing a prediction neural network model MM-Net, wherein the method comprises two independent feature extraction branches: and finally, carrying out fusion processing on the high-dimensional characteristic information extracted by the two branches respectively, outputting a final cardiac MRI image classification result, and assisting in predicting whether a severe prognosis event occurs to the dilated cardiomyopathy patient. Experimental results show that after clinical indexes are introduced into the auxiliary prediction method, various indexes of the auxiliary prediction are improved, and the adopted feature layer fusion strategy performance is excellent.

Description

Auxiliary prediction method for cardiomyopathy prognosis by fusing clinical information and magnetic resonance image

Technical Field

The invention relates to the field of medical image processing, in particular to an auxiliary prediction method for myocardial disease prognosis by fusing clinical information and magnetic resonance images.

Background

Dilated cardiomyopathy is a common chronic heart disease, is the type of cardiomyopathy with highest worldwide morbidity, and accounts for 90% of cardiomyopathy, and can cause heart failure, thromboembolism and sudden death. The multi-parameter and multi-sequence imaging of the cardiac magnetic resonance image can predict the prognosis situation of the dilated cardiomyopathy patient from the changes of the cardiac morphology and the structure, judge the myocardial histology abnormality of the dilated cardiomyopathy patient according to the LGE, the T1 mapping and the ECV, and provide a new imaging index for the dilated cardiomyopathy prognosis. Therefore, cardiac MRI has a very important role in prognosis prediction and assessment of dilated cardiomyopathy. However, related studies indicate that prognosis of dilated cardiomyopathy patients is closely related to factors such as gender, age, heart function classification of new york heart association (New York Heart Association, NYHA), LVEF, hypertension, diabetes and the like, and these clinical indexes can provide important information for prognosis prediction of dilated cardiomyopathy, so as to assist clinicians in carrying out prognosis risk assessment to make rapid and accurate judgment.

Traditional medical image classification methods include K-Nearest Neighbor (KNN), na-ve bayesian classification (Na-ary Bayes), support vector machines (Support Vector Machine, SVM), back Propagation (BP) neural networks, and the like. The popular deep learning classification method in recent years overcomes the problems of manual feature selection, weak adaptability and the like of the traditional method, and gradually becomes a main stream method for medical image classification.

In 1998, students proposed classical LeNet for handwriting digital recognition and classification initially with an accuracy of 98%. Deep convolutional neural networks have not attracted much attention for a long time after the appearance of LeNet. Until 2012, alex et al designed an 8-layer convolutional neural network AlexNet, greatly improving classification performance and leading the capture of the crown in the 2012 ImageNet race, thanks to the substantial increase in the internet and multimedia data, the significant improvement in the computer hardware performance, and the optimization of the training method. VGGNet by the university of Oxford computer vision group and Google deep Inc. obtained the first name of the image Net game location item and the second name of the classification item in 2014. VGG16 has been proposed by researchers to predict cancer grade for images of pathological sections of patients with cystoma. Still other scholars have proposed a concentrated VGG19 for breast cancer enhancement CT image classification. The GoogleNet obtains a first name (a second name is VGGNet) of an image Net classification project in 2014, the network adopts an acceptance structure to extract multi-scale features and expand the width and depth of the network, and a learner adopts an improved GoogleNet to classify the positive side of a chest X-ray picture, so that the accuracy is close to 100%. Foreign scholars have used GooLeNet successfully for multi-classification of MRI images of alzheimer's disease patients. When the network reaches a certain depth, the network layer number is increased, and the network performance is not increased and reduced. To address performance degradation, microsoft institute He Kaiming et al proposed a res et that successfully trained a 152-layer depth convolutional network by continuously stacking residual blocks, and subsequently used res et to classify alzheimer's disease patients and normal on 3D MRI. Wu Yunfeng et al propose an admission-ResNet classification of CT images of the lungs, which effectively reduces the parameters of the model. The DenseNet expands the cross-layer connection mode of ResNet, each layer in the network is directly connected with the front layer in the dense connection module, so that the reuse of the characteristics is realized, the number of network parameters is reduced, and the gradient disappearance is relieved. Subsequently, a 121-layer DenseNet was proposed to classify 14 diseases of chest X-ray pictures. And classifying the lung X-ray images by combining DenseNet with CapsNet to diagnose the new coronavirus infection.

The prior art has the following defects:

1. the existing model is a single-mode model, is mainly used for training magnetic resonance image data, is not designed into a multi-source data fusion model for clinical indexes and magnetic resonance images, and ignores the effect of clinical information.

2. In feature extraction, the purpose of high-dimensional feature extraction is achieved by adopting continuously stacked cavity convolution, so that training difficulty is high, gradient disappears and computational complexity is too high.

3. In the feature fusion, the original data is directly fused (i.e. data layer fusion) without any feature extraction, so that redundant information is excessive and the model cannot be processed in real time.

Disclosure of Invention

Aiming at the defects of the prior art, the invention provides an auxiliary prediction method for the prognosis of cardiomyopathy by fusing clinical information and magnetic resonance images, which firstly adopts a Relief feature selection algorithm to screen clinical indexes, then carries out feature fusion on the screened clinical indexes and cardiac MRI images to construct a prediction neural network MM-Net, and comprises two independent feature extraction branches: the clinical feature branch and the image feature branch are respectively subjected to feature extraction of clinical indexes and cardiac MRI images, and finally high-dimensional feature information respectively extracted by the two branches is subjected to fusion processing, and a final cardiac MRI image classification result is output to assist in predicting whether a severe prognosis event occurs in an dilated cardiomyopathy patient, and the method specifically comprises the following steps:

step 1: clinical index screening based on a Relief feature selection algorithm, comprising:

step 11: the collected clinical indexes form a clinical index data set, the clinical indexes in the clinical index data set are subjected to data cleaning and preprocessing, data with larger index deletion degree are deleted, data with smaller deletion degree are supplemented, and then non-numerical data items are converted;

step 12: sequentially calculating the weights of 18 clinical indexes in the clinical index data set by adopting a Relief characteristic selection algorithm, and obtaining the correlation degree of each index and the dilated cardiomyopathy prognosis through sequencing;

step 13: eliminating clinical indexes with negative weights, and reserving clinical indexes with positive weights;

step 2: the clinical characteristic branch extracts the clinical index high-dimensional characteristic, the clinical index with the weight of positive value reserved in the step 13 is sent into the clinical characteristic branch, the clinical characteristic branch comprises three fully connected layers with the number of 16, 32 and 64 neurons which are cascaded in sequence, and the high-dimensional characteristic of the clinical index is obtained after the clinical characteristic branch is passed;

step 3: inputting the acquired cardiac MRI image set into the image feature branch, extracting high-dimensional classification features of the dilated cardiomyopathy MRI image through an encoder-decoder structure of the image feature branch, specifically comprising:

step 31: first, preprocessing cardiac MRI image data, and processing the space between the original image and the marked image to 1.0X1.0 mm ² Normalizing the pixels to be distributed at [ -1.0,1.0]And enhancing the positive sample data by a rotation and translation operation;

step 32: shallow layer feature extraction is carried out through 2 layers of common convolution with the size of 3 multiplied by 3, which is cascaded at the front end of the encoder;

step 33: the encoder further comprises 6 depth separation residual error modules SRM which are sequentially cascaded, a fast feed-forward scanning from bottom to top is constructed to perform feature extraction and fast increase receptive fields, a residual error learning strategy is adopted, the depth separation residual error modules SRM are connected by residual errors, multi-scale feature extraction is performed in each depth separation residual error module by utilizing a separation convolution layer with the sequential cascade void ratio of 3, 5 and 7 and the convolution kernel size of 3 multiplied by 3, and shallow layer features extracted in the step 32 are input into the depth separation residual error modules to obtain cross-task features;

step 34: inputting the cross-task features into a decoder for decoding, introducing an attention mechanism at the decoder part, adopting a residual attention module RAM for global information expansion from top to bottom, pertinently extracting features with important guiding effect on classification tasks, enhancing the effect of effective features and inhibiting redundant information;

step 35: the last layer of the decoder reduces the feature dimension by using a global average pooling layer with a pooling window of 3 multiplied by 3 size to obtain the high-dimensional feature of the MRI image of the heart;

step 4: and fusing and predicting the high-dimensional features of the clinical indexes and the high-dimensional features of the cardiac MRI images by adopting a multi-source information fusion strategy, wherein the fusion strategy is feature layer fusion, splicing the high-dimensional features of the two mode data in the image feature branches and the clinical feature branches, sending the spliced features into two layers of cascaded full-connection layers, and simultaneously, adopting Focal loss to relieve the problem of unbalanced class and output a final classification result.

Compared with the prior art, the invention has the beneficial effects that:

1. in the prior art, clinical information is not considered to be fused into the prediction model when classification prediction is carried out, a Relief characteristic selection algorithm is introduced, important clinical indexes which are related to the disease prediction height are screened out to be fused into the prognosis prediction model, clinical factors which are closely related to the dilated cardiomyopathy prognosis, such as gender, age, heart function classification, hypertension and the like are fused with a heart magnetic resonance image, and the accuracy of auxiliary prediction is improved.

2. The invention adopts the encoder-decoder structure, the depth separation convolution module and the residual error attention module to extract the image characteristics, pertinently extracts the characteristics with important guiding function on the classification task, enhances the function of the effective characteristics and suppresses redundant information, simultaneously reduces the training difficulty of a network by utilizing a residual error network structure, and effectively avoids the gradient vanishing phenomenon.

3. The invention adopts the feature layer fusion strategy, can reserve enough important information in two different data, can mutually supplement and improve the classification result, can obtain the most accurate prediction result, and makes up the defects of data layer fusion and decision layer fusion.

Drawings

FIG. 1 is a diagram of a network architecture of MM-Net constructed in accordance with the present invention;

fig. 2 is a schematic structural diagram of a depth separation residual module SRM;

FIG. 3 is a schematic diagram of the structure of a residual attention module RAM;

FIG. 4 illustrates three different multi-source information fusion strategies;

fig. 5 shows experimental results of the Relief feature selection algorithm.

Detailed Description

The objects, technical solutions and advantages of the present invention will become more apparent by the following detailed description of the present invention with reference to the accompanying drawings. It should be understood that the description is only illustrative and is not intended to limit the scope of the invention. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the present invention.

The MM-Net network of the present invention means: multi-mode Network, multi-modal neural Network.

The ROI of the present invention refers to: region of Interest, a region of interest.

The LGE of the present invention refers to: (late gadolinium enhancement gadolinium agents delay late intensification.

T1 mapping of the present invention means: longitudinal relaxation time quantitative imaging.

The following detailed description refers to the accompanying drawings.

Aiming at the defects existing in the prior art, the invention provides a method for predicting the prognosis of dilated cardiomyopathy by fusing clinical information and magnetic resonance images. According to the method, firstly, a Relief characteristic selection algorithm is adopted to screen clinical indexes, and then the screened clinical indexes are effectively fused with a heart MRI image.

As shown in fig. 1, the constructed predictive neural network MM-Net includes two independent feature extraction branches: and finally, carrying out fusion processing on the high-dimensional characteristic information extracted by the two branches respectively, outputting a final cardiac MRI image classification result, and assisting in predicting whether a severe prognosis event occurs to the dilated cardiomyopathy patient.

Step 11: and (3) carrying out data cleaning and preprocessing on the acquired clinical indexes, deleting the data with larger index deletion degree, supplementing the data with smaller index deletion degree, then converting the non-numerical data items, for example, mapping the sex into a male and mapping the sex into a female, and finally normalizing each item of data between 0 and 1, so as to reduce errors caused by numerical difference.

Step 12: and sequentially calculating the weights of 18 clinical indexes in the data set by adopting a Relief characteristic selection algorithm, and obtaining the correlation degree of each index and the dilated cardiomyopathy prognosis through sequencing.

The 18 clinical indexes include gender, age, height, weight, NYHA cardiac function classification and the like.

Step 13: eliminating clinical indexes with negative weights, and reserving clinical indexes with positive weights;

step 2: and (3) extracting clinical index features by using a clinical feature branch, and sending the clinical index with the weight of positive value reserved in the step (13) into the clinical feature branch, wherein the clinical feature branch comprises three fully-connected layers with the neuron numbers of 16, 32 and 64 which are sequentially cascaded, and the clinical data features are obtained after the clinical feature branch is passed.

In the clinical index branch, because the input clinical index is one-dimensional data, the full-connection layer is adopted for feature extraction, and each full-connection layer contains a ReLU activation function, so that the gradient disappearance problem is relieved, and the model training is more stable and converged. Through three cascaded full-connection layers, the nonlinear capability of the model is enhanced, the complexity and learning capability of the model are gradually improved, and beneficial clinical index characteristics are extracted.

Step 3: inputting the acquired cardiac MRI image into the image feature branch, extracting classification features of the dilated cardiomyopathy MRI image by means of an encoder-decoder structure, comprising:

step 31: preprocessing cardiac MRI image data, and processing the space distance between an original image and a marked image into 1.0 multiplied by 1.0mm by adopting an image preprocessing method in order to ensure the consistency of the space distance of the data ² Normalizing the pixels to be distributed at [ -1.0,1.0]Within the floating point range. And cropping the ROI for each original image and the corresponding annotation image.

Because of the fewer severe prognostic events that occur in a patient in practice, the negative sample of the dataset is a majority and the positive-negative sample ratios of the LGE and T1 mapping data are about 1:20 and 1:12, respectively. In order to avoid the phenomenon of overfitting caused by class unbalance, the invention adopts affine transformation methods such as rotation, translation and the like used in the prior art to carry out data enhancement on the positive class with less sample size. LGE and T1 mapping are two commonly used magnetic resonance image modalities, both of which are associated with dilated cardiomyopathy prognosis, thus using two data sets to verify the effectiveness of the present method

Step 32: cascading 2 layers of common convolution with the size of 3 multiplied by 3 in the encoder part to extract shallow layer characteristics; each normal convolution contains one BN layer and one ReLU layer.

Step 33: with the residual learning strategy, the encoder further comprises a step of constructing a bottom-up fast feed-forward scan by cascading 6 depth separation residual modules (Separable Residual Module, SRM) in sequence to extract features and rapidly increase receptive fields, and fig. 2 is a schematic structural diagram of the depth separation residual modules SRM. The depth separation residual error module SRM adopts residual error connection, a separation convolution layer with the sequential cascade void ratio of 3, 5 and 7 and the convolution kernel size of 3 multiplied by 3 is utilized in each depth separation residual error module to carry out multi-scale feature extraction, and shallow layer features extracted in the step 32 are input into the depth separation residual error module to obtain cross-task features;

the separable convolution layer completely separates the learning tasks of the spatial correlation and the inter-channel correlation, reduces the parameter number of the model, and replaces identical mapping with 1 multiplied by 1 common convolution for residual connection so as to enhance the expression capacity of the network and obtain the cross-task characteristics.

Step 34: inputting the cross-task features into a decoder for decoding, introducing an attention mechanism at the decoder part, adopting a residual attention module RAM (Residual Attention Module, RAM) for global information expansion from top to bottom, pertinently extracting features with important guiding function on classification tasks, enhancing the function of effective features and inhibiting redundant information; meanwhile, the training difficulty of the network is reduced by utilizing a residual error network structure, and the gradient disappearance phenomenon is effectively avoided.

Residual attention moduleA schematic diagram of the RAM structure is shown in fig. 3. The residual attention module (residual attention module) RAM comprises a trunk branch and a mask branch, wherein the trunk branch is formed by a plurality of cascaded residual convolution modules, performs feature processing on an input feature map and outputs a trunk feature map T _i,c (x) Where i and c represent spatial and channel positions, respectively.

The mask branches adopt a coding-decoding structure, and a mode of combining attention from bottom to top and from top to bottom is adopted to learn and obtain an attention characteristic mask M with the same size as the main output _i,c (x)。

The mask branching obtains the attention feature mask by: firstly, performing quick feed-forward scanning from bottom to top on an encoder part, and performing feature extraction and quick increase of receptive fields by using three residual units and two largest pooling layers; secondly, global information expansion from top to bottom is carried out on the decoder part, input features of each position are guided, symmetrical up-sampling is carried out by using two layers of linear interpolation with the same number as the maximum pooling layer number in the encoder, so that the same size of the input and output features is ensured, and finally, jump connection among decoder structures of the encoder is further added, so that information with different scales is captured.

Step 35: the last layer of the decoder uses a global average pooling layer with a pooling window of 3 multiplied by 3 to reduce the feature dimension, and obtain the high-dimensional image featureH _i,c (x) So as to be fused with clinical index features, and simultaneously, spatial information is reserved;

step 4: the method comprises the steps of adopting a multi-source information fusion strategy to fuse and predict clinical index features and high-dimensional image features, adopting the fusion strategy as feature layer fusion, splicing the high-dimensional features of two mode data in an image branch and a clinical index branch, sending the spliced features to a two-layer cascade full-connection layer to solve the problem of network nonlinearity, adopting Focal loss to relieve the problem of class imbalance, outputting a final classification result, and predicting whether a serious prognosis event occurs to a patient.

The multi-source information fusion strategy comprises three types of data layer fusion, feature layer fusion and decision layer fusion, as shown in fig. 4. The data layer fusion is low-level fusion, namely, original data from different sources are directly fused, then the fused data are subjected to subsequent feature processing and decision making, the method can save the original information in the source data to the greatest extent, more details in the original data are mined, but the calculation burden is large, and the real-time processing cannot be performed.

The feature layer fusion is to extract feature information from different data sources respectively, then to analyze and process the extracted feature information after fusion, and to provide support for later decision analysis.

The decision layer fusion is high-level fusion, firstly, analysis processing is carried out on different data sources to obtain respective preliminary decisions, then, the preliminary decisions are fused by adopting a certain rule to obtain a final decision result, and the method is flexible and has the best real-time performance, but can lose a large amount of detail information, and has larger dependence on the previous stage and difficult realization.

The invention adopts four objective evaluation indexes of widely used Accuracy (Accumavailable), sensitivity (called Recall), specificity (Specificity) and Area Under Curve (AUC) in image classification experiments. Accuracy is the most common evaluation index in classification tasks, and refers to the ratio of the number of correctly predicted samples to the number of all samples, in general, the higher the Accuracy is, the better the classifier effect is. The Sensitivity represents the proportion of the samples of all positive classes which are divided, and the recognition capability of the classifier on the samples of the positive classes is measured, and the higher the recognition capability of the model on the samples of the positive classes is, the stronger the recognition capability of the model on the samples of the positive classes is. Specificity represents the proportion of all negative samples which are divided into pairs, and the recognition capability of the classifier on the negative samples is measured, and the higher the recognition capability of the model on the negative samples is, the stronger the recognition capability of the model on the negative samples is. AUC refers to the area size under the ROC (Receiver Operating Characteristic) curve. The ROC curves are on the abscissa and ordinate of FPR (false negative rate) and TPR (true rate), respectively, the smaller the FPR, the higher the TPR, the better the model. Thus the larger the area under ROC (AUC), or the closer the curve is to the upper left corner, the more desirable the model classification effect.

In order to verify the effectiveness of the MM-Net provided by the invention for the prognosis auxiliary prediction of dilated cardiomyopathy, the invention compares the invention with five currently mainstream image classification methods of VGG16, resNet50, denseNet121, acceptance v3 and Xacceptance.

Table 1 shows the results of quantitative assessment of classification of several methods on both T1 mapping and LGE data, both from data collected from the cardiac department of the collaborative hospital.

As can be seen from Table 1, the accuracy of MM-Net was 99.4% higher than VGG16, resNet50, denseNet121, acceptance v3 and Xacceptance by 13.7%, 1.5%, 0.7% and 0.5%, respectively, on LGE data, and the AUC index of MM-Net was also highest, which fully demonstrates that the prognosis of MM-Net on LGE data was best. Meanwhile, the sensitivity and the specificity of the MM-Net are higher than those of other methods, and 100% and 99.7% are achieved respectively, which shows that the MM-Net can well treat the problem of class imbalance and avoid being trapped in overfitting.

On the T1 mapping data, the accuracy, sensitivity and AUC of MM-Net are all best, namely 97.1%, 91.2% and 98.4%, which shows that the best classification effect can be obtained on the T1 data. While ResNet50 and Xreception are as high as 99.5% and 97.8% specific respectively, their sensitivity is very low, indicating that these methods cannot better deal with the problem of data class imbalance, which falls into overfitting. The good classification performance of MM-Net on T1 mapping data further suggests that it has good classification capability on different cardiac MRI datasets.

Table 1 comparison of objective evaluation index for different network models

In addition, the predictive result of comprehensively comparing the LGE data and the T1 data can show that the predictive effect of prognosis based on the LGE data is more ideal than that of T1 mapping, so that the LGE is more suitable for the predictive prediction of dilated cardiomyopathy than T1 mapping, and the LGE is an independent predictive factor of poor prognosis of dilated cardiomyopathy, which proves that the LGE has an important relation with the long-term prognostic evaluation of dilated cardiomyopathy patients in clinic.

In order to further verify the beneficial effects of the innovation points, the invention carries out an ablation experiment on LGE data aiming at the MM-Net model so as to find the optimal setting of the structure and parameters of the model, and particularly relates to the following problems: (1) impact of clinical indicators on model performance; (2) impact of different fusion strategies on model performance; (3) whether the data enhancement step is effective; (4) selection of a loss function.

1. Clinical index

In order to verify the effectiveness of the Relief feature selection algorithm and the influence of clinical indexes on a prognosis prediction network model, the weight of each clinical index is calculated by adopting the Relief feature selection algorithm in an experiment, and the correlation degree of each index and the prognosis of the dilated cardiomyopathy is obtained through sequencing. Then, based on the weight of the Relief, the weight of the Relief is sequentially added into a network model from large to small for experiments, and the effectiveness of the Relief algorithm is verified. Table 2 shows the Chinese and English names and their shorthand representations of the clinical indicators used in the present invention.

Table 2 clinical indicators Chinese and English designations and abbreviations thereof show

Fig. 5 shows the weight values of the Relief (i.e., the correlation with dilated cardiomyopathy prognosis prediction) and the correlation ranking for each clinical index calculated based on the Relief algorithm. From fig. 5, the following points can be seen:

(1) According to the sequence of the Relief values from large to small, the importance degree of each clinical index on the prognosis of the dilated cardiomyopathy is as follows: hyp, DCM type, age, HB, SBP, sex, smoker, DBP, weight, his, height, NYHA, ACEIARB, new HF, diu, alcohol, beta blocker, antis.

(2) The index of hyp (hypertension) has the highest weight value of Relief which reaches more than 0.1, which indicates hyp is the index most relevant to the prognosis of dilated cardiomyopathy in all indexes of the experiment, and indicates that hypertension is an important influence factor of adverse events occurring in the prognosis of dilated cardiomyopathy.

(3) The Relief values of the DCM type index and the DCM type index age, HB, SBP index are all lower than hyp but are all higher than 0.01, which indicates that the DCM type index and the DCM type index are related to the dilated cardiomyopathy prognosis, and indicate that the characteristics of DCM type, age, hemoglobin and systolic pressure and the dilated cardiomyopathy occurrence adverse prognosis event have a prompt effect.

(4) The values of the Relief of the sex, smoker, DBP, weight, his, height index and the NYHA index are between 0.01 and 0.1, which shows that the indexes are also related to the prognosis of dilated cardiomyopathy, and the characteristics of gender, smoking, diastolic pressure, weight, disease duration, height and heart function grading have prognostic judgment value on adverse events of dilated cardiomyopathy.

(5) The negative Relief values for the six indices of antis, β blocker, alcohol, diu, new HF and ACEIARB indicate that in this experiment these indices are not correlated with dilated cardiomyopathy prognosis and even interfere with prognosis prediction accuracy.

Table 3 shows the effect of different clinical indexes on the performance of the prognosis prediction model according to the experimental results of the clinical indexes added sequentially from large to small according to the correlation. From table 3 the following conclusions can be drawn:

(1) Along with the increase of clinical indexes, the prediction capacity of the model is gradually improved until the first 12 indexes with the maximum correlation are added, the prediction capacity of the model reaches the highest, and three indexes of accuracy, specificity and AUC reach 99.4%, 98.7% and 99.8%, so that clinical indexes screened based on a Relief algorithm are fully explained to be capable of assisting the model in improving the prediction capacity.

(2) When added to the first four indices according to correlation, the model predictive power is instead slightly reduced, which may be due to just the interaction and cancellation between the several indices, reducing the model predictive power.

(3) With the sequential addition of the first 12 clinical indexes, the improvement of the model performance begins to be slow, the correspondence between the ordering of the related indexes and the importance degree of prognosis prediction is proved, and the Relief value can reflect the related degree of the indexes.

(4) When the index of negative number such as ACEIARB and the like is continuously added, the model performance starts to decline, which indicates that clinical indexes have no promotion effect on the model performance when the index is smaller than 0, but can cause negative influence, and the effectiveness of the Relief feature selection algorithm is further verified.

Table 3 experimental results of the addition of MM-Net to different clinical indicators

2. Fusion strategy

The network MM-Net provided by the invention adopts a feature layer fusion strategy, so that good performance is obtained. In order to verify the influence of different fusion strategies on a prognosis prediction model of multi-source information fusion, the invention expands experiments aiming at a data layer fusion strategy and a decision layer fusion strategy respectively based on a basic structure and a functional module of MM-Net.

The data layer fusion strategy adopts a data fusion mode, and because a separate branch is not needed to process clinical indexes, clinical branches in the MM-Net are removed, and image branches are reserved for feature extraction. Specifically, the preprocessed cardiac MRI image and the corresponding clinical index are spliced, then sent into the cardiac MRI image data branch of the MM-Net for feature extraction, and finally the classification prediction result is output.

The decision layer fusion strategy adopts a post-fusion mode, clinical index branches and image branches based on MM-Net respectively carry out classification feature extraction on clinical indexes and cardiac MRI images, then softmax is added after the last full-connection layer of the two branches respectively to enable the branches to respectively output classification results (two-dimensional feature vectors) of the branches, and finally, the classification results of the two branches are spliced and finally, a final prognosis prediction result is output through one full-connection layer.

Table 4 shows the prediction results of the model under different fusion strategies, and it can be seen from the table that the data layer fusion cannot fully utilize the complementary information among the data of multiple modes, and a large amount of redundant information can be caused. Decision layer fusion can lose a great deal of detail information and has larger dependence on the previous layer, so that the prediction effect is worst. The feature layer fusion can reserve enough important information in two different data, can mutually supplement and improve the classification result, and can obtain the most accurate prediction result.

TABLE 4 experimental results of different fusion strategies

3. Data enhancement

In the experiment of the invention, in order to relieve the problem of class unbalance during data preprocessing, data enhancement is carried out on positive class samples with smaller sample sizes. To verify its effectiveness, the present invention performed experiments on data sets that were not data enhanced, the results of which are shown in table 5.

Table 5 data enhanced experimental results

As can be seen from Table 5, the model prediction results without data enhancement were only slightly more specific than the results with data enhancement, because the proportion of negative samples with large sample size was relatively reduced after data enhancement. But the accuracy, sensitivity and AUC are lower than the model prediction result with data enhancement, and the sensitivity is most obvious, which is reduced by 12.9%, thus fully illustrating that the data enhancement can improve the classification accuracy of the positive sample with small sample size and alleviate the class imbalance.

4. Loss function

The loss function used by the MM-Net is Focal loss, and in order to verify whether the loss function is optimal, the invention adopts three common loss functions of CE loss, dice loss and Jaccard loss for comparison experiments respectively, and the results are shown in Table 6.

TABLE 6 Effect of loss function

As can be seen from Table 6, the evaluation indexes of the Focal loss are higher than the CE loss, the Dice loss and the Jaccard loss, and the sensitivity and the specificity reach 100% and 99.7% respectively, which fully illustrates that the Focal loss is helpful for improving the accuracy of the difficult-to-separate samples, and can better process unbalanced-like data. The indexes of CE loss are slightly lower than or equal to those of Focal loss, and the indexes of Dice loss and Jaccard loss are obviously reduced, which means that the capability of the loss functions in handling the class imbalance problem is still to be improved.

It should be noted that the above-described embodiments are exemplary, and that a person skilled in the art, in light of the present disclosure, may devise various solutions that fall within the scope of the present disclosure and fall within the scope of the present disclosure. It should be understood by those skilled in the art that the present description and drawings are illustrative and not limiting to the claims. The scope of the invention is defined by the claims and their equivalents.

Claims (1)

1. The cardiomyopathy prognosis auxiliary prediction method of fusion clinical information and magnetic resonance image is characterized in that the method firstly adopts a Relief feature selection algorithm to screen clinical indexes, then carries out feature fusion on the screened clinical indexes and cardiac MRI images, and constructs a prediction neural network MM-Net, comprising two independent feature extraction branches: the clinical feature branch and the image feature branch are respectively subjected to feature extraction of clinical indexes and cardiac MRI images, and finally high-dimensional feature information respectively extracted by the two branches is subjected to fusion processing, and a final cardiac MRI image classification result is output to assist in predicting whether a severe prognosis event occurs in an dilated cardiomyopathy patient, and the method specifically comprises the following steps:

step 1: clinical index screening based on a Relief feature selection algorithm, comprising:

step 11: the collected clinical indexes form a clinical index data set, the clinical indexes in the clinical index data set are subjected to data cleaning and preprocessing, data with larger index deletion degree are deleted, data with smaller deletion degree are supplemented, and then non-numerical data items are converted;

step 12: sequentially calculating the weights of 18 clinical indexes in the clinical index data set by adopting a Relief characteristic selection algorithm, and obtaining the correlation degree of each index and the dilated cardiomyopathy prognosis through sequencing;

step 13: eliminating clinical indexes with negative weights, and reserving clinical indexes with positive weights;

step 2: the clinical characteristic branch extracts high-dimensional characteristics of clinical indexes, the clinical indexes with positive weights reserved in the step 13 are sent into the clinical characteristic branch, the clinical characteristic branch comprises three fully-connected layers with the number of 16, 32 and 64 neurons which are cascaded in sequence, and the high-dimensional characteristics of the clinical indexes are obtained after the clinical characteristic branch is passed;

step 3: inputting the acquired cardiac MRI image set into the image feature branch, extracting high-dimensional classification features of the dilated cardiomyopathy MRI image through an encoder-decoder structure of the image feature branch, specifically comprising:

step 31: first, preprocessing cardiac MRI image data, and processing the space between the original image and the marked image to 1.0X1.0 mm ² Normalizing the pixels to be distributed at [ -1.0,1.0]And enhancing the positive sample data by a rotation and translation operation;

step 32: shallow layer feature extraction is carried out through 2 layers of common convolution with the size of 3 multiplied by 3, which is cascaded at the front end of the encoder;

step 33: the encoder further comprises 6 depth separation residual error modules SRM which are sequentially cascaded, a fast feed-forward scanning from bottom to top is constructed to perform feature extraction and fast increase receptive fields, a residual error learning strategy is adopted, the depth separation residual error modules SRM are connected by residual errors, multi-scale feature extraction is performed in each depth separation residual error module by utilizing a separation convolution layer with the sequential cascade void ratio of 3, 5 and 7 and the convolution kernel size of 3 multiplied by 3, and shallow layer features extracted in the step 32 are input into the depth separation residual error modules to obtain cross-task features;

step 34: inputting the cross-task features into a decoder for decoding, introducing an attention mechanism at the decoder part, adopting a residual attention module RAM for global information expansion from top to bottom, pertinently extracting features with important guiding effect on classification tasks, enhancing the effect of effective features and inhibiting redundant information;

step 35: the last layer of the decoder reduces the feature dimension by using a global average pooling layer with a pooling window of 3 multiplied by 3 size to obtain the high-dimensional feature of the MRI image of the heart;

step 4: and fusing and predicting the high-dimensional features of the clinical indexes and the high-dimensional features of the cardiac MRI images by adopting a multi-source information fusion strategy, wherein the fusion strategy is feature layer fusion, splicing the high-dimensional features of the two mode data in the image feature branches and the clinical feature branches, sending the spliced features into two layers of cascaded full-connection layers, and simultaneously, adopting Focal loss to relieve the problem of unbalanced class and output a final classification result.

Images