CN115281688A - Cardiac hypertrophy multi-label detection system based on multi-mode deep learning - Google Patents

Abstract

The invention discloses an atrial and ventricular hypertrophy multi-label detection system based on multi-mode deep learning, and belongs to the field of cardiac hypertrophy detection on electrocardiograms. By adopting the methods of self-adaptive wavelet threshold denoising, heart beat segmentation based on morphological characteristics, a deep neural network, multi-modal characteristic fusion and the like, the method is used for simultaneously classifying four kinds of cardiac hypertrophy and normality by fusing the time domain characteristics and the morphological characteristics of the electrocardiosignals and individual information of a patient and building a multi-label detection frame based on multi-modal deep learning when the popular computer vision technology is transferred to the classification of the electrocardiosignals, thereby solving the problems of poor accuracy and poor generalization capability of the electrocardiosignals on the cardiac hypertrophy detection, realizing the accurate cardiac hypertrophy monitoring based on the electrocardiogram and being used for clinical diagnosis.

Description

Cardiac hypertrophy multi-label detection system based on multi-mode deep learning

Technical Field

The invention belongs to the field of electrocardiosignal detection, and particularly relates to a cardiac hypertrophy multi-label detection system based on multi-mode deep learning.

Background

Cardiac hypertrophy (cardioc hypertphy) is very common in the clinical diagnostic treatment of cardiology. Cardiac hypertrophy is further classified into atrial hypertrophy (atrial hypertrophy) and ventricular hypertrophy (ventricular hypertrophy), and is mainly associated with atrial and ventricular overload. The heart is overloaded due to the atrial hypertrophy, the problems of hypertension, coronary heart disease, cardiomyopathy and the like of heart blood vessels can be caused, and myocarditis, angina, heart failure and other diseases are caused due to the difficulty in the ventricular hypertrophy. The heart diseases have the characteristics of high disability rate, high death rate and the like, and seriously affect the life health of human beings.

The clinical diagnosis of cardiac hypertrophy is mainly performed by Electrocardiographic (ECG) examination and echocardiogram (UCG) examination. The heart color ultrasound detection has higher sensitivity to the change of the inner diameter of each chamber of the heart, lower misdiagnosis rate and missed diagnosis rate and higher detection rate to the cardiac hypertrophy. However, the heart color ultrasound detection is extremely susceptible to factors such as examination time and equipment, has high limitation and relatively high cost, and is difficult to popularize in primary hospitals. An electrocardiogram is a cardiac examination that detects the electrical activity produced by the heart and thus detects abnormalities in the heart. The clinical application of electrocardiogram should always be advocated by the great majority of clinical obligations with its advantages of economy, practicality, convenience and rapidness. In particular, electrocardiograms are widely used in disease diagnosis in remote areas and primary hospitals. Compared with heart color Doppler diagnosis, the electrocardiogram has the advantages of simpler operation, wider clinical application range and higher repeatability, so the electrocardiogram has stronger assistance in diagnosis of atrial and ventricular hypertrophy and is more commonly used. Moreover, some heart diseases are mainly electrically changed in the early stage, and morphological changes occur in the middle and late stages, so that the electrical examination by means of an electrocardiogram helps patients to find early treatment. However, the accuracy of diagnosing atrial hypertrophy by using an electrocardiogram is low, and therefore, a method for rapidly and effectively detecting atrial hypertrophy by using an electrocardiogram is urgently needed.

In the traditional electrocardiogram analysis of cardiac hypertrophy, a doctor diagnoses the electrocardiogram waveform by combining own knowledge and clinical experience, so that the accuracy of the method depends heavily on the experience and the diagnosis level of the doctor, and meanwhile, the time is consumed under the condition of large data volume, so that the traditional method has great limitation and cannot completely meet the clinical requirement. In recent years, deep learning has made automatic diagnosis of an electrocardiogram a research focus in the field of medical research. But currently, the studies for diagnosing cardiac hypertrophy by electrocardiogram with depth are very few, and most studies focus on diagnosis of a single category only, which makes the diagnosis of electrocardiogram still have great limitations; meanwhile, only the electrocardio characteristics are concerned when the deep learning method is used for processing the electrocardiosignals, and the influence of individual difference on medical detection is ignored, so that the generalization capability of the detection method in common people is poor.

Disclosure of Invention

In view of the above defects or improvement needs of the prior art, the present invention provides a cardiac hypertrophy multi-label detection system based on multi-modal deep learning, thereby solving the problem of low accuracy of cardiac hypertrophy electrocardiogram classification according to single-modal feature representation in the prior art.

To achieve the above object, according to one aspect of the present invention, there is provided a cardiac hypertrophy multi-label detection system based on multi-modal deep learning, comprising:

the medical record information extraction module is used for extracting preset attributes from the electronic medical record to be diagnosed, splicing the preset attributes and converting the spliced attributes into attribute vectors;

the electrocardiosignal denoising module is used for removing baseline noise, power frequency interference and muscle interference in the electrocardiosignals corresponding to the electronic medical record to obtain denoised electrocardiosignals;

a heart beat image extraction module for respectively carrying out QRS wave positioning on the II-lead, V1-lead and V6-lead electrocardiosignals in the denoised electrocardiosignals and selecting any adjacent 3 QRS wave peak points Q _i-1 ，Q _i ，Q _i+1 Intercept delete Q _i-1 Last K electrocardio sampling points and Q _i+1 Front K electrocardio sampling points and rear Q electrocardio sampling points _i-1 And Q _i+1 Taking the electrocardio sampling points as electrocardio beat images; carrying out channel splicing on the electrocardio beat images of the II-lead electrocardiosignal, the V1-lead electrocardiosignal and the V6-lead electrocardiosignal to obtain a heart beat image;

the multi-mode diagnosis module is used for respectively extracting the attribute vector, the denoised electrocardiosignal and the semantic feature, the time domain feature and the morphological feature of the heart-beat image, performing feature fusion and then performing multi-label classification to obtain a detection result;

wherein the detection result is at least one of Left Atrial Hypertrophy (LAH), left Ventricular Hypertrophy (LVH), right Atrial Hypertrophy (RAH) and Right Ventricular Hypertrophy (RVH), or Normal (Normal).

In general, compared with the prior art, the above technical solution contemplated by the present invention can achieve the following beneficial effects:

1. the cardiac hypertrophy multi-label detection system based on multi-mode deep learning provided by the invention adopts methods such as cardiac beat segmentation, a deep neural network and multi-mode feature fusion based on morphological features, migrates the current popular computer vision technology to classification of electrocardiosignals, simultaneously judges four cardiac hypertrophy and normal conditions by fusing time domain features, morphological features and individual patient information of the electrocardiosignals, and builds a multi-label detection frame based on multi-mode deep learning, thereby solving the problems of poor accuracy and poor generalization capability of the electrocardiosignals on cardiac hypertrophy detection, realizing accurate cardiac hypertrophy monitoring based on electrocardiogram, and being applicable to clinical diagnosis.

2. According to the cardiac hypertrophy multi-label detection system based on multi-mode deep learning, 3 leads closely related to cardiac hypertrophy diagnosis in a medical theory are selected as cardiac beat input, redundant information in electrocardiosignals is reduced, feature focusing is facilitated, feature extraction is more concerned with morphological features of cardiac hypertrophy through cardiac beat segmentation, and classification accuracy is improved.

3. According to the multi-label detection system for cardiac hypertrophy based on multi-modal deep learning, the age and the gender are extracted from the electronic medical record of a patient and serve as medical record metadata, and the age and the gender have obvious individual difference, so that the generalization capability of the model can be effectively improved by incorporating the age and the gender into model training input. Meanwhile, age and sex are also attributes that are closely related to the diagnosis of cardiac hypertrophy in medical theory. Therefore, the age and the gender as the medical record metadata can effectively improve the generalization capability of the model and the accuracy of diagnosis.

4. The cardiac hypertrophy multi-label detection system based on multi-mode deep learning provided by the invention can remove noise by a wavelet self-adaptive threshold denoising method, simultaneously ensure that the distortion of a reconstructed signal is as small as possible, and can ensure the smoothness of the signal and improve the denoising effect by selecting a mode of combining soft and hard thresholds in a threshold function, thereby realizing the denoising fidelity of an electrocardiosignal.

5. According to the heart hypertrophy multi-label detection system based on multi-mode deep learning, the time sequence feature extraction module extracts frequency domain variation through the one-dimensional convolutional neural network CNN, the system has strong feature extraction capability and robustness, meanwhile, the long-term and short-term memory network LSTM is used for extracting historical information of a long-term sequence, and the time implicit features are extracted. The heart beat image feature extraction module effectively extracts local waveform features on the heart beat image by using the two-dimensional convolutional neural network CNN, and simultaneously retains original input information by using a residual error structure, so that local micro features of the electrocardiosignals are extracted better. The medical record data feature extraction module extracts the age and the gender of data through the full connection layer DENSE layer to serve as semantic features, and the individual difference is considered, so that the generalization capability of the detection model is improved. The manifestation of cardiac hypertrophy on an electrical cardiac signal is both temporally abnormal and morphologically abnormal, and there is both context dependence and a small waveform change, so that cardiac hypertrophy can be detected better by combining temporal features with morphological features. The multi-modal diagnosis module can effectively detect the two characteristics, and simultaneously considers the individual difference, thereby improving the accuracy and generalization capability of the detection model and realizing the effective detection of the neural network on the cardiac hypertrophy characteristics.

Drawings

Fig. 1 is a structural diagram of a cardiac hypertrophy multi-label detection system based on multi-modal deep learning according to an embodiment of the present invention;

FIG. 2 is a schematic diagram showing the components of an electrocardiogram of a healthy adult;

fig. 3 (a) and fig. 3 (b) are a structural diagram of a convolutional neural network and a structural diagram of a cyclic neural network, respectively, according to an embodiment of the present invention;

FIG. 4 is a diagram of a framework of a multi-modal deep neural network provided by an embodiment of the present invention;

FIG. 5 is a schematic diagram of denoising processing of an electrocardiographic signal according to an embodiment of the present invention;

FIG. 6 is a diagram illustrating a morphological heartbeat segmentation framework according to an embodiment of the present invention;

fig. 7 (a), 7 (b), 7 (c), 7 (d), and 7 (e) are the experimental results obtained by the multi-label detection system for cardiac hypertrophy based on multi-modal deep learning provided by the present invention on the test set according to the embodiment of the present invention and AUC scores of Left Atrial Hypertrophy (LAH), left Ventricular Hypertrophy (LVH), right Atrial Hypertrophy (RAH), right Ventricular Hypertrophy (RVH), and Normal (Normal), respectively.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

The embodiment of the invention provides a cardiac hypertrophy multi-label detection system based on multi-modal deep learning, as shown in fig. 1, comprising:

and the medical record information extraction module is used for extracting preset attributes from the electronic medical record to be diagnosed, splicing the preset attributes and converting the spliced attributes into attribute vectors.

Specifically, the medical record information extraction module is used for extracting preset attributes from the electronic medical record to serve as medical record structured data, splicing the extracted attribute information together to obtain medical record metadata, and converting the medical record metadata into vectors and storing the vectors.

Further, the preset attributes include age and gender.

Specifically, the medical record information extraction module extracts two types of age and gender data which have obvious individual differences and are closely related to cardiac hypertrophy diagnosis from the electronic medical record as medical record metadata.

And the electrocardiosignal denoising module is used for removing baseline noise, power frequency interference and muscle interference in the electrocardiosignals corresponding to the electronic medical record to obtain denoised electrocardiosignals.

Specifically, the electrocardiosignal denoising module is used for acquiring electrocardiosignals (12-lead electrocardiosignals) corresponding to the electronic medical record, baseline noise, power frequency interference and muscle interference are eliminated from original electrocardiosignals, so that the waveforms of the electrocardiosignals are not influenced by noise, and interference-free electrocardio sequence data are obtained,

a heart beat image extraction module for selecting any adjacent 3 QRS peak points Q after QRS wave positioning is carried out on the II-lead, V1-lead and V6-lead electrocardiosignals in the de-noised electrocardiosignals _i-1 ，Q _i ，Q _i+1 Intercept delete Q _i-1 Last K electrocardio sampling points and Q _i+1 After the first K electrocardio sampling points, Q _i-1 And Q _i+1 Taking the sampling points of the electrocardio as the electrocardio beat images, thereby respectively obtaining the hearts of the II lead, the V1 lead and the V6 leadAn electrocardiographic beat image of the electrical signal; carrying out channel splicing on the electrocardio beat images of the II-lead electrocardio signals, the V1-lead electrocardio signals and the V6-lead electrocardio signals to obtain a heart beat image;

specifically, the heart beat image extraction module is used for carrying out QRS wave positioning on the denoised electrocardio time sequence signal, and storing a heart beat sequence with a QRS wave band as the time length of a patient to obtain a heart beat image as heart beat image data. That is, II-lead, V1-lead and V6-lead are selected from the denoised 12-lead electrocardiogram for cardiac image extraction. And (3) realizing QRS wave identification and positioning by utilizing a Pan-Tompkins algorithm, and slicing each electrocardio beat according to the peak time of a Q wave. And processing the electrocardiosignals of each lead to obtain an electrocardio beat. Connecting the electrocardio beats of 3 leads (II lead, V1 lead and V6 lead) into a complete heart beat image.

The multi-mode diagnosis module is used for respectively extracting the attribute vector, the denoised electrocardiosignal and the semantic feature, the time domain feature and the morphological feature of the heart-beat image, performing feature fusion and then performing multi-label classification to obtain a detection result;

wherein the detection result is at least one of left atrial hypertrophy, left ventricular hypertrophy, right atrial hypertrophy and right ventricular hypertrophy, or normal.

Furthermore, the multi-modal diagnosis module comprises a medical record data feature extraction module, an electrocardiosignal feature extraction module, a heartbeat image feature extraction module, a multi-modal feature fusion module and a multi-label classifier;

the medical record data feature extraction module is used for extracting semantic features of the attribute vectors;

the electrocardiosignal feature extraction module is used for extracting the time domain features of the denoised electrocardiosignals;

the heart beat image feature extraction module is used for extracting morphological features of the heart beat image;

the multi-mode feature fusion module is used for fusing the semantic features, the time domain features and the morphological features to obtain a multi-mode feature fusion vector;

the multi-label classifier is used for calculating the probability that the multi-modal feature fusion vector belongs to each label category, and reserving and outputting the label category with the probability higher than a preset threshold value.

Further, the preset threshold is 50%.

Further, the medical record data feature extraction module comprises a full connection layer and a long-term and short-term memory network;

the semantic feature electrocardiosignal feature extraction module comprises a one-dimensional convolutional neural network, a residual error network, a long-term and short-term memory network and a global average pooling layer;

the heart beat image feature extraction module comprises a two-dimensional convolution neural network, a residual error network and a global average pooling layer;

the multi-modal feature fusion module comprises a Concatenate layer;

the multi-label classifier includes a sense layer and a Sigmoid layer.

The multi-mode diagnosis module is used for splicing the medical record metadata, the electrocardiogram sequence data and the heartbeat image data together to obtain multi-mode fusion data, and then inputting the multi-mode fusion data into a trained multi-mode deep neural network to obtain a multi-label diagnosis result.

Specifically, the medical record data feature extraction module comprises a full-connection Dense layer and a long-short term memory network LSTM, and is used for vectorizing input medical record metadata and then obtaining semantic features of the medical record data by utilizing the semantic information of network learning age and gender;

the electrocardio sequence feature extraction module comprises a one-dimensional convolutional neural network CNN, a residual error network and a long-short term memory network LSTM, and is used for performing feature extraction on the input preprocessed electrocardiosignals and performing time sequence feature extraction analysis on the electrocardiosignals to obtain time domain features of the electrocardiosignals;

the heart beat image feature extraction module comprises a two-dimensional convolution neural network CNN and a residual error network, and is used for performing morphological feature extraction on an input heart beat image, performing morphological feature analysis on an electrocardiosignal, and focusing on waveform change to obtain morphological features of the electrocardio image;

the multi-modal feature fusion module and the multi-label classifier jointly form a fusion classification module, which comprises a connection Concatenate layer, a full connection Dense layer and a Sigmoid layer, and is used for fusing the time domain features, the morphological features and the semantic features, and then performing Sigmoid transformation to obtain a multi-label classification result after feature morphological mapping.

The multi-label classifier outputs a string of probability sequences, which represent whether the cardiac hypertrophy problems such as left atrial hypertrophy, left ventricular hypertrophy, right atrial hypertrophy and right ventricular hypertrophy exist on the original signals or whether diseases do not exist and the signals are normal. Classes with a probability higher than 50% are considered to be present, thus leading to a multi-label diagnosis about cardiac hypertrophy.

Further, the electrocardiosignal denoising module removes baseline noise, power frequency noise and electromyographic interference in the electrocardiosignals based on a wavelet self-adaptive threshold denoising method.

Further, the electrocardiosignal denoising module determines the maximum frequency of the electrocardiosignals (12-lead electrocardiosignals in the electronic medical record to be diagnosed) based on Nyquist sampling theorem analysis and performs wavelet decomposition and reconstruction to remove baseline noise; and performing wavelet decomposition and reconstruction again, determining a corresponding threshold based on a fixed threshold estimation method, and performing power frequency interference and muscle interference denoising according to a threshold function combining a hard threshold and a soft threshold.

Further, for the electrocardiosignals with the sampling frequency of 500Hz, the maximum frequency of the electrocardiosignals is determined to be 250Hz based on Nyquist sampling theorem analysis, and wavelet decomposition and reconstruction with the decomposition scale of 7 are carried out to remove baseline noise; decomposing and reconstructing the wavelet with the decomposition scale of 5, determining a corresponding threshold lambda based on a fixed threshold estimation method, and performing a threshold function according to the combination of a hard threshold and a soft threshold

Denoising power frequency interference and muscle interference; wherein w is the wavelet coefficient after decomposition, w _λ And (4) wavelet coefficients after threshold denoising, wherein sgn is a step function.

Specifically, the electrocardiosignal denoising module performs denoising processing in the following manner:

the electrocardiosignal denoising module analyzes the sampling frequency of the electrocardiosignals corresponding to the electronic medical record based on the Nyquist sampling theorem to determine wavelet functions and decomposition scales, and thresholds and threshold functions of the decomposition scales, removes baseline drift noise by adopting a mode of setting approximate coefficients to zero, and removes power frequency interference and muscle interference based on soft and hard threshold function processing detail coefficients and wavelet reconstruction. Namely: the electrocardiosignal denoising module carries out Nyquist sampling theorem analysis on the sampling frequency of the electrocardiosignals, determines a wavelet function and a decomposition scale, then determines a proper threshold value by using a fixed threshold value estimation method, and selects a mode of combining a hard threshold value and a soft threshold value in the threshold value function. Wavelet coefficients larger than a threshold value on each scale of wavelet transformation are reserved, corresponding processing is carried out on the wavelet coefficients smaller than the threshold value, and signals without other noises such as baseline drift noise, power frequency interference, muscle interference and the like are obtained by signal reconstruction according to the processed wavelet coefficients.

The method provided by the present invention is further illustrated below by a specific example.

As shown in fig. 1, the invention provides a cardiac hypertrophy label detection method based on multi-modal deep learning, which comprises the following steps:

(1) First, the individual information of the patient in the electronic medical record is extracted.

Two types of data with obvious individual difference, namely age and gender, are extracted from the electronic medical record to be used as medical record metadata, and the medical record metadata are converted into vector data and then stored.

(2) And then simply denoising the electrocardiosignals corresponding to the medical record.

Before training with the neural network, the electrocardiosignal needs to be preprocessed to eliminate interferences such as baseline wander, muscle interference, power frequency interference and the like, as shown in fig. 5. As shown in figure 2, the sampling frequency of the electrocardio data used according to the invention is 500Hz, and the maximum frequency of the signals obtained by the Nyquist sampling theorem is 250Hz. In order to eliminate the low-frequency baseline noise with the frequency of 0.5Hz-2Hz, the number of decomposition layers is selected to be 7, and 7 orders of approximate components (cA 7) and detail components (cD 1, cD2, cD3, cD4, cD5, cD6 and cD 7) are arranged from top to bottom. To eliminate baseline wander, the cA7 approximation component is set to 0.

After baseline drift is removed, power frequency noise and electromyographic interference also need to be removed. In order to make the reconstructed signal distortion as small as possible, the number of decomposition layers cannot be chosen too large. For a db8 wavelet with a maximum decomposition level of 8, a decomposition level of 5 is chosen. The mode of combining the hard threshold and the soft threshold is selected from the threshold function, so that the signal distortion is ensured to be as small as possible while the signal smoothness is ensured.

Where λ is the selected threshold, w is the decomposed wavelet coefficient, w _λ And (4) wavelet coefficients after threshold denoising, wherein sgn is a step function.

The most common fixed threshold estimation method is utilized to determine the threshold in the threshold processing function, the fixed threshold can obtain a good noise reduction result in the soft threshold processing function, and the selected algorithm formula is as follows:

wherein, N is the sum of the number of wavelet coefficients obtained on each scale after the actual measurement signal is decomposed by wavelet transform, and σ is the standard deviation of the additional noise signal, which is the median of the absolute value of the first-level wavelet coefficient decomposed from the signal.

(3) And (4) carrying out heart beat segmentation on the denoised electrocardiosignals through QRS wave peak value positioning, and reserving a heart beat image with morphological characteristics.

The detection model selects 3 leads that are medically closely related theoretically to cardiac hypertrophy: II, V1 and V6 leads. Positioning the QRS wave peak value by utilizing a Pan-Tompkins algorithm, selecting a middle (except the first and the last) QRS wave peak value for the electrocardiosignals of the selected three leads, removing 40 electrocardio sampling points behind the previous Q wave peak value point, and removing 40 electrocardio sampling points in front of the next Q wave peak value point, as shown in figure 6, and finally taking the signal between the previous Q wave peak value point and the next Q wave peak value point as an electrocardio beat. Each single lead electrocardiosignal is processed as follows:

T(Q _peak (n-1)+40)≤T(Q _peak (n))≤T(Q _peak (n+1)-40)

and performing channel connection on 3 leads of the electrocardio beat images in the 12 leads of the electrocardio signals to form a heart beat image.

(4) The resulting data fusion process is used as model input.

And splicing and fusing the obtained medical record data, the electrocardiosignal data and the cardiac image data. Since the feature of cardiac hypertrophy on the cardiac electrical signal is mainly represented by morphological changes, such as small changes in waveform, it is difficult to identify cardiac hypertrophy by the cardiac electrical signal. In order to enable the model to pay attention to the morphological characteristics of the electrocardiosignals, the processed electronic medical record vector data, the electrocardiosignal data corresponding to the processed electronic medical record vector data and the corresponding cardiac image data are spliced together and input as the model.

(5) And constructing a multi-mode deep learning model as a classifier for training.

After the input data are processed, a multi-mode deep neural network is designed according to the cardiac hypertrophy electrocardio characteristics, and a convolutional neural network and a cyclic neural network are mainly used as a basis for building.

A Convolutional Neural Network (CNN) is a deep feedforward Neural network with local connections and weight sharing, and a network basic unit is composed of a Convolutional layer, a pooling layer and a full connection layer, as shown in fig. 3 (a). The convolutional neural network has fewer parameters than the feedforward neural network. For the image recognition task, the convolutional neural network achieves a better learning effect by keeping important parameters as much as possible and removing a large number of unimportant parameters. The convolutional neural network utilizes superposition of a plurality of convolutional layers to extract the characteristics of input data, performs characteristic selection and information filtering through a pooling layer, inputs the data into a full-connection layer to remove a multi-dimensional structure and expands the data into vectors. Common convolutional neural networks include one-dimensional convolutional neural networks and two-dimensional convolutional neural networks. One-dimensional convolution is often used for signal processing, and can well process time series analysis of data and obtain interesting features from fixed segments in the whole data set. Two-dimensional convolution is commonly used in image processing to extract local features corresponding to convolution kernels on an image. The method designs the neural network structure according to the characteristics of the electrocardiosignals and the cardiac hypertrophy, and is applied to time series characteristic extraction and cardiac image characteristic extraction.

A Recurrent Neural Networks (RNN) is a type of neural network that processes time series, and a network basic unit is composed of an input layer, a hidden state, and an output layer, as shown in fig. 3 (b). The hidden state H of the RNN can retain the output information Y of the current neuron and continue to act on itself at the next time and input X, capturing all current historical sequence information. The Long Short-Term Memory network (LSTM) is an RNN with a special structure, can solve the problems of Long-Term dependence, different intervals of useful information and the like, and has better performance when processing the embodiment.

The neural network framework for classifying electrocardio is shown in fig. 4, and the network consists of four modules: the system comprises a medical record feature extraction module, a time sequence feature extraction module, a heart beat image feature extraction module and a fusion classification module (comprising a multi-mode feature fusion module and a multi-label classifier).

For a medical record feature extraction module, vectorized age and gender information is input into a network of 4 fully-connected layers, each fully-connected layer comprises a nonlinear unit, output channels of all fully-connected layers are respectively 8, 16, 32 and 64, and extracted semantic features are obtained.

For a time sequence module, 12-lead electrocardiosignal data are input into a 2-layer one-dimensional convolutional neural network (1D CNN) framework, each convolutional layer comprises a normalization unit, a nonlinear unit and a normalization layer, the convolutional core of each convolutional layer is 15, and output channels are 32. After the convolutional neural network is operated, fine feature extraction is carried out by using 3 repeated residual error modules, after features are extracted by 3 convolutional layers of each residual error module, the inputs of a feature output layer and the residual error modules are spliced by utilizing residual error connection, wherein the convolutional layers respectively comprise a normalization unit, a nonlinear unit and a maximum pooling layer, the convolutional kernels of the 3 modules are 15, and output channels are respectively 64, 128 and 256. The residual module then uses a one-dimensional LSTM containing 128 hidden units to derive the extracted time series features.

For a heart beat image module, II leads, V1 leads and V6 leads are input as three channels of a heart beat image, firstly, a two-dimensional convolutional neural network (2D CNN) frame is utilized to carry out feature extraction, the convolution kernel of a convolution layer is 3, and output channels are all 32. And then, extracting deep features by using 3 repeated residual modules, splicing the input of a feature output layer and the input of the residual module by using residual connection after each residual module extracts features by using 3 convolutional layers, wherein the convolutional layers comprise a normalization unit, a nonlinear unit and a normalization layer, the sizes of convolutional kernels of the 3 modules are 3, and output channels are respectively 64, 128 and 256, so that the extracted heart beat image features are obtained.

And finally, fusing the information together to help classification. The extracted time sequence features are utilized by a one-dimensional global average pooling layer to obtain the most important one-dimensional time features, meanwhile, the extracted heart beat image features are input into a two-dimensional average pooling layer to obtain the most important two-dimensional image features, the connecting layer is spliced with the medical history semantic features and then input into two Dense layers respectively containing 64 and 5 neurons, and finally output is subjected to Sigmoid transformation, namely the multi-label classification result after feature form mapping is obtained.

The precision rate of the result of the heart hypertrophy obtained by heart color ultrasound diagnosis is very high, but the result obtained by electrocardiogram is poor, and the heart hypertrophy public electrocardiogram data set using the heart color ultrasound as a label is not provided at present, so that the invention constructs a heart hypertrophy data set using the heart color ultrasound diagnosis as a gold standard.

The standard 12-lead electrocardiogram data are recorded in the cardiac hypertrophy data set, the sampling time is 10s, the frequency is 500Hz, and the cardiac hypertrophy data set is identified by professional doctors and cardiac hypertrophy color ultrasonography as labels.

In addition, 500 pieces of data were individually selected as a test set.

The model evaluation mainly comprises four indexes: absolute accuracy (Subsetaccuracy), F ₁ Fraction, confusion matrix and area under the ROC curve. The multi-modal deep network multi-label classifier is a core framework of the invention, and whether various types of cardiac hypertrophy can be accurately detected depends on the performance of the classifier. Absolute accuracy scores if and only if the classifier classifies correctly for all categories. Comparison with absolute accuracy F ₁ The score is the primary performance measure of the accuracy of the current test, especially when multi-classified, F ₁ The score metric is more important.

The model performance can be reflected more directly by using the ROC curve and the confusion matrix. The confusion matrix is also called an error matrix, each column of the confusion matrix represents a prediction class, each row represents a real class, and the performance of the algorithm can be visually presented by the confusion matrix. The AUC value is the area under the ROC curve, the ROC curve uses the false positive rate as the abscissa, the true positive rate is the ordinate, its advantage lies in: the curve shape remains substantially unchanged when the positive and negative sample distributions are changed. Therefore, the score can reduce the interference caused by different test sets and more objectively measure the performance of the model. To obtain the ROC curve, only the True Positive Ratio (TPR) and the False Positive Ratio (FPR) are needed, which are expressed as follows:

after having obtained the ROC curve, the AUC meaning can be drawn: the larger the value, the better the classification effect of the model.

Therefore, the invention takes the AUC score as the main index for evaluating the performance of the model and simultaneously refers to F ₁ The scores and confusion matrix analyze the model results.

The invention makes various attempts on the network performance of each part of the multi-modal deep learning model and gives a final model.

First, the present invention tries a single-mode classification model, i.e. directly using 12-lead electrocardiographic signals for classification. Table 1 shows the multi-label classification experiments performed by using the network model designed by the electrocardiographic time series. The electrocardio time sequence is classified by utilizing a simple one-dimensional CNN network, the manifestation of cardiac hypertrophy on electrocardiosignals is mainly the tiny change of electrocardiosignal waveforms, the classification difficulty is higher, and the diagnosis difficulty of multi-label classification is further increased, so the effect is poorer, and the improvement is made on the basis. The accuracy rate of the RecnN obtained by combining the CNN and the residual module can be improved, and the performance can be further improved by connecting an LSTM network behind the RecnN. The accuracy of using BilSTM and LSTM is similar, and the calculation time is increased, so that a framework with RecnN-LSTM as a classifier is finally adopted for reducing the complexity. Results from other related studies are also presented as shown in table 1.

Multi-mode fusion is tried on the basis of an electrocardiogram time sequence classifier, the cardiac hypertrophy electrocardiogram has obvious morphological characteristic change, and meanwhile, the electrocardiogram is also influenced by age and gender and has obvious individual difference. Firstly, a heartbeat feature extraction module is added into a multi-modal deep network model, and morphological features are mainly processed. The two-dimensional residual convolutional neural network CNN model is used for processing the heart shot image, the morphological characteristics on the electrocardiosignals are extracted, the influence of redundant information is reduced, and the accuracy and the AUC are improved. Then, a medical record information feature extraction module is added into the multi-modal depth network model, semantic information of a patient closely related to cardiac hypertrophy in medical diagnosis is extracted through a plurality of layers of full connection layers, the accuracy is further improved, and the AUC is not changed greatly. Therefore, in the multi-mode deep network model, the electrocardio sequence feature extraction module, the heart beat morphological feature extraction module and the patient individuality information extraction module can be complementary in advantages, so that the overall network performance can be improved by using the multi-mode deep neural network. The control for the study structure is shown in table 2.

TABLE 1

TABLE 2

The invention makes more attempts on the aspect of network architecture, and because the network of the multi-mode model is more complex, the network of the heart beat feature extraction part is basically consistent with the network of the electrocardio sequence feature extraction part, and only the LSTM part is removed. In the two feature extraction parts, an inclusion network, a DenseNet network and the like which have better performance in the field of computer vision image classification in recent years are tested, but the embodiment is not the traditional image classification and therefore has poor performance, because the information of one-dimensional signals is too redundant and the information amount is far less than that of two-dimensional images, the improvement of the number of network layers and parameters cannot effectively improve the performance of the model. Thus, simple residual concatenation and self-building of the network can achieve the desired effect and gradually improve performance through parameter adjustment, as shown in table 3. After all models are trained, testing is carried out on the test set, and a section of probability sequence and a corresponding electrocardiogram are obtained after information extraction and feature classification network output. The absolute accuracy of the 500 electrocardiograms of the data set was 63% and the average ROC AUC score was 0.91.

TABLE 3

The invention establishes a cardiac hypertrophy electrocardiosignal multi-label detection model based on multi-mode deep learning, and can directly diagnose 4 types of cardiac hypertrophy or normality on the input 12-lead electrocardiosignals after training. Multiple experiments prove that the model can accurately identify and position the morphological heart beat characteristics of the cardiac hypertrophy and has specific diagnosis effect on different types of the cardiac hypertrophy; the core framework of the method is a multi-modal feature extraction module for identifying different categories of cardiac hypertrophy, so that the model is evaluated by the accuracy of heart beat identification and the finally measured multi-label classification error through the network. The feasibility of the test data set analysis model was selected and analyzed for each category, respectively, as shown in fig. 7 (a), 7 (b), 7 (c), 7 (d), and 7 (e), it was found that AUC for left atrial hypertrophy reached 89%, AUC for left ventricular hypertrophy reached 87%, AUC for right atrial hypertrophy reached 92%, AUC for right ventricular hypertrophy reached 89%, and normal AUC reached 98%. The recognition capability of the model is proved, and the model can be used for medical application and clinical diagnosis.

It will be understood by those skilled in the art that the foregoing is only an exemplary embodiment of the present invention, and is not intended to limit the invention to the particular forms disclosed, since various modifications, substitutions and improvements within the spirit and scope of the invention are possible and within the scope of the appended claims.

Claims (8)

1. A cardiac hypertrophy multi-label detection system based on multi-modal deep learning, comprising:

the medical record information extraction module is used for extracting preset attributes from the electronic medical record to be diagnosed, splicing the preset attributes and converting the spliced attributes into attribute vectors;

the electrocardiosignal denoising module is used for removing baseline noise, power frequency interference and muscle interference in the electrocardiosignals corresponding to the electronic medical record to obtain denoised electrocardiosignals;

a heart beat image extraction module for respectively carrying out QRS wave positioning on the II-lead, V1-lead and V6-lead electrocardiosignals in the denoised electrocardiosignals and selecting any adjacent 3 QRS wave peak points Q _i-1 ，Q _i ，Q _i+1 Intercept delete Q _i-1 Last K electrocardio sampling points and Q _i+1 Front K electrocardio sampling points and back Q _i-1 And Q _i+1 Taking the electrocardio sampling points as electrocardio beat images; carrying out channel splicing on the electrocardio beat images of the II-lead electrocardiosignal, the V1-lead electrocardiosignal and the V6-lead electrocardiosignal to obtain a heart beat image;

the multi-mode diagnosis module is used for respectively extracting the attribute vector, the denoised electrocardiosignal and the semantic feature, the time domain feature and the morphological feature of the heart-beat image, performing feature fusion and then performing multi-label classification to obtain a detection result;

wherein the detection result is at least one of left atrial hypertrophy, left ventricular hypertrophy, right atrial hypertrophy and right ventricular hypertrophy, or normality.

2. The system of claim 1, wherein the preset attributes include age and gender.

3. The system of claim 1, wherein the multi-modal diagnosis module comprises a medical record data feature extraction module, an electrocardiosignal feature extraction module, a cardiac image feature extraction module, a multi-modal feature fusion module and a multi-label classifier;

the medical record data feature extraction module is used for extracting semantic features of the attribute vectors;

the electrocardiosignal characteristic extraction module is used for extracting the time domain characteristics of the denoised electrocardiosignals;

the heart beat image feature extraction module is used for extracting morphological features of the heart beat image;

the multi-mode feature fusion module is used for fusing the semantic features, the time domain features and the morphological features to obtain a multi-mode feature fusion vector;

the multi-label classifier is used for calculating the probability that the multi-modal feature fusion vector belongs to each label category, and reserving and outputting the label category with the probability higher than a preset threshold value.

4. The system of claim 3, wherein the predetermined threshold is 50%.

5. The system of claim 3, wherein the medical record data feature extraction module comprises a full connectivity layer and a long-short term memory network;

the semantic feature electrocardiosignal feature extraction module comprises a one-dimensional convolutional neural network, a residual error network, a long-term and short-term memory network and a global average pooling layer;

the heart beat image feature extraction module comprises a two-dimensional convolution neural network, a residual error network and a global average pooling layer;

the multi-modal feature fusion module comprises a Concatenate layer;

the multi-label classifier includes a sense layer and a Sigmoid layer.

6. The system of claim 1, wherein the denoising module removes baseline noise, power frequency noise, and electromyographic interference in the electrocardiographic signal based on a wavelet adaptive threshold denoising method.

7. The system of claim 6, wherein the ecg signal denoising module determines a maximum frequency of the ecg signal based on nyquist sampling theorem analysis and performs wavelet decomposition and reconstruction to remove baseline noise; and performing wavelet decomposition and reconstruction again, determining a corresponding threshold based on a fixed threshold estimation method, and performing power frequency interference and muscle interference denoising according to a threshold function combining a hard threshold and a soft threshold.

8. The system of claim 7, wherein for an ecg signal having a sampling frequency of 500Hz, the maximum frequency of the ecg signal is determined to be 250Hz based on nyquist sampling theorem analysis, and wavelet decomposition and reconstruction are performed with a decomposition scale of 7 to remove baseline noise; decomposing and reconstructing the wavelet with the decomposition scale of 5, determining a corresponding threshold lambda based on a fixed threshold estimation method, and performing a threshold function according to the combination of a hard threshold and a soft threshold

Denoising power frequency interference and muscle interference; wherein w is the wavelet coefficient after decomposition, w _λ And (4) wavelet coefficients after threshold denoising, wherein sgn is a step function.

Images