CN114767163A

CN114767163A - Intelligent diagnosis system for congenital heart disease of children based on echocardiogram

Info

Publication number: CN114767163A
Application number: CN202210267483.6A
Authority: CN
Inventors: 李健; 颜波; 曹银银; 谭伟敏; 林青; 黄国英
Original assignee: Fudan University
Current assignee: Fudan University
Priority date: 2022-03-17
Filing date: 2022-03-17
Publication date: 2022-07-22
Anticipated expiration: 2042-03-17
Also published as: CN114767163B

Abstract

The invention discloses an echocardiogram-based intelligent diagnosis system for congenital heart diseases of children. The system comprises an ultrasonic cardiac video key frame selection module for automatically predicting whether a patient has a heart defect, and a congenital heart defect positioning detection module for providing an accurate position of the heart defect of the patient, wherein the ultrasonic cardiac video key frame selection module is a classification network designed based on ResNet; the latter is a detection network designed based on fast-RCNN; simultaneously using gray and color ultrasonic images of the echocardiogram as input of the system; the invention also comprises 5025 large-scale ultrasonic cardiograms of children, and the established deep learning model based on the ultrasonic cardiograms can diagnose common congenital heart defects at the same time. The experimental results show that when standard slices of a single echocardiogram are input, the diagnosis of three common congenital heart diseases on an external independent validation set reaches 100% sensitivity and specificity.

Description

Intelligent diagnosis system for congenital heart disease of children based on echocardiogram

Technical Field

The invention belongs to the technical field of medical image intelligent processing, in particular to an intelligent diagnosis system for congenital heart disease of children,

background

Congenital Heart Disease (CHD), abbreviated as Congenital heart disease, refers to a disease caused by abnormal cardiac anatomy due to the formation of heart and great vessels during embryonic development or abnormal development, or by the failure of the channel to close automatically after birth (which is normal during fetal life). CHD is the first birth defect in China, is the main cause of death of newborns and children, and has a morbidity rate of about 9 per mill in live newborns. Among all subtypes of CHD, Ventricular Septal Defect (VSD), Atrial Septal Defect (ASD), and Patent Ductus Arteriasus (PDA) are the three most common types.

ASD refers to abnormal development of atrial septa during embryonic period, which results in defect of the septa between left and right atria. ASD is one of the most common congenital heart diseases, accounting for 25% of all CHD. The defect location can be classified into secondary pore type ASD, primary pore type ASD, venous sinus type ASD and coronary venous sinus type ASD, wherein the secondary pore type ASD is the most common type. VSD is an abnormal traffic between the left and right ventricles, the most common congenital heart malformation, accounting for 50% of all CHDs. VSD can be further classified into perimembranous, infracristal, intraclass, infrasternal, and muscular defects, depending on the anatomical site. With the pericarp VSD being the most common, accounting for about 60% of all VSDs. The arterial duct is a conduit connecting between the descending part of the aortic arch and the pulmonary artery, and the fetal circulatory system depends on the existence of the ductus duct during the fetal period, but the ductus duct is naturally closed after birth, if the ductus duct fails to be closed, a channel between the aorta and the pulmonary artery is remained, which is called PDA. The incidence of isolated PDA in term infants is about 1/2000 in live babies, accounting for 5% to 10% of all congenital heart diseases.

A few of congenital heart diseases with mild symptoms can be naturally healed in the growth and development process of children. Severe congenital heart disease can lead to the death of the infant in infancy due to complications such as hypoxia, recurrent pneumonia or heart failure. Due to the lack of effective methods for the prevention of CHD, early identification and diagnosis of CHD is of great importance, not only as an important guarantee for the timely and effective treatment of congenital heart disease, but also as a key to reducing the natural mortality of the disease. Transthoracic echocardiography is considered to be the preferred method of detecting and diagnosing congenital heart disease, and is also the primary basis for planning surgical or interventional procedures and assessing efficacy. The complicated congenital heart disease is mostly expressed as obvious echocardiogram abnormity, is easy to be identified by doctors in primary hospitals and can be timely transferred to superior children hospitals for treatment. However, due to the inexperienced echocardiographs, there are still many congenital heart diseases that are diagnosed with delays, especially some subtypes of simple congenital heart diseases, such as ASD, VSD and PDA. Misdiagnosis or missed diagnosis often causes recurrent lower respiratory tract infection of the infant patient or progresses to the epstein-barr syndrome, misses the best operation opportunity, and seriously affects the prognosis and future life of the infant patient.

In recent years, artificial intelligence has been widely used for computer-aided diagnosis. CNN has been applied to several aspects of echocardiography diagnosis, including hypercardia plane identification^[1-2]Cardiac chamber delineation^[2-3]Measurement of cardiac structural parameters and cardiac function indices^[2,4]And so on. Madani et al used the echocardiogram of 267 subjects to train the CNN model to distinguish different cardiogram slices, and as a result, the model can automatically recognize 15 kinds of hypercardia views, the total accuracy rate reaches 91.7%, and lays a foundation for realizing full-automatic computer-aided echocardiogram diagnosis^[1]. The 13-layer CNN model constructed by Zhang and the like can draw each heart chamber under 5 conventional ultrasonic sections and automatically output estimated values of the length, the area, the volume and the like of the heart chamber, and the internal consistency of the output values is superior to that of manual measurement^[2]。

In the aspect of heart disease diagnosis based on echocardiography images, the application of CNN is still in the initial stage, has already preliminarily shown good performance, and has great clinical transformation potential. If the fat cardiomyopathy and the heart amyloidosis are detected by using the ultrasonic cardiogram image training CNN model of the long axis section of the left ventricle beside the sternum and the four-cavity section of the apex, the points under the ROC curve reach 0.93 and 0.87 respectively^[2]. However, at presentNo research report is found on an intelligent diagnosis model of the congenital heart disease based on the echocardiogram. In China, echocardiogram examination is popularized and popularized everywhere, and the establishment of an echocardiogram-based intelligent diagnosis system for congenital heart disease is beneficial to the sinking of high-quality medical resources to the basic level, so that patients in various regions and hospitals at all levels can enjoy large-data-based diagnosis decisions at the expert level or even higher than the expert level. In addition, huge amounts of clinical data and various medical image data are accumulated in the routine diagnosis and treatment work, and if the existing resources can be fully developed and utilized, the method has great significance for promoting the health of patients and the development of medical career.

Disclosure of Invention

In order to overcome the defects of the prior art, the invention aims to provide an echocardiogram-based intelligent diagnosis system for common child congenital heart diseases, which eliminates the influence of human factors and realizes intelligent diagnosis of various common child congenital heart diseases.

The invention constructs a first data set recorded by the echocardiography of the large-sized children, and the key views are marked by doctors with rich experience.

The echocardiogram-based common child congenital heart disease intelligent diagnosis system can diagnose three common congenital heart diseases. Specifically, the invention constructs a two-stage model of key frame selection and congenital heart defect positioning based on key frames in an echocardiogram video, and simultaneously takes two modes of the echocardiogram, namely gray and color echograms as input, and simultaneously improves the robustness and the precision of the model. The method specifically comprises the following steps: the system comprises an ultrasonic cardiac video key frame selection module and a congenital heart defect positioning detection module; wherein:

(1) the echocardiogram video key frame selecting module can greatly reduce the calculated amount of subsequent positioning models.

In order to help doctors diagnose the heart defect, the invention designs a module for selecting key frames from an ultrasonic cardiogram video, which can automatically predict whether the patient has the heart defect and give a corresponding confidence value. Specifically, the selection module uses a modified version of ResNet as a classification network model; the classification network model has 34 layers and consists of three parts:

the first part is an input head, which consists of a 7 × 7 convolution layer, a BN layer, a Relu layer and a pooling layer; after passing through the input head, converting the input data of six channels into feature maps of 64 channels;

the second part consists of four residual blocks with similar structures; the inside of the residual block is a series of convolution, BN and Relu cascade, and a jump link is added between two convolution layers, so that the gradient is easier to transfer back to an early layer during training, and the convergence of the model is accelerated;

the third part is a classification head which consists of a global average pooling function, a full connection layer and a Softmax function; wherein:

global average pooling is the calculation of the average of all pixels in each channel of the feature map, given by the following equation:

where w and h are the width and height of the feature map, and f (x, y) represents the value at (x, y) on the feature map;

the full connection layer converts the channel number of the feature map into the classified category number;

the Softmax function maps to the neuron output between [0,1] using the following equation:

wherein x is_iRepresenting the output value of the ith neuron in the fully-connected layer, S_iRepresenting x after SoftMax function_iThis can be seen as a probability value that the model considers that the patient belongs to the i-th class.

(2) Congenital heart defect positioning module

To further assist physicians in diagnosis and provide more information, the present invention develops a module for localization of congenital heart defects,can provide accurate position of the heart defect of the patient, and the congenital heart defect positioning module is based on fast-RCNN^[5]And (5) designing a detection network model. For a cardiac ultrasound image, the test model will output several test boxes, each representing a defect region, and the length of the diagonal of the test box representing the length of the defect. The intuitive auxiliary result can greatly improve the diagnosis efficiency of doctors. The detection model consists of three parts: the method comprises the steps that a characteristic extraction network and a candidate box generation network and output head part are obtained; wherein:

the feature extraction network adopts a pre-trained VGG or Resnet model, and has the main function of generating high-level features; here, the high-level feature refers to a feature that can reflect image semantics and understand context;

the candidate box generation network is used for generating a series of rectangular candidate boxes, each candidate box has a confidence score and represents the probability of the existence of the heart defect in the box. The detection model generates approximately 2000 candidate boxes for each input image. These candidate boxes are passed through a non-maximum suppression (NMS) algorithm^[6]And (5) further screening. Finally, the remaining candidate boxes and the advanced features generated by the feature extraction network are sent to an output header for final output. The candidate box generates network and feature extraction network sharing parameters.

The output head is provided with two parallel branches; the first branch uses the fully-connected layer and the Softmax function to predict the defect type in the candidate box, and the other branch outputs the position of the detection box.

In order to prevent overfitting caused by an excessively complex model, five times of cross validation experiments are carried out, namely different training sets and testing sets are selected for the model, and the model is trained for multiple times to obtain an optimized model. The method comprises the following specific steps:

firstly, uniformly mixing a patient image and a normal contrast image; then, dividing all the images into five parts, selecting four parts as a training set each time, selecting the rest parts as a test set, repeating the steps five times, and selecting different test sets each time; finally, the average of the five results was used as the final evaluation index of the model. Both the classification model and the detection model may be trained end-to-end.

In the classification model, cross entropy is used as a loss function of the model, and the formula is as follows:

wherein, y_iRepresenting the real category of the ith image, p_iIs the probability value that the model predicts that the ith image is a patient, and N is the number of images.

In the detection model, for the first branch, cross entropy is used as a classification loss function; for the second branch, a smoothed version of the first norm is used as the regression loss function, the formula is as follows:

where x is the difference between the predicted value and the true value. The overall loss function of the detection model is a weighted sum of the classification loss function and the regression loss function.

During testing, firstly, inputting an atrial septal image into a classification network model, diagnosing whether the patient has atrial septal defect by the classification network model, and if the patient is diagnosed as the atrial septal defect, inputting the image into a detection network model to provide the patient with the exact position of the atrial septal defect;

the present invention also provides the first dataset of large child echocardiographic recordings for training and testing with the system of the present invention, the key views being labeled by experienced physicians. The echocardiogram for children comprises two modalities-gray and color sonograms; the construction of the data set specifically comprises:

(1) data collection

A total of 5,025 children were collected from the pediatric hospital affiliated at the university of fudan, including 2,161 healthy controls, 1,039 ASDs, 965 VSDs, and 860 PDAs. Each member has an echocardiographic video and a still image sufficient for diagnosis. The PHILIPS iE 33 was used as the instrument and the frequency range of the sensor was 3-8MHz or 1-5 MHz. Two-dimensional imaging in conjunction with color doppler flow mapping can display the location, size, and direction of flow of the defect. According to the anatomical features of the three congenital heart diseases, whether the atrial septum, the ventricular septum and the left pulmonary artery have defects or not is observed. Two standard two-dimensional views and a color doppler flowgram were collected, with a short axis view of the parasternal aorta (PSSAX) for diagnosis of VSD and PDA, and two sub-xiphoid long axis views of the atrium (SXLAX) for diagnosis of ASD. The diagnosis of all patients was confirmed by at least two advanced sonographers or intraoperative final diagnoses.

(2) Data set labeling and quality control

Each echocardiogram consists of a video and a still image acquired simultaneously by the patient. The invention downloads the echocardiograms in all DICOM formats, and the key frames are manually selected by experienced doctors. The process of data annotation is as follows: each echocardiogram is evaluated by a three-stage evaluation system. Primary assessments were performed by medical students with a certain quality control basis. The second level of assessment was performed by two echocardiographers and the third level of assessment was performed by two echocardiographers with more than 10 years of clinical experience. This three-level assessment system ensures that each echocardiogram has the correct diagnostic label and lesion location. After completion of the training set data collection, 100 participants were randomly selected and examined by a third echocardiographer with over 20 years of clinical experience to minimize human error in the computational modeling process. Finally, 1,039 ASDs, 965 VSDs and 860 PDA patients were randomly selected for model training. The corresponding normal control groups were 629, 805 and 727, respectively.

To evaluate the present invention, 100 subjects were first randomly selected from the training set and the diagnostic performance of the initial test network model was verified internally. Second, external validation is performed using a validation dataset that is independent of the training set. External verification patients receiving treatment for congenital heart disease and children receiving echocardiography at the subsidiary pediatric hospital at the university of counterdenier, from 7/1/2020 to 11/30/2020 using echocardiography as prospective data. The criteria for including and excluding images are the same as those for the training data set. The external validation dataset included the subcoxiphoid sections of both atria and the parasternal aortic short axis section for each subject, including ASD, VSD and PDA of approximately 110 subjects. According to the result of a senior sonographer or the ROC curve of the final intraoperative diagnostic standard analysis model, the sensitivity, specificity and accuracy of the model can be evaluated.

The experimental result shows that when the standard section of a single echocardiogram is input, the diagnosis of the invention on the external independent verification set to three common congenital heart diseases can reach 100 percent of sensitivity and specificity.

The invention has the beneficial effects that: according to the large-scale two-dimensional standard view or color Doppler blood flow image data set collected by the invention, the CNN technology is utilized to establish an intelligent diagnosis model of common congenital heart disease, and the selected two-dimensional standard view or color Doppler blood flow image can be diagnosed. The model has high diagnostic accuracy for ASD, VSD and PDA, and has wide clinical impact in the future, especially in primary hospitals. For example, a physician may select a two-dimensional standard view or color Doppler flow imaging and make a preliminary diagnosis of a patient using an artificial intelligence model. The positive cases can then be transferred to specialized hospitals for further diagnosis and treatment. With the development of artificial intelligence and deep learning technology, the deep learning method can help doctors diagnose the congenital heart disease, so that diagnosis errors caused by artificial fatigue diagnosis can be reduced, and the auxiliary diagnosis and education training level of doctors in remote areas can be improved.

Drawings

FIG. 1 is a network framework diagram of the present invention.

FIG. 2 is a depiction of the short axis section of the parasternal aorta and the inferior cardiac section of the two atria under the sword for ASD, VSD, PDA.

FIG. 3 is an example of visualization of the results of a VSD, ASD, PDA congenital heart disease test using the system of the present invention. The red box is the doctor's annotation and the green box is the predicted outcome of the CHD network model.

Detailed Description

The embodiments of the present invention will be described in detail below, but the scope of the present invention is not limited to the examples.

The network structure in FIG. 1 is adopted, a GeForce RTX 2080Ti GPU is used for training a classification model and a detection model, and a Pythrch framework is used for constructing the model of the invention. For the classification model, the classification model is trained by minimizing the cross-entropy loss between the prediction classes and the true classes. In the optimization process, an Adam optimizer and a learning rate decay strategy are used. For the detection model, the classification branch is trained by minimizing the cross entropy loss between the prediction class and the real class, and the regression branch is trained by minimizing the difference between the real bounding box and the prediction bounding box. Similar to the classification model, an Adam optimizer was also used, with an initial learning rate of 0.0001. The hyper-parameters including confidence value, lot size and NMS threshold are set to 0.1, 6 and 0.1, respectively. The model may converge after 300 iterations.

The classification network model uses a modified version of ResNet, with 34 layers, consisting of three parts:

the first part is an input header whose main purpose is to extract low-level semantic information of the image, such as texture, shape, color, etc. The part consists of 7 by 7 convolutional layers, a BN layer, a Relu layer and a pooling layer; after passing through the input head, the input head converts the input data of six channels into feature maps of 64 channels, thereby extracting image feature representations in a high-dimensional space;

the second part is a residual block group, and the main purpose is to increase the image receptive field and extract higher-level semantic information. The residual block group consists of four residual blocks with similar structures, and a series of convolution, BN and Relu cascade connections are arranged inside the residual blocks. There is a skip link between the two convolutional layers, which adds the initial input of the residual block and the residual of the network prediction to retain more original information of the image. Meanwhile, the design also enables the gradient during training to be more easily transmitted back to the early layer, and accelerates the convergence of the model;

the third part is a classification head which classifies the images by using the image characteristics obtained by the above parts. The classification header consists of global average pooling, full connectivity layer and Softmax functions. And according to the extracted image features, the classification head predicts the probability of the image on each class, and takes the class with the maximum probability value as the final classification result of the image.

During testing, the atrial septal image is first input into a classification model, which will diagnose whether the patient has an atrial septal defect, and if the patient is diagnosed with an atrial septal defect, the image is input into a detection model, which will provide the patient with the exact location of the atrial septal defect.

The classification model is based on 5-fold cross validation of the classification effect of ASD, VSD and PDA as shown in Table 1.

The effect of the detection model on the abnormal areas of the cases of ASD, VSD and PDA is shown in table 2.

TABLE 1

Type (B)	Sensitivity of the composition	Specificity of the drug	AUC
				ASD	100％±0.0％	100％±0.0％	1.0±0.0
VSD	100％±0.0％	100％±0.0％	1.0±0.0
				PDA	100％±0.0％	100％±0.0％	1.0±0.0

TABLE 2

Type (B)	Accuracy of measurement	Recall rate	F1-score
				ASD	0.955	0.907	0.930
VSD	0.983	0.851	0.912
				PDA	0.922	0.922	0.922

FIG. 2 is a representation of the ASD, VSD, PDA, parasternal aortic short axis section and the inferior-to-xiphoid atrial section.

FIG. 3 is an example of visualization of VSD, ASD, PDA heart disease detection results using the system of the present invention. The red box is the doctor's annotation and the green box is the predicted outcome of the CHD network model.

Reference documents

[1]Madani A,Arnaout R,Mofrad M,et al.Fast and accurate view classification of echocardiograms using deep learning[J].NPJ Digit Med,2018,1:6.

[2]Zhang J,Gajjala S,Agrawal P,et al.Fully Automated Echocardiogram Interpretation in Clinical Practice[J].Circulation,2018,138(16):1623-1635.

[3]Li Y,Yi G,Wang Y,et al.Segmentation of Fetal Left Ventricle in Echocardiographic Sequences Based on Dynamic Convolutional Neural Networks[J].IEEE Transactions on Biomedical Engineering,2017,PP(8):1-1.

[4]

A,Smistad E,Aase SA,Haugen BO,Lovstakken L.Real-Time Standard View Classification in Transthoracic Echocardiography Using Convolutional Neural Networks[J].Ultrasound Med Biol.2019,45(2):374-384.

[5]Ren,S.,He,K.,Girshick,R.&Sun,J.Faster R-CNN:Towards Real-Time Object Detection with Region Proposal Networks.IEEE Trans.Pattern Anal.Mach.Intell.39,1137–1149(2017).

[6]Hosang,J.,Benenson,R.&Schiele,B.Learning non-maximum suppression.in Proceedings-30th IEEE Conference on Computer Vision and Pattern Recognition,CVPR 2017(2017)。

Claims

1. An echocardiogram-based intelligent diagnosis system for congenital heart disease of children is characterized by specifically comprising: the ultrasonic cardiogram video key frame selecting module and the congenital heart defect positioning detecting module take two modes of an ultrasonic cardiogram, namely a gray ultrasonic image and a color ultrasonic image, as the input of the system; wherein:

(1) the echocardiogram video key frame selecting module is used for automatically predicting whether the patient has heart defects or not and giving a corresponding confidence value; the selection module is a classification network model designed based on ResNet; the classification network model has 34 layers and consists of three parts:

the first part is an input head and consists of 7 by 7 volume layers, a BN layer, a Relu layer and a pooling layer; after passing through the input head, converting the input data of six channels into feature maps of 64 channels;

the second part consists of four residual blocks with similar structures; the inside of the residual block is cascaded by a series of convolutions, BN and Relu, and a jump link is added between two convolution layers, so that the gradient is easier to transfer back to an early layer during training, and the convergence of the model is accelerated;

the full connection layer converts the channel number of the feature graph into a classified category number;

wherein x is_iRepresenting the output value of the ith neuron in the fully-connected layer, S_iX after representing SoftMax function_iThis can be seen as the probability value that the model considers the patient to belong to the ith class;

(2) the congenital heart defect positioning module is used for providing an accurate position of the heart defect of the patient; the congenital heart defect positioning module is based on fast-RCNN^[5]A designed detection network model; three parts of detection network modelComprises the following components: the method comprises the steps that a network and an output head part are generated by a feature extraction network and a candidate box; wherein:

the feature extraction network adopts a pre-trained VGG or Resnet model for generating high-level features;

the candidate box generation network is used for generating a series of rectangular candidate boxes, each candidate box has a confidence score and represents the probability of the existence of the heart defect in the box; further screening a series of rectangular candidate frames through a non-maximum suppression algorithm, and finally sending the remaining candidate frames and high-level features generated by the feature extraction network to an output head for final output;

the output head comprises two parallel branches: the first branch uses the fully-connected layer and the Softmax function to predict the type of defect in the candidate box, and the other branch outputs the position of the detection box.

2. The intelligent diagnosis system for child congenital heart disease according to claim 1, wherein the classification network model and the detection network model in the system are trained for a plurality of times by selecting different training sets and test sets, and finally an optimized model is obtained; the specific process is as follows:

firstly, uniformly mixing a patient image and a normal contrast image; then, dividing all the images into five parts, selecting four parts as a training set each time, selecting the rest parts as a test set, repeating the steps for five times, and selecting different test sets each time; finally, taking the average value of the five results as the final evaluation index of the model; the classification network model and the detection network model are trained end to end;

in a classification network model, cross entropy is used as a loss function of the model, and the formula is as follows:

wherein, y_iRepresenting the true class of the ith image, p_iIs the probability value of the model predicting the ith image as the patient, N is the number of imagesAn amount;

in the detection network model, for the first branch, using cross entropy as a classification loss function; for the second branch, a smoothed version of the first norm is used as the regression loss function, the formula is as follows:

wherein x is the difference between the predicted value and the true value; the total loss function of the test network model is a weighted sum of the classification loss function and the regression loss function.

3. The intelligent diagnostic system for children's congenital heart disease according to claim 1 or 2, further comprising a data set of children's echocardiography records for classification network model and detection network model training and testing in the system, the data set comprising: 5,025 children collected from the hospital, 2,161 healthy controls, 1,039 ASDs, 965 VSDs, and 860 PDAs, each with echocardiographic video and still images; wherein the key views are marked by experienced physicians.