WO2022047627A1 - Deep learning prediction method and application thereof - Google Patents

Abstract

A deep learning prediction method and an application thereof, relating to the technical field of imaging. Current Covid-19 CT image diagnostic technologies rely on single images, require a technologist to select a single image from a CT image sequence so as to accomplish a diagnostic estimation, and are thus time-consuming for the technologist. For diagnosis of an individual patient, multiple frames of continuous CT images have higher error tolerance than single images. The method comprises: pre-processing symptom data and image data; fusing symptom features and image features; performing channel feature extraction on fused data; designing a loss function; using an Adam optimization algorithm to optimize; constructing, according to a patient dataset, pairing as network input; and training the network to obtain the deep learning prediction method. Accuracy can be improved using the symptom information of the patient, which can be quickly accessed clinically.

Description

A deep learning estimation method and its application technical field

The present application belongs to the field of image imaging technology, and in particular relates to a deep learning estimation method and its application.

Background technique

CT examination is a modern, more advanced medical scanning technology, mainly for scanning the human brain. CT examination generally includes plain CT, contrast-enhanced CT and cisternal contrast CT. CT scans a layer of a certain thickness of a certain part of the human body with an X-ray beam. The detector receives the X-ray that passes through the layer, converts it into visible light, and converts it from photoelectric to electrical signal. /digitalconverter) into numbers and input to the computer for processing.

[Correction 27.09.2020 under Rule 91]
Imaging examination is one of the quick and convenient means of medical diagnosis. Chest X-ray examination has a high missed diagnosis rate. CT, especially high resolution CT (HRCT), plays an important role in the diagnosis of this disease.

The current COVID-19 CT impact diagnosis technology relies on a single image, and the radiologist needs to select a single image from the CT sequence images to complete the diagnosis estimation, which takes up a lot of the radiologist's time; for a single patient, multiple consecutive CT images are essential for disease diagnosis. The error tolerance rate is higher than that of a single image.

SUMMARY OF THE INVENTION

1. Technical problems to be solved

Based on the current COVID-19 impact diagnosis technology relying on a single image, the radiologist needs to select a single image from the CT sequence images to complete the diagnosis estimation, which takes up a lot of radiologist time; for a single patient, multiple consecutive CT images are critical to the disease. To diagnose the problem that the error tolerance rate is higher than that of a single image, the present application provides a deep learning estimation method and its application.

2. Technical solutions

In order to achieve the above-mentioned purpose, the present application provides a deep learning estimation method, the method includes the following steps:

Step 1: Preprocess symptom data and image data; use convolution (convolution size 3x3) to extract features from image data, and use convolution (convolution size 1x1) to extract features from symptom data;

Step 2: Fusion of symptom features and image features; let the attention mechanism fuse the symptom features into the image features, the symptom features are mapped to channel masks (values are distributed between 0-1) through convolution, and the mask and Image feature point multiplication to enhance and suppress the data of some channels;

Step 3: Perform channel feature extraction on the fused data; use average channel pooling and maximized channel pooling to convert the fused data into vector form, thereby compressing the fused data, and then use convolution to convert the converted vector perform feature extraction;

Step 4: Designing the Loss Function Our proposed network model functions like a function, with corresponding predicted output results for the data input. In the training process, the error measurement between the predicted data results and the real data results is completed by designing a loss function;

Step 5: Use the Adam optimization algorithm to optimize; use the Adam optimization algorithm to optimize the loss function in step 4, and complete the update of the parameters of the network model (including the convolution in steps 1, 2, and 3);

Step 6: Construct pairings based on patient datasets as network inputs;

Step 7: Train the network to obtain the deep learning estimation method.

Further, preprocessing the symptom data in the step 1 includes coding the symptom data, according to the clinical symptoms of the patient, if the patient has specific symptoms, the bit value corresponding to the symptom code is set to 1, otherwise it is 0. .

Another embodiment provided by the present application is: in the step 1, the preprocessing of the symptom data includes coding the symptom data, and according to the clinical symptoms of the patient, if the patient has specific symptoms, the corresponding bit of the symptom code is displayed. The value is set to 1, otherwise 0.

Another embodiment provided by the present application is: the clinical symptoms include fever, cough, muscle aches, fatigue, headache, nausea, diarrhea, abdominal pain, and dyspnea.

Another embodiment provided by the present application is: the symptom data code also includes gender and age.

Another embodiment provided by the present application is: the image data selects a continuous image sequence as the image data inputted by the network data; the image sequence data and the image sequence data first undergo a convolution to perform rough image feature extraction.

Another embodiment provided by the present application is: the image is a CT image.

Another embodiment provided by the present application is: in the step 2, a symptom information fusion unit is used to complete the fusion of image features and symptom features.

Another embodiment provided by the present application is: the symptom information fusion unit includes several symptom information fusion modules, the symptom information fusion modules are cascaded together, and the symptom information fusion modules use residual connection.

Another embodiment provided by the present application is: in the step 3, channel average pooling and channel maximization pooling in the prediction module are used to complete channel feature extraction.

The application also provides an application of a deep learning estimation method, and the deep learning estimation method is applied to the early clinical classification and diagnosis of new coronary pneumonia or the diagnosis and evaluation of other diseases.

3. Beneficial effects

Compared with the prior art, the beneficial effects of a deep learning estimation method provided by this application are:

The deep learning estimation method provided in this application is a deep learning estimation method for early clinical classification and diagnosis of new coronary pneumonia, which can effectively help radiographers to diagnose new coronary pneumonia, optimize the diagnosis process, and save medical resources.

The deep learning estimation method provided in this application can improve the diagnostic accuracy by combining the patient's clinical symptoms and CT images.

The deep learning estimation method provided in this application solves the problem of early clinical classification of new coronary pneumonia and the improvement of diagnostic accuracy.

The deep learning estimation method provided by the present application is a rapid diagnosis network integrating clinical symptoms based on deep learning technology, and the input of the network includes CT image sequences and clinical symptoms of patients. As a priori knowledge, clinical symptoms are added to the image features through the symptom information fusion module, and then the prediction module is used to complete the diagnosis estimation of the patient (whether you have COVID-19) and clinical classification estimation (specific degree of COVID-19: mild or severe).

The deep learning prediction method provided in this application can improve the fault tolerance rate of diagnosis based on a CT image sequence instead of a single CT image.

The deep learning estimation method provided in this application uses a continuous CT image sequence to reduce the time it takes for the pharmacist to select a specific single sheet, and the lung image data can be directly input into the network method.

The deep learning estimation method provided in this application can improve the accuracy rate in combination with the patient's symptom information, and at the same time, the patient's symptom information can be quickly obtained in the clinic.

The deep learning estimation method provided in this application uses channel averaging and maximizing pooling instead of the usual fully connected layers, which can effectively reduce network parameters.

Description of drawings

FIG. 1 is a schematic diagram of the network architecture relationship of the deep learning estimation method of the present application.

detailed description

Hereinafter, specific embodiments of the present application will be described in detail with reference to the accompanying drawings, from which those skilled in the art can clearly understand the present application and be able to implement the present application. Without departing from the principles of the present application, the features of the various embodiments may be combined to obtain new embodiments, or instead of certain features of certain embodiments, to obtain other preferred embodiments.

Referring to FIG. 1, the present application provides a deep learning estimation method, and the method includes the following steps:

Step 1: Preprocess symptom data and image data; use convolution (convolution size 3x3) operation to extract features from image data, and use convolution (convolution size 1x1) to perform feature extraction on symptom data.

Step 2: Fusion of symptom features and image features; let the attention mechanism fuse the symptom features into the image features, the symptom features are mapped to channel masks (values are distributed between 0-1) through convolution, and the mask and Image feature point multiplication to enhance and suppress the data of some channels.

Step 3: Perform channel feature extraction on the fused data; use average channel pooling and maximized channel pooling to convert the fused data into vector form, thereby compressing the fused data, and then use convolution to convert the converted vector Perform feature extraction.

Step 4: Designing the Loss Function Our proposed network model functions like a function, with corresponding predicted output results for the data input. During the training process, the error measurement between the predicted data results and the real data results is done by designing a loss function.

Step 5: Use the Adam optimization algorithm to optimize; use the Adam optimization algorithm to optimize the loss function in step 4, and complete the update of the parameters of the network model (including the convolution in steps 1, 2, and 3).

Step 6: Construct pairings based on patient datasets as network inputs;

Step 7: Train the network to obtain the deep learning estimation method. Further, preprocessing the symptom data in the step 1 includes coding the symptom data, according to the clinical symptoms of the patient, if the patient has specific symptoms, the bit value corresponding to the symptom code is set to 1, otherwise it is 0. .

Further, the clinical symptoms include fever, cough, muscle aches, fatigue, headache, nausea, diarrhea, abdominal pain, and dyspnea.

Further, the symptom data code also includes gender and age.

Further, the image data selects a continuous image sequence as the image data inputted by the network data; the image sequence data first undergoes a convolution to perform rough image feature extraction.

Further, the image is a CT image.

Further, in the step 2, a symptom information fusion unit is used to complete the fusion of image features and symptom features.

Further, the symptom information fusion unit includes several symptom information fusion modules, the symptom information fusion modules are cascaded together, and the symptom information fusion module adopts residual connection.

Further, in the step 3, channel average pooling and channel maximization pooling in the prediction module are used to complete channel feature extraction.

The present application also provides an application of a deep learning estimation method, wherein the deep learning estimation method described in any one of claims 1 to 9 is applied to the early clinical classification and diagnosis of new coronary pneumonia or the diagnosis and evaluation of other diseases.

Example

Step 1: Symptom data and CT image data preprocessing

Code the symptom data according to the clinical symptoms of the patient (fever, cough, muscle pain, fatigue, headache, nausea, diarrhea, abdominal pain, dyspnea), if the patient has specific symptoms, the corresponding bit value in the symptom code Set to 1, otherwise 0. In addition, taking into account the gender and age of the patient, the symptom data coding needs to add gender (male 1, female 2) and age.

For the CT image data, a continuous image sequence (eg, 160 consecutive CT images) containing the lung region (from the upper lung to the lower lung) is selected as the image data for the network data input. The image sequence data is first subjected to a convolution (the convolution kernel is 1x1x32) for rough extraction of image features.

Step 2: Design symptom information fusion module

As shown in Figure 1, for the input information of the module, first use two convolutions for feature extraction to obtain the feature Fs. Batch regularization is to constrain the data to make the output follow a normal distribution with a mean of 0 and a variance of 1, so as to avoid variable Distribution shift problem. The symptom code is mapped to channel feature Fc through convolution Hcap, which is the same as the number of convolution feature channels. The formula is expressed as

F _c =H _cap (F _s )

Among them, Hcap represents the convolution operation.

The convolved features are subjected to channel average pooling, and the image features are mapped to the channel features Fe(1x1x32), and the formula is expressed as

F _e =H _e (y)

where He is denoted as a convolution operation (1x1x32), and y is the symptom encoding.

Similar to the self-attention mechanism, the symptom information is fused into the image features, and the formula is expressed as

F _CA = Sigmoid(H _ca (Sigmoid(F _e )*F _c ))

Among them, Hca represents the convolution operation (1x1x32), Simoid represents the activation function to map the value between 0-1, and '*' represents the dot product operation.

A total of 5 symptom information fusion modules are cascaded together. For each cascade module, residual connection is used to reduce information loss, and the formula is expressed as

in,

is expressed as the input information of the ith cascade module,

Represented as the output information of the ith cascaded module.

Table 1 Parameter settings of symptom information fusion module

part	convolution kernel
Convolution 1	3x3x32x16
Convolution 2	3x3x16x32
Convolution 3 (He)	1x1x11x32
Convolution 4 (Hca)	1x1x32x32

Step 3: Set up the prediction module

Based on the process in step 2, the feature processing result M ⁿ is obtained, and two channel features are obtained by channel average pooling and channel maximization pooling. The formula is expressed as:

F _SK = H _SK (Concatenation(H _cap (M ⁿ ), H _cmp (M ⁿ )))

Among them, Hsk represents convolution (the convolution kernel is 1x1x64x32), Hcap represents the channel average pooling operation, and Hcmp represents the channel maximization pooling operation.

Based on the fusion feature Fsk, the diagnostic estimate z1 is predicted by convolution Hd (convolution kernel is 1x1x32x2), and the formula is expressed as:

z ₁ =H _d (F _SK )

Based on the fusion feature Fsk, the clinical classification estimate z2 is predicted by convolution Hs (convolution kernel is 1x1x32x3), and the formula is expressed as:

z ₂ =H _s (F _SK )

Step 4: Design the Loss Function

Given a training dataset D = {(x ₁ , y ₁ , r ₁ , s ₂ ), (x ₂ , y ₂ , r ₂ , s ₂ ),...,(x _n , y _n , r _n , s _n )}, where xi is the CT image sequence scanned by the ith patient, yi is the symptom code of the ith patient, n is the total number of training samples, and ri is the COVID-19 diagnosis result of the ith patient (0 means normal , 1 means suffering from the disease), si is the diagnosis result of COVID-19 clinical classification of the i-th patient (0 means normal, 1 means mild patient, 2 means severe patient). _x = _{{ x 1} , _{x 2} , . The loss function is expressed as

Loss=a*CrossEntropy(G(x,y),r)+b*CrossEntropy(G(x,y),s)

where a and b represent balance factors, a=1 and b=1. G denotes the proposed deep learning method. CrossEntropy is represented as a cross-entropy loss function.

Step 5: Use Adam optimization algorithm to optimize.

Step 6: Construct pair D from the patient dataset as network input.

Step 7: Train the network to obtain the deep learning prediction method G for early clinical classification and diagnosis of new coronary pneumonia.

Table 2: Experimental Results

The method proposed in this application can be applied to disease prediction, such as the new crown disease. By obtaining the CT image data of the patient and the clinical symptom information of the corresponding patient, the method of the present application can be used to predict whether the current patient has the new crown disease, and at the same time, when the patient has the disease, the severity of the patient's disease (normal/severe) can be predicted. .

Although the present application has been described above with reference to specific embodiments, it will be understood by those skilled in the art that many modifications may be made in the configuration and details disclosed herein within the spirit and scope of the present disclosure. The scope of protection of the present application is determined by the appended claims, and the claims are intended to cover all modifications encompassed by the literal meaning or scope of equivalents to the technical features in the claims.

Claims (10)

1. A deep learning estimation method, characterized in that: the method comprises the following steps:

   Step 1: Preprocess symptom data and image data;

   Step 2: Fusion of symptom features and image features;

   Step 3: Perform channel feature extraction on the fused data;

   Step 4: Design a loss function, and complete the error measurement between the predicted data result and the real data result by designing the loss function;

   Step 5: Use the Adam optimization algorithm to optimize the loss function in step 4 to complete the parameter update of the network model;

   Step 6: Construct pairings based on patient datasets as network inputs;

   Step 7: Train the network to obtain the deep learning estimation method.
2. The deep learning estimation method according to claim 1, wherein the preprocessing of the symptom data in the step 1 includes coding the symptom data, according to the clinical symptoms of the patient, if the patient has specific symptoms, then The bit value corresponding to the symptom code is set to 1, otherwise it is 0.
3. The deep learning estimation method according to claim 2, wherein the clinical symptoms include fever, cough, muscle aches, fatigue, headache, nausea, diarrhea, abdominal pain, and dyspnea.
4. The deep learning estimation method according to claim 2, wherein the symptom data code further includes gender and age.
5. The deep learning prediction method according to claim 1, wherein: the image data selects a continuous image sequence as the image data inputted by the network data; extract.
6. The deep learning prediction method according to any one of claims 1 to 5, wherein the image is a CT image.
7. The deep learning estimation method according to claim 1, wherein in the step 2, a symptom information fusion unit is used to complete the fusion of image features and symptom features.
8. The deep learning estimation method according to claim 7, wherein the symptom information fusion unit comprises several symptom information fusion modules, the symptom information fusion modules are cascaded together, and the symptom information fusion module adopts residual error connect.
9. The deep learning estimation method according to claim 1, wherein in said step 3, channel average pooling and channel maximization pooling in the prediction module are used to complete channel feature extraction.
10. An application of a deep learning estimation method, characterized in that the deep learning estimation method according to any one of claims 1 to 9 is applied to the early clinical classification and diagnosis of new coronary pneumonia or the diagnosis and evaluation of other diseases.

Images