CN114795114B - Carbon monoxide poisoning delayed encephalopathy prediction method based on multi-modal learning - Google Patents
Carbon monoxide poisoning delayed encephalopathy prediction method based on multi-modal learning Download PDFInfo
- Publication number
- CN114795114B CN114795114B CN202210348106.5A CN202210348106A CN114795114B CN 114795114 B CN114795114 B CN 114795114B CN 202210348106 A CN202210348106 A CN 202210348106A CN 114795114 B CN114795114 B CN 114795114B
- Authority
- CN
- China
- Prior art keywords
- carbon monoxide
- feature
- monoxide poisoning
- encephalopathy
- vector
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7271—Specific aspects of physiological measurement analysis
- A61B5/7275—Determining trends in physiological measurement data; Predicting development of a medical condition based on physiological measurements, e.g. determining a risk factor
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Physics & Mathematics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physiology (AREA)
- Psychiatry (AREA)
- Signal Processing (AREA)
- Veterinary Medicine (AREA)
- Public Health (AREA)
- Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Mathematical Physics (AREA)
- Evolutionary Computation (AREA)
- Fuzzy Systems (AREA)
- Magnetic Resonance Imaging Apparatus (AREA)
Abstract
The invention belongs to the technical field of medical information, and particularly relates to a carbon monoxide poisoning delayed encephalopathy prediction method based on multi-modal learning, which constructs a characteristic dependency graph by combining a Bayesian network structure learning algorithm and doctor suggestions; based on the characteristic dependency graph, embedding the building graph into a feedforward neural network, and integrating the dependency relationship among the characteristics into a neural network structure to obtain a vector representation of the structural characteristics; and designing a feature fusion structure, obtaining vector representation fused with multi-mode information, inputting the vector into a nonlinear neural network layer, and finally obtaining a prediction result of whether a patient with carbon monoxide poisoning will have delayed encephalopathy. The invention fully fuses the multi-mode diagnosis and treatment information, only needs structured data as model input when being applied to early prediction, thus reducing the requirement of a prediction model on the integrity of the multi-mode data in practical application to a certain extent and solving the problem that accurate prediction cannot be realized under the condition of data loss.
Description
Technical Field
The invention belongs to the technical field of medical information, and particularly relates to a carbon monoxide poisoning delayed encephalopathy prediction method based on multi-modal learning.
Background
Carbon monoxide poisoning is a common unexpected incident in daily life. After the toxic symptoms are relieved, 10% -30% of patients suffering from carbon monoxide poisoning have a probability of suffering from mental system diseases, namely delayed encephalopathy, which are mainly dementia, mental symptoms and extrapyramidal symptoms after a pseudo-healing period of 2-60 days. The late encephalopathy has low awareness rate, serious illness state and long illness course, once the late encephalopathy occurs, the patient loses work ability, people need nursing in daily life, most of treatment time is as long as half a year or 1 year or longer, and serious people can leave over permanent nerve function disabilities such as cognitive dysfunction and the like, thereby influencing life quality, increasing economic burden and even being poor due to illness. Therefore, the early prediction of the risk of the carbon monoxide poisoning delayed encephalopathy is beneficial to the timely intervention of the diseases, the incidence frequency of the diseases is reduced, and the burden of the diseases on families and society is effectively reduced.
At present, the existing delayed encephalopathy prediction mainly adopts a traditional medical statistical method to fit historical data, and the risk factors of delayed encephalopathy patients in the early stage of carbon monoxide poisoning are found, but the method is single and lacks accuracy.
Currently, clinical examination indicators, electroencephalograms (EEG), brain CT and brain MRI are effective means for finding abnormal signs in patients with carbon monoxide poisoning. The accuracy of the delayed encephalopathy risk prediction can be effectively improved by fully utilizing multi-modal data (including structural data such as clinical indexes, electroencephalogram, brain CT and brain MRI). However, in practical applications, since there is often a problem of partial examination data missing in patients at an early stage of carbon monoxide poisoning, this will affect the accuracy of the delayed encephalopathy risk prediction.
Disclosure of Invention
In order to solve the technical problems in the prior art, the invention provides a method for predicting the carbon monoxide poisoning delayed encephalopathy based on multi-mode learning, which aims to solve the technical problems in the background art that the accuracy of predicting the risk of the delayed encephalopathy is affected by the loss of check data.
In order to solve the technical problems, the invention adopts the following technical scheme:
a method for predicting carbon monoxide poisoning delayed encephalopathy based on multi-modal learning comprises the following steps:
step 1: acquiring multi-mode data of a patient suffering from carbon monoxide poisoning, and preprocessing the multi-mode data to obtain a data set of the patient suffering from carbon monoxide poisoning;
step 2: based on the data set obtained in the step 1, constructing a feature dependency graph through a Bayesian network structure learning algorithm and doctor suggestions, wherein nodes represent features in the feature dependency graph, and edges represent the relationship among the features and are embodied as the dependency relationship among the features; then deleting the nodes of the characteristic dependency graph, which represent the delayed encephalopathy labels of the carbon monoxide poisoning, and reserving the nodes representing the characteristics; all features appearing in the feature dependency graph form a set F, and a corresponding adjacency matrix A is obtained based on the structure of the feature dependency graph;
step 3: respectively establishing a feature extraction model based on the data types in the data set, and extracting multi-mode data feature vectors based on the feature extraction model;
step 4: constructing a graph embedded feedforward neural network based on a characteristic dependency graph;
step 5: fusing the multi-mode data feature vectors to obtain fusion vectors C1 and C2;
step 6: establishing a carbon monoxide poisoning delayed encephalopathy prediction model based on the graph embedded feedforward neural network and fusion vectors C1 and C2;
step 7: and testing the carbon monoxide poisoning delayed encephalopathy prediction model, and selecting parameters meeting the prediction performance requirements as parameters of the carbon monoxide poisoning delayed encephalopathy prediction model.
The prediction model of the carbon monoxide poisoning delayed encephalopathy fully fuses multi-mode diagnosis and treatment information during training, only structured data is needed to be used as model input during application and early prediction, so that the requirement of the prediction model on multi-mode data integrity in practical application is reduced to a certain extent, and the problem that accurate prediction cannot be realized under the condition of partial detection data loss is solved.
Preferably, the multi-modal data includes images, waveforms, and structured data;
the image data includes brain CT and brain MRI; the oscillogram data includes an electroencephalogram; the structured data includes first visit information of CO poisoning, auxiliary examination and treatment condition.
Preferably, in the step 1, if the acquired patient has data missing in a certain mode, a missing mark is performed;
the step 1 further comprises the steps of obtaining follow-up information of the patient suffering from carbon monoxide poisoning and constructing a prediction label; if the patient suffers from delayed encephalopathy within a period of time after carbon monoxide poisoning, the predictive label is 1, otherwise, the predictive label is 0.
Preferably, the feature extraction model in step 3 includes: a feature extraction model M1, a feature extraction model M2, and a feature extraction model M3;
the feature extraction model M1 is composed of a convolution module of EEGNet, wherein the convolution module is composed of three convolution layers, namely Conv2D, depthwiseConv2D and SeparableConv2D in sequence;
the feature extraction model M2 and the feature extraction model M3 are respectively composed of a first convolutional neural network and a second convolutional neural network.
Preferably, the feature extraction model M1, the feature extraction model M2 and the feature extraction model M3 are all obtained by training in a training set 1 divided from a data set; respectively training electroencephalogram, brain CT and brain MRI data in a data set to obtain three classifiers, adjusting parameters of a feature extraction model to enable classification performance of the corresponding classifier to meet preset requirements, removing a sigmoid classification layer in the classifier, extracting convolution structures of the classifier meeting preset conditions under three modes, and forming feature extraction models M1, M2 and M3; the feature extraction models M1, M2 and M3 are used to extract an electroencephalogram feature vector E2, a brain CT feature vector E3 and a brain MRI feature vector E4, respectively.
Preferably, the definition of the graph-embedded feedforward neural network described in step 4 is as follows:
E1＝MLP(σ(x(W·A)+b))；
wherein: a is an adjacency matrix of the feature dependency graph; x is a feature vector formed by the values of the structural features of the patient, wherein the structural features of the patient comprise the features appearing in the feature set F in the step 2; w and b are training parameters; representing the Hadamard product operation; sigma represents an activation function; MLP is a multi-layer perceptron.
The graph embedded feedforward neural network part of the invention is embedded with the characteristic dependency graph in the step 2, so as to integrate the dependency relationship among the characteristics and further construct the characteristic vector of the structural characteristics.
Preferably, the step 5 includes the steps of:
step 5.1: taking the multi-mode data feature vector extracted in the step 3 as a training sample [ x, E2, E3, E4, y ]; the structural feature vector x is embedded into a neural network code through a graph to obtain a feature vector E1;
step 5.2: the feature vector E1 is converted into an electroencephalogram feature vector E2 based on the first layer encoder-decoder, and a fusion vector C for capturing joint information between the feature vector E1 and the electroencephalogram feature vector E2 is generated in the conversion process:
C＝f E-D (E1,E2)；
wherein: f (f) E-D () An encoder-decoder unit representing a first layer, obtaining a fusion vector representation C; wherein the encoder-decoder unit may be of neural network architecture;
step 5.3: the fused vector C is input to a coder-decoder unit of the second layer, and conversion of the fused vector C and brain CT and brain MRI eigenvectors E3 and E4 is realized. In the encoder-decoder unit of the second layer, the conversion of fusion vector C into brain CT and brain MRI uses one encoder and two separate decoders, which in the encoding and decoding process produce fusion vector representations C1 and C2, respectively:
C1＝f E1-D1 (C,E3)；
C2＝f E1-D2 (C,E4)；
wherein: f (f) E1-D1 ()、f E1-D2 () An encoder-decoder unit representing a second layer for obtaining fusion vector representations C1 and C2, respectively; wherein the encoder-decoder unit may be of neural network architecture.
The feature fusion of the invention designs a two-layer encoder-decoder unit, which respectively fuses the electroencephalogram features and the brain image features (including brain CT and brain MRI), so that the fusion features specific to the prediction task of the delayed encephalopathy caused by carbon monoxide poisoning can be fully mined, and the applicability and the stability of the prediction model of the delayed encephalopathy caused by carbon monoxide poisoning can be enhanced.
Preferably, the step 6 is specifically as follows:
splicing the fusion vectors C1 and C2, inputting the spliced fusion vectors C1 and C2 into a full-connection layer, and outputting the probability of delayed encephalopathy of a patient suffering from carbon monoxide poisoning through nonlinear fitting:
wherein: i represents vector concatenation, W, b is a training parameter, g represents an activation function,
a predictive output indicative of whether there is a risk of tardive encephalopathy;
the loss function of the predictive model of delayed encephalopathy in carbon monoxide poisoning is defined as:
L＝λ 1 L E2 +λ 2 L E3 +λ 3 L E4 +λ p L p ；
wherein: l (L) Ei (i=2, 3, 4) represents an error generated in the feature fusion process, L p Representing errors resulting from classification predictions. Lambda (lambda) i (i=1, 2, 3) is a super parameter, and the duty ratio of the loss functions of different parts can be weighed according to actual conditions;
specifically, loss L of predicted portion p Is calculated by the following steps:
L p ＝-y log p-(1-y)log(1-p)；
wherein: y is the real label of a patient sample with carbon monoxide poisoning, delayed encephalopathy occurs, the label is 1, otherwise, the label is 0; p represents the probability that a patient suffering from carbon monoxide poisoning is predicted to develop a delayed encephalopathy;
loss of fusion moiety L Ei Is calculated by the following steps:
in the method, in the process of the invention,
the outputs of the decoders described in step 5.2 and step 5.3, respectively, ei (i=2, 3, 4) represent the vector representations of the electroencephalogram, brain CT and brain MRI described in step 3, respectively.
Preferably, the structured feature vector x in the test set is input into a carbon monoxide poisoning delayed encephalopathy prediction model, and the prediction performance of the carbon monoxide poisoning delayed encephalopathy prediction model is estimated by the accuracy, recall rate and F1 index based on the prediction result of whether the patient of the carbon monoxide poisoning delayed encephalopathy prediction model will develop delayed encephalopathy.
Preferably, the data set in the step 1 is divided into a training set and a testing set, the training set is divided into a training set 1 and a training set 2, and the training set 1 is used for training a feature extraction model and a feature dependency graph; the training set 2 is used for training a carbon monoxide poisoning delayed encephalopathy prediction model, no data is lost in the training set, and data such as brain CT, brain MRI or electroencephalogram can be lost in the test set.
The beneficial effects of the invention include:
1. the feature dependency graph can mine dependency relationships among features and serve as priori knowledge to enhance the prediction accuracy of the carbon monoxide poisoning delayed encephalopathy prediction model. The graph embedded feedforward neural network selects the characteristics based on the structure of the characteristic dependency graph, so that the connection relationship between the input layer and the hidden layer of the feedforward neural network becomes sparse.
2. The multi-modal feature fusion of the invention designs a two-layer encoder-decoder unit, and mines the fusion features specific to the prediction task of the delayed encephalopathy caused by carbon monoxide poisoning. Compared with the simple combination of different modal characteristics by means of a concentration mechanism and the like, the prediction model of the carbon monoxide poisoning delayed encephalopathy designs a special multi-modal characteristic learning process, and loss generated by part of modal conversion participates in optimization of the model, so that multi-modal information can be more effectively utilized.
3. And constructing a robust and accurate prediction model of the delayed encephalopathy caused by carbon monoxide poisoning through the integration of the feature dependency graph and the fusion of multi-modal information. The prediction model of the carbon monoxide poisoning delayed encephalopathy fully fuses multi-mode diagnosis and treatment information during training; and when the method is applied to early prediction, only structured data is needed to be used as input of a carbon monoxide poisoning delayed encephalopathy prediction model. Therefore, the method reduces the demand of the carbon monoxide poisoning delayed encephalopathy prediction model on the integrity of the multi-mode data in practical application to a certain extent, and solves the problem that accurate and reliable prediction cannot be realized under the condition of partial check data loss.
Drawings
FIG. 1 is a schematic representation of different modal feature vectors of the present invention.
Fig. 2 is a schematic diagram of the training of the prediction model of the delayed encephalopathy caused by carbon monoxide poisoning according to the present invention.
Fig. 3 is a schematic diagram of the training and testing flow of the carbon monoxide poisoning delayed encephalopathy prediction model of the present invention.
Fig. 4 is a schematic overall flow chart of the present invention.
Fig. 5 is a schematic representation of the characteristic dependency of the present invention.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, which are generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, as provided in the accompanying drawings, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, are intended to be within the scope of the present application.
The invention is further described in detail below with reference to fig. 1 and 5:
a method for predicting carbon monoxide poisoning delayed encephalopathy based on multi-modal learning comprises the following steps:
step 1: acquiring multi-mode data of a patient suffering from carbon monoxide poisoning, and preprocessing the multi-mode data to obtain a data set of the patient suffering from carbon monoxide poisoning;
the multi-modal data includes an image, a waveform diagram, and structured data;
the image data includes brain CT and brain MRI; the oscillogram data includes an electroencephalogram; the structured data includes first visit information of CO poisoning, auxiliary examination and treatment condition.
Specifically, the first diagnosis information of the CO poisoning includes: age, sex, occupation, mode of poisoning, raw material of poisoning, location of poisoning, labor type before poisoning, cultural degree, past medical history, time of poisoning, time of treatment for first hyperbaric oxygen treatment, clinical manifestations at time of poisoning (headache, vomiting, limb weakness, coma, palpitation, dyspnea, etc.), coma time, glasgow score at coma, conscious state at time of first hyperbaric oxygen; the auxiliary examination and treatment condition information includes: blood carboxyhemoglobin, blood qi, blood routine, C-reactive protein, D dimer, liver function, blood lipid, blood glucose and some specific biomarker detection, related drug treatment (whether hormone is used, brain function is improved, brain pressure is reduced by dehydration, waking is promoted, etc.), and other treatments (hyperbaric oxygen treatment scheme, rehabilitation treatment).
In the step 1, if the acquired data of the patient in a certain mode is missing, performing missing marking;
the step 1 further comprises the steps of obtaining follow-up information of the patient suffering from carbon monoxide poisoning and constructing a prediction label; if the patient suffers from delayed encephalopathy within a period of time after carbon monoxide poisoning, the predictive label is 1, otherwise, the predictive label is 0.
Dividing the data set in the step 1 into a training set and a testing set, dividing the training set into a training set 1 and a training set 2, wherein the training set 1 is used for training a feature extraction model and a feature dependency graph; the training set 2 is used for training a carbon monoxide poisoning delayed encephalopathy prediction model, no data is missing in the training set, and the missing of brain CT, brain MRI or electroencephalogram data can exist in the test set.
Step 2: based on the data set obtained in the step 1, constructing a characteristic dependency graph through a Bayesian network structure learning algorithm and doctor suggestions, deleting nodes representing delayed encephalopathy labels in the characteristic dependency graph, and reserving nodes representing the characteristics; all features appearing in the feature dependency graph form a set F, and a corresponding adjacency matrix A is obtained based on the structure of the feature dependency graph;
referring to fig. 5, the feature dependency graph is a directed acyclic graph, wherein nodes represent features, edges represent dependency relationships between the nodes, and the nodes pointed by the directed edges depend on the departure nodes of the directed edges.
All features appearing in the feature dependency graph constitute a set F, for example, as shown in fig. 5, feature set f= { occupation, age, poisoning pattern, poisoning raw material, past history-cardiovascular disease, coma time, hyperbaric oxygen treatment }. From the structure of the feature dependency graph, a corresponding adjacency matrix a can be obtained.
Step 3: respectively establishing a feature extraction model based on the data types in the data set, and extracting multi-mode data feature vectors based on the feature extraction model;
referring to fig. 1, the feature extraction model described in step 3 includes: a feature extraction model M1, a feature extraction model M2, and a feature extraction model M3;
the feature extraction model M1 is composed of a convolution module of EEGNet, wherein the convolution module is composed of three convolution layers, namely Conv2D, depthwiseConv2D and SeparableConv2D in sequence;
the feature extraction model M2 and the feature extraction model M3 are respectively composed of a first convolutional neural network and a second convolutional neural network.
The feature extraction model M1, the feature extraction model M2 and the feature extraction model M3 are all obtained by training in a training set 1 divided from a data set; respectively training electroencephalogram, brain CT and brain MRI data in a data set to obtain three classifiers, adjusting parameters of a feature extraction model to enable classification performance of the corresponding classifier to meet preset requirements, removing a sigmoid classification layer in the classifier, extracting convolution structures of the classifier meeting preset conditions under three modes, and forming feature extraction models M1, M2 and M3; the feature extraction models M1, M2 and M3 are used to extract an electroencephalogram feature vector E2, a brain CT feature vector E3 and a brain MRI feature vector E4, respectively.
Optionally, the method can initialize the parameters of the feature extraction model by adopting a pre-training strategy, thereby accelerating the training speed and improving the precision of the feature extraction model.
Step 4: constructing a graph embedded feedforward neural network based on a characteristic dependency graph;
referring to fig. 2, the definition of the graph embedded feedforward neural network described in step 4 is as follows:
E1＝MLP(σ(x(W·A)+b))；
wherein: a is an adjacency matrix of the feature dependency graph; x is a feature vector formed by the values of the structural features of the patient, wherein the structural features of the patient comprise the features appearing in the feature set F in the step 2; w and b are training parameters; representing the Hadamard product operation; sigma represents an activation function; MLP is a multi-layer perceptron.
The graph embedded feedforward neural network part of the invention is embedded with the characteristic dependency graph in the step 2, so as to integrate the dependency relationship among the characteristics and further construct the characteristic vector of the structural characteristics.
Step 5: fusing the multi-mode data feature vectors to obtain fusion vectors C1 and C2;
referring to fig. 2, the step 5 includes the steps of:
step 5.1: taking the multi-mode data feature vector extracted in the step 3 as a training sample [ x, E2, E3, E4, y ]; the structural feature vector x is embedded into a neural network code through a graph to obtain a feature vector E1;
step 5.2: the feature vector E1 is converted into an electroencephalogram feature vector E2 based on the first layer encoder-decoder, and a fusion vector C for capturing joint information between the feature vector E1 and the electroencephalogram feature vector E2 is generated in the conversion process:
C＝f E-D (E1,E2)；
wherein: f (f) E-D () An encoder-decoder unit representing a first layer, obtaining a fusion vector representation C; wherein the encoder-decoder unit may be of neural network architecture;
step 5.3: the fused vector C is input to a coder-decoder unit of the second layer, and conversion of the fused vector C and brain CT and brain MRI eigenvectors E3 and E4 is realized. In the encoder-decoder unit of the second layer, the conversion of fusion vector C into brain CT and brain MRI uses one encoder and two separate decoders, which in the encoding and decoding process produce fusion vector representations C1 and C2, respectively:
C1＝f E1-D1 (C,E3)；
C2＝f E1-D2 (C,E4)；
wherein: f (f) E1-D1 ()、f E1-D2 () An encoder-decoder unit representing a second layer for obtaining fusion vector representations C1 and C2, respectively; wherein the encoder-decoder unit may be of neural network architecture.
The feature fusion of the invention designs a two-layer encoder-decoder unit, which respectively fuses the electroencephalogram features and the brain image features (including brain CT and brain MRI), so that the fusion features specific to the prediction task of the delayed encephalopathy caused by carbon monoxide poisoning can be fully mined, and the applicability and the stability of a prediction model of the delayed encephalopathy caused by carbon monoxide poisoning are enhanced.
Step 6: establishing a carbon monoxide poisoning delayed encephalopathy prediction model based on the graph embedded feedforward neural network and fusion vectors C1 and C2;
referring to fig. 2, after fusion vectors C1 and C2 are spliced, the fusion vectors are input into a full-connection layer, and after nonlinear fitting, the probability of delayed encephalopathy of a patient suffering from carbon monoxide poisoning is output:
wherein: i represents vector concatenation, W, b is a training parameter, g represents an activation function,
a predictive output indicative of whether there is a risk of tardive encephalopathy;
the loss function of the predictive model is defined as:
L＝λ 1 L E2 +λ 2 L E3 +λ 3 L E4 +λ p L p ；
wherein: l (L) Ei (i=2, 3, 4) represents an error generated in the feature fusion process, L p Representing errors resulting from classification predictions. Lambda (lambda) i (i=1, 2, 3) is a super parameter, which can be based onThe actual situation balances the duty ratio of the loss functions of different parts;
specifically, loss L of predicted portion p Is calculated by the following steps:
L p ＝-y log p-(1-y)log(1-p)；
wherein: y is the real label of a patient sample with carbon monoxide poisoning, delayed encephalopathy occurs, the label is 1, otherwise, the label is 0; p represents the probability that a patient suffering from carbon monoxide poisoning is predicted to develop a delayed encephalopathy;
loss of fusion moiety L Ei Is calculated by the following steps:
in the method, in the process of the invention,
the outputs of the decoders described in step 5.2 and step 5.3, respectively, ei (i=2, 3, 4) represent the vector representations of the electroencephalogram, brain CT and brain MRI described in step 3, respectively.
According to the carbon monoxide poisoning delayed encephalopathy prediction model, the parameters of the carbon monoxide poisoning delayed encephalopathy prediction model are updated through optimizing weighted sum of feature fusion loss and classification loss, so that the more accurate carbon monoxide poisoning delayed encephalopathy prediction model is obtained.
Step 7: and testing the carbon monoxide poisoning delayed encephalopathy prediction model, and selecting parameters meeting the prediction performance requirements as parameters of the carbon monoxide poisoning delayed encephalopathy prediction model.
And (3) inputting the structural feature vector x in the test set into a carbon monoxide poisoning delayed encephalopathy prediction model, and evaluating the prediction performance of the carbon monoxide poisoning delayed encephalopathy prediction model through accuracy, recall rate and F1 indexes based on the prediction result of whether the patient of the carbon monoxide poisoning delayed encephalopathy prediction model can generate delayed encephalopathy.
The prediction model of the carbon monoxide poisoning delayed encephalopathy fully fuses multi-mode diagnosis and treatment information during training, only structured data is needed to be used as model input during application and early prediction, so that the requirement of the prediction model on multi-mode data integrity in practical application is reduced to a certain extent, and the problem that accurate prediction cannot be realized under the condition of partial detection data loss is solved.
The foregoing examples merely represent specific embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that, for those skilled in the art, several variations and modifications can be made without departing from the technical solution of the present application, which fall within the protection scope of the present application.
Claims (6)
1. A method for predicting carbon monoxide poisoning delayed encephalopathy based on multi-modal learning is characterized by comprising the following steps:
step 1: acquiring multi-mode data of a patient suffering from carbon monoxide poisoning, and preprocessing the multi-mode data to obtain a data set of the patient suffering from carbon monoxide poisoning;
the multi-modal data includes an image, a waveform diagram, and structured data;
the images include brain CT and brain MRI; the waveform map comprises an electroencephalogram; the structured data comprises first diagnosis information, auxiliary examination and treatment conditions of CO poisoning;
step 2: based on the data set obtained in the step 1, constructing a feature dependency graph through a Bayesian network structure learning algorithm and doctor suggestions, wherein nodes represent features in the feature dependency graph, and edges represent the relationship among the features and represent the dependency relationship among the features; then deleting the nodes of the characteristic dependency graph, which represent the delayed encephalopathy labels of the carbon monoxide poisoning, and reserving the nodes representing the characteristics; all features appearing in the feature dependency graph form a set F, and a corresponding adjacency matrix A is obtained based on the structure of the feature dependency graph;
step 3: respectively establishing a feature extraction model based on the data types in the data set, and extracting multi-mode data feature vectors based on the feature extraction model;
step 4: constructing a graph embedded feedforward neural network based on a characteristic dependency graph, wherein the definition of the graph embedded feedforward neural network is as follows:
E1 = MLP(σ(x(W·A) + b)) ；
wherein: a is an adjacency matrix of the feature dependency graph; x is a feature vector formed by the values of the structural features of the patient, wherein the structural features of the patient comprise the features appearing in the feature set F in the step 2; w and b are training parameters; representing the Hadamard product operation; sigma represents an activation function; MLP is a multi-layer perceptron;
step 5: fusing the multi-mode data feature vectors to obtain fusion vectors C1 and C2, wherein the detailed steps comprise:
step 5.1: taking the multi-mode data feature vector extracted in the step 3 as a training sample [ x, E2, E3, E4, y ]; the structural feature vector x is embedded into a neural network code through a graph to obtain a feature vector E1;
step 5.2: the feature vector E1 is converted into an electroencephalogram feature vector E2 based on the first layer encoder-decoder, and a fusion vector C for capturing joint information between the feature vector E1 and the electroencephalogram feature vector E2 is generated in the conversion process:
C = fE-D (E1, E2) ；
wherein: fE-D () represents the encoder-decoder unit of the first layer, obtaining a fusion vector representation C; wherein the encoder-decoder unit may be of neural network architecture;
step 5.3: inputting the fusion vector C to a coder-decoder unit of a second layer to realize the conversion of the fusion vector C and brain CT and brain MRI eigenvectors E3 and E4; in the encoder-decoder unit of the second layer, the conversion of fusion vector C into brain CT and brain MRI uses one encoder and two separate decoders, which in the encoding and decoding process produce fusion vector representations C1 and C2, respectively:
C1 = fE1-D1 (C, E3)；
C2 = fE1-D2 (C, E4) ；
wherein: fE1-D1 (), fE1-D2 () represent encoder-decoder units of the second layer for obtaining fusion vector representations C1 and C2, respectively; wherein the encoder-decoder unit adopts a neural network structure;
step 6: based on the graph embedded feedforward neural network and fusion vectors C1 and C2, a carbon monoxide poisoning delayed encephalopathy prediction model is established, fusion vectors C1 and C2 are spliced, fusion vectors C1 and C2 are input into a full-connection layer after being spliced, and the probability of delayed encephalopathy occurrence of a carbon monoxide poisoning patient is output through nonlinear fitting:
yˆ = g(W *(C1 || C2) + b) ；
wherein: i represents vector stitching, W, b is a training parameter, g represents an activation function, y ˆ represents a predicted output of whether there is a risk of tardive encephalopathy;
the loss function of the predictive model of delayed encephalopathy in carbon monoxide poisoning is defined as:
L = λ1LE 2 + λ2 LE3 + λ3 LE 4 + λp Lp ；
wherein: LEi; i=2, 3, 4; representing errors generated in the feature fusion process, wherein Lp represents errors generated by classification prediction; λi; i=1, 2, 3; the duty ratio of the loss functions of different parts can be weighed according to actual conditions as super parameters;
specifically, the calculation method of the loss Lp of the predicted portion:
Lp = -y log p - (1- y) log(1- p) ；
wherein: y is the real label of a patient sample with carbon monoxide poisoning, delayed encephalopathy occurs, the label is 1, otherwise, the label is 0; p represents the probability that a patient suffering from carbon monoxide poisoning is predicted to develop a delayed encephalopathy;
calculation method of loss LEi of fusion part:
LEi =|| Ei - Eˆi || 2 , i = 2, 3, 4 ；
wherein E ˆ i, i=2, 3, 4; the outputs of the decoders described in step 5.2 and step 5.3, ei, respectively; i=2, 3, 4;
vector representations of the electroencephalogram, brain CT and brain MRI described in step 3 are represented respectively;
step 7: and testing the carbon monoxide poisoning delayed encephalopathy prediction model, and selecting parameters meeting the prediction performance requirements as parameters of the carbon monoxide poisoning delayed encephalopathy prediction model.
2. The method for predicting carbon monoxide poisoning delayed encephalopathy based on multimodal learning according to claim 1, which
Is characterized in that in the step 1, if the acquired patient has data missing in a certain mode, the missing mark is carried out;
the step 1 further comprises the steps of obtaining follow-up information of the patient suffering from carbon monoxide poisoning and constructing a prediction label; if the patient suffers from delayed encephalopathy within a period of time after carbon monoxide poisoning, the predictive label is 1, otherwise, the predictive label is 0.
3. The method for predicting carbon monoxide poisoning delayed encephalopathy according to claim 1, wherein the feature extraction model in step 3 comprises: a feature extraction model M1, a feature extraction model M2, and a feature extraction model M3;
the feature extraction model M1 is composed of a convolution module of EEGNet, wherein the convolution module is composed of three convolution layers, namely Conv2D, depthwiseConv2D and SeparableConv2D in sequence; the feature extraction model M2 and the feature extraction model M3 are respectively composed of a first convolutional neural network and a second convolutional neural network.
4. The method for predicting carbon monoxide poisoning delayed encephalopathy based on multi-modal learning according to claim 3, wherein the feature extraction model M1, the feature extraction model M2 and the feature extraction model M3 are all obtained by training from a training set 1 divided from a dataset; respectively utilizing electroencephalogram, brain CT and brain MRI data in a data set to train to obtain three classifiers, adjusting parameters of a feature extraction model to enable classification performance of the corresponding classifier to meet preset requirements, removing a sigmoid classification layer in the classifier, extracting convolution structures of the classifier meeting preset conditions under three modes, and forming feature extraction models M1, M2 and M3; the feature extraction models M1, M2 and M3 are used to extract an electroencephalogram feature vector E2, a brain CT feature vector E3 and a brain MRI feature vector E4, respectively.
5. The method for predicting carbon monoxide poisoning late-onset encephalopathy based on multi-modal learning according to claim 1, wherein the structured feature vector x in the test set is input into a model for predicting carbon monoxide poisoning late-onset encephalopathy, and the prediction result of whether a patient based on the model for predicting carbon monoxide poisoning late-onset encephalopathy will develop late-onset encephalopathy.
6. The method for predicting carbon monoxide poisoning delayed encephalopathy based on multimodal learning according to claim 1, which
The method is characterized in that the data set in the step 1 is divided into a training set and a testing set, the training set is divided into a training set 1 and a training set 2, and the training set 1 is used for training a feature extraction model and a feature dependency graph; the training set 2 is used for training a carbon monoxide poisoning delayed encephalopathy prediction model, no data is missing in the training set, and the missing of brain CT, brain MRI or electroencephalogram data can exist in the test set.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210348106.5A CN114795114B (en) | 2022-03-29 | 2022-03-29 | Carbon monoxide poisoning delayed encephalopathy prediction method based on multi-modal learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210348106.5A CN114795114B (en) | 2022-03-29 | 2022-03-29 | Carbon monoxide poisoning delayed encephalopathy prediction method based on multi-modal learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114795114A CN114795114A (en) | 2022-07-29 |
| CN114795114B true CN114795114B (en) | 2023-05-02 |
Family
ID=82533239
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210348106.5A Active CN114795114B (en) | 2022-03-29 | 2022-03-29 | Carbon monoxide poisoning delayed encephalopathy prediction method based on multi-modal learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114795114B (en) |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| SE534542C2 (en) * | 2009-09-30 | 2011-09-27 | Calmark Sweden Ab | Test system to determine hypoxia-triggered cell damage |
| EP2492690A1 (en) * | 2011-02-22 | 2012-08-29 | BIOCRATES Life Sciences AG | Method and use of metabolites for the diagnosis of inflammatory brain injury in preterm born infants |
| US11326182B2 (en) * | 2016-04-29 | 2022-05-10 | Voyager Therapeutics, Inc. | Compositions for the treatment of disease |
| US10423861B2 (en) * | 2017-10-16 | 2019-09-24 | Illumina, Inc. | Deep learning-based techniques for training deep convolutional neural networks |
| CN109859834A (en) * | 2018-12-28 | 2019-06-07 | 首都医科大学附属北京地坛医院 | A kind of processing method and processing device for predicting the death rate |
| CN110298479B (en) * | 2019-05-20 | 2021-09-03 | 北京航空航天大学 | Brain volume atrophy prediction method based on brain function network |
| CN112001887B (en) * | 2020-07-20 | 2021-11-09 | 南通大学 | Full convolution genetic neural network method for infant brain medical record image segmentation |
| CN113080929A (en) * | 2021-04-14 | 2021-07-09 | 电子科技大学 | anti-NMDAR encephalitis image feature classification method based on machine learning |
| CN113367706B (en) * | 2021-06-08 | 2022-12-06 | 北京大学第一医院 | Multi-mode detection system for newborn brain function |
| CN113707319A (en) * | 2021-08-30 | 2021-11-26 | 西安交通大学医学院第一附属医院 | Construction method of carbon monoxide poisoning delayed encephalopathy prediction model |
-
2022
- 2022-03-29 CN CN202210348106.5A patent/CN114795114B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN114795114A (en) | 2022-07-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112508077B (en) | Social media emotion analysis method and system based on multi-modal feature fusion | |
| CN116682553B (en) | Diagnosis recommendation system integrating knowledge and patient representation | |
| CN110335653A (en) | Non-standard case history analytic method based on openEHR case history format | |
| Jo et al. | Combining LSTM and latent topic modeling for mortality prediction | |
| WO2022227280A1 (en) | Smart glasses-based disaster rescue triage and auxiliary diagnosis method | |
| CN114564959B (en) | Chinese clinical phenotype fine granularity named entity identification method and system | |
| CN111540470B (en) | Social network depression tendency detection model based on BERT transfer learning and training method thereof | |
| CN117033568A (en) | Medical data index interpretation method, device, storage medium and equipment | |
| Escolà-Gascón | Researching unexplained phenomena: empirical-statistical validity and reliability of the Multivariable Multiaxial Suggestibility Inventory-2 (MMSI-2) | |
| CN113342973A (en) | Diagnosis method of auxiliary diagnosis model based on disease two-classifier | |
| CN113139974A (en) | Focus segmentation model training and application method based on semi-supervised learning | |
| CN109119155B (en) | ICU death risk assessment system based on deep learning | |
| CN114795114B (en) | Carbon monoxide poisoning delayed encephalopathy prediction method based on multi-modal learning | |
| CN114550896A (en) | Dizziness patient emergency pre-examination triage decision-making method, device and model based on artificial neural network | |
| CN117151215B (en) | Coronary heart disease multi-mode data characteristic extraction method based on knowledge graph | |
| CN107085655B (en) | Traditional Chinese medicine data processing method and system based on attribute constraint concept lattice | |
| CN117497140A (en) | Multi-level depression state detection method based on fine granularity prompt learning | |
| Kumar | SCL-SSC: Supervised Contrastive Learning for Sleep Stage Classification | |
| Nissimagoudar et al. | AlertNet: Deep convolutional-recurrent neural network model for driving alertness detection | |
| Ren et al. | A Contrastive Predictive Coding‐Based Classification Framework for Healthcare Sensor Data | |
| CN115089112B (en) | Post-stroke cognitive impairment risk assessment model building method and device and electronic equipment | |
| CN111611893B (en) | Intelligent measuring and judging method applying neural network deep learning | |
| Long et al. | DiseaseNet: A novel disease diagnosis deep framework via fusing medical record summarization | |
| CN112749564A (en) | Medical record event element extraction method and device, electronic equipment and storage medium | |
| Sun et al. | START: Automatic Sleep Staging with Attention-based Cross-modal Learning Transformer |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |