CN114795114B - Carbon monoxide poisoning delayed encephalopathy prediction method based on multi-modal learning - Google Patents

Abstract

The invention belongs to the technical field of medical information, and particularly relates to a carbon monoxide poisoning delayed encephalopathy prediction method based on multi-modal learning, which constructs a characteristic dependency graph by combining a Bayesian network structure learning algorithm and doctor suggestions; based on the characteristic dependency graph, embedding the building graph into a feedforward neural network, and integrating the dependency relationship among the characteristics into a neural network structure to obtain a vector representation of the structural characteristics; and designing a feature fusion structure, obtaining vector representation fused with multi-mode information, inputting the vector into a nonlinear neural network layer, and finally obtaining a prediction result of whether a patient with carbon monoxide poisoning will have delayed encephalopathy. The invention fully fuses the multi-mode diagnosis and treatment information, only needs structured data as model input when being applied to early prediction, thus reducing the requirement of a prediction model on the integrity of the multi-mode data in practical application to a certain extent and solving the problem that accurate prediction cannot be realized under the condition of data loss.

Description

A prediction method for delayed encephalopathy in carbon monoxide poisoning based on multimodal learning

technical field

The invention belongs to the technical field of medical information, and in particular relates to a method for predicting delayed encephalopathy in carbon monoxide poisoning based on multimodal learning.

Background technique

Carbon monoxide poisoning is a common accident in daily life. After the symptoms of carbon monoxide poisoning are relieved, after a 2-60 day false recovery period, there will be a 10%-30% chance of developing mental system diseases mainly dementia, mental symptoms and extrapyramidal symptoms, that is, delayed onset encephalopathy. The awareness rate of delayed encephalopathy is low, the condition is severe and the course of the disease is long. Once delayed encephalopathy occurs, the patient will lose the ability to work and needs other people's care in daily life. Most of the treatment time is as long as half a year or 1 year or even longer. Cognitive dysfunction and other permanent neurological disabilities, which affect the quality of life, increase the economic burden, and even "disease impoverishment". Therefore, realizing early prediction of the risk of CO poisoning delayed encephalopathy will facilitate the timely intervention of the disease, reduce its incidence frequency, and effectively reduce the burden of the disease on the family and society.

At present, the existing delayed encephalopathy prediction mainly adopts the traditional medical statistical method to fit historical data, and finds the risk factors of patients with delayed encephalopathy in the early stage of carbon monoxide poisoning, but this method is single and lacks accuracy.

At present, clinical examination indicators, electroencephalogram (EEG), brain CT and brain MRI are effective means to find abnormal signs in patients with carbon monoxide poisoning. Making full use of multimodal data (including structured data such as clinical indicators, EEG, brain CT and brain MRI) can effectively improve the accuracy of risk prediction for delayed encephalopathy. However, in practical applications, due to the lack of some examination data of patients in the early stage of carbon monoxide poisoning, this will affect the accuracy of the risk prediction of delayed encephalopathy.

Contents of the invention

In order to solve the technical problems existing in the above-mentioned prior art, the present invention provides a method for predicting delayed encephalopathy in carbon monoxide poisoning based on multimodal learning, which intends to solve the problem mentioned in the background technology that the loss of inspection data affects delayed encephalopathy Technical issues of the accuracy of risk predictions.

In order to solve the problems of the technologies described above, the technical scheme adopted in the present invention is as follows:

A method for predicting delayed encephalopathy in carbon monoxide poisoning based on multimodal learning, comprising the following steps:

Step 1: Obtain multimodal data of patients with carbon monoxide poisoning, and preprocess the multimodal data to obtain a data set of patients with carbon monoxide poisoning;

Step 2: Based on the data set obtained in step 1, construct a feature dependency graph through the Bayesian network structure learning algorithm and doctor's advice. In the feature dependency graph, nodes represent features, and edges represent the relationship between features, which is specifically reflected in the relationship between features. Dependency relationship; then delete the node representing the label of carbon monoxide poisoning delayed encephalopathy in the feature dependency graph, and retain the node representing the feature; form a set F of all the features appearing in the feature dependency graph, and based on the structure of the feature dependency graph, get the corresponding adjacency matrix A;

Step 3: Based on the data types in the data set, respectively establish feature extraction models, and extract multimodal data feature vectors based on the feature extraction models;

Step 4: Construct a graph embedding feed-forward neural network based on feature-dependent graphs;

Step 5: Fusion the multimodal data feature vectors to obtain fusion vectors C1 and C2;

Step 6: Based on the graph embedding feed-forward neural network and the fusion vectors C1 and C2, a prediction model for CO poisoning delayed encephalopathy is established;

Step 7: Test the CO poisoning delayed encephalopathy prediction model, and select parameters that meet the prediction performance requirements as parameters of the CO poisoning delayed encephalopathy prediction model.

The carbon monoxide poisoning delayed encephalopathy prediction model of the present invention fully integrates multimodal diagnosis and treatment information during training, and only needs structured data as model input during application and early prediction, thus reducing practical application to a certain extent The demand for multi-modal data integrity in the forecasting model solves the problem that accurate forecasting cannot be achieved when some inspection data is missing.

Preferably, the multimodal data includes images, waveform diagrams and structured data;

The image data includes brain CT and brain MRI; the waveform image data includes EEG; and the structured data CO poisoning first visit information, auxiliary examination and treatment conditions.

Preferably, in step 1, if there is data missing in a certain modality in the acquired patient, mark the missing data;

The step 1 also includes obtaining follow-up information of patients with carbon monoxide poisoning, and constructing a prediction label; if the patient develops delayed encephalopathy within a period of time after carbon monoxide poisoning, the prediction label is 1, otherwise the prediction label is 0.

Preferably, the feature extraction model described in step 3 includes: feature extraction model M1, feature extraction model M2 and feature extraction model M3;

The feature extraction model M1 is composed of a convolution module of EEGNet, and the convolution module is composed of three convolution layers, followed by Conv2D, DepthwiseConv2D and SeparableConv2D;

The feature extraction model M2 and the feature extraction model M3 are respectively composed of a first convolutional neural network and a second convolutional neural network.

Preferably, the feature extraction model M1, the feature extraction model M2 and the feature extraction model M3 are all trained from the training set 1 divided in the data set; respectively use the EEG, brain CT and brain MRI data training in the data set to obtain three classifier, adjust the parameters of the feature extraction model so that the classification performance of the corresponding classifier meets the predetermined requirements, then remove the sigmoid classification layer in the classifier, and extract the convolution structure of the classifier that meets the predetermined conditions under the three modes, Feature extraction models M1, M2 and M3 are formed; the feature extraction models M1, M2 and M3 are respectively used to extract EEG feature vector E2, brain CT feature vector E3 and brain MRI feature vector E4.

Preferably, the graph embedding feedforward neural network described in step 4 is defined as follows:

E1=MLP(σ(x(W·A)+b));

In the formula: A is the adjacency matrix of the feature dependency graph; x is the feature vector composed of the values of the patient's structural features, and the patient's structural features include the features that appear in the feature set F described in step 2; W and b are training parameters; · means Hadamard product operation; σ means activation function; MLP means multi-layer perceptron.

The graph embedding feedforward neural network of the present invention partially embeds the feature dependency graph described in step 2 to integrate the dependencies between features, and further constructs the feature vector of structured features.

Preferably, said step 5 includes the following steps:

Step 5.1: Use the multimodal data feature vector extracted in step 3 as the training sample [x, E2, E3, E4, y]; the structured feature vector x is encoded by the graph embedding neural network to obtain the feature vector E1;

Step 5.2: Convert the feature vector E1 to the EEG feature vector E2 based on the first layer of encoder-decoder, and generate the joint information for capturing the joint information between the feature vector E1 and the EEG feature vector E2 during the conversion process Fusion vector C:

C = f _ED (E1, E2);

In the formula: f _ED () represents the encoder-decoder unit of the first layer, and obtains the fusion vector representation C; wherein, the encoder-decoder unit can choose a neural network structure;

Step 5.3: The fused vector C is input to the encoder-decoder unit of the second layer to realize the conversion of the fused vector C and brain CT and brain MRI feature vectors E3 and E4. In the encoder-decoder unit of the second layer, the conversion of the fusion vector C to brain CT and brain MRI uses an encoder and two separate decoders. During the encoding and decoding process, the fusion vector representations C1 and C2 are generated respectively:

C1=f _E1-D1 (C, E3);

C2=f _E1-D2 (C, E4);

In the formula: f _E1-D1 (), f _E1-D2 () represent the encoder-decoder unit of the second layer, which are used to obtain fusion vector representations C1 and C2 respectively; among them, the encoder-decoder unit can choose neural network structure.

The feature fusion of the present invention designs a two-layer encoder-decoder unit, which fuses EEG features and brain image features (including brain CT and brain MRI) respectively, so that it can fully mine the delayed encephalopathy specific to carbon monoxide poisoning Fusion features of a prediction task to enhance the applicability and stability of a predictive model for delayed-onset encephalopathy in carbon monoxide poisoning.

Preferably, the step 6 is specifically as follows:

Splicing the fusion vectors C1 and C2, after splicing the fusion vectors C1 and C2, input to the fully connected layer, after nonlinear fitting, output the probability of delayed encephalopathy in patients with carbon monoxide poisoning:

表示是否具有迟发性脑病风险的预测输出；In the formula: || means vector splicing, W and b are training parameters, g means activation function,

Indicates the predicted output of whether there is a risk of delayed encephalopathy;

The loss function of the CO poisoning delayed encephalopathy prediction model is defined as:

L＝λ ₁ L _E2 +λ ₂ L _E3 +λ ₃ L _E4 +λ _p L _p ;

In the formula: L _Ei (i=2,3,4) represents the error generated during the feature fusion process, and L _p represents the error generated by classification prediction. λ _i (i=1,2,3) is a hyperparameter, which can weigh the proportion of different parts of the loss function according to the actual situation;

Specifically, the calculation method of the loss L _p of the prediction part:

L _p =-y log p-(1-y)log(1-p);

In the formula: y is the real label of the sample of the patient with carbon monoxide poisoning, if delayed encephalopathy occurs, the label is 1, otherwise the label is 0; p represents the probability that the patient with carbon monoxide poisoning is predicted to develop delayed encephalopathy;

The calculation method of the loss L _Ei of the fusion part:

分别表示步骤5.2和步骤5.3中所述的解码器的输出，Ei(i＝2,3,4)分别表示步骤3所述的脑电图、脑CT和脑MRI的向量表示。In the formula,

represent the outputs of the decoders described in step 5.2 and step 5.3, respectively, and Ei (i=2, 3, 4) represent the vector representations of the EEG, brain CT and brain MRI described in step 3, respectively.

Preferably, the structured feature vector x in the test set is input into the carbon monoxide poisoning delayed encephalopathy prediction model, whether the patients based on the carbon monoxide poisoning delayed encephalopathy prediction model will have delayed encephalopathy prediction results, through accuracy, Recall and F1 metrics to assess the predictive performance of a carbon monoxide poisoning delayed encephalopathy prediction model.

Preferably, the data set described in step 1 is divided into a training set and a test set, and the training set is divided into a training set 1 and a training set 2, and the training set 1 is used to train a feature extraction model and a feature dependency graph; training Set 2 is used to train a carbon monoxide poisoning delayed encephalopathy prediction model. There is no missing data in the training set, and there may be missing data such as brain CT, brain MRI, or EEG in the test set.

The beneficial effects of the present invention include:

1. The feature dependency graph can mine the dependencies between features and use it as prior knowledge to enhance the prediction accuracy of the CO poisoning delayed encephalopathy prediction model. The graph embedding feedforward neural network selects features based on the structure of the feature-dependent graph, so that the connection relationship between the input layer and the hidden layer of the feedforward neural network becomes sparse.

2. The multimodal feature fusion of the present invention designs a two-layer encoder-decoder unit to mine fusion features specific to the CO poisoning delayed encephalopathy prediction task. Compared with the simple combination of different modal features through the attention mechanism and other methods, the carbon monoxide poisoning delayed encephalopathy prediction model in the present invention is designed with a special multi-modal feature learning process, and part of the loss caused by modal conversion is involved in The optimization of the model can make more effective use of multimodal information.

3. Construct a robust and accurate CO poisoning delayed encephalopathy prediction model through the integration of feature-dependency graphs and the fusion of multimodal information. The carbon monoxide poisoning delayed encephalopathy prediction model of the present invention fully integrates multi-modal diagnosis and treatment information during training; when applied to early prediction, only structured data is needed as the carbon monoxide poisoning delayed encephalopathy prediction model enter. Therefore, to a certain extent, the demand for multimodal data integrity of the CO poisoning delayed encephalopathy prediction model in practical applications has been reduced, and the problem of inability to achieve accurate and reliable predictions in the absence of some inspection data has been solved.

Description of drawings

FIG. 1 is a schematic representation of different modal eigenvectors of the present invention.

Fig. 2 is a schematic diagram of training the carbon monoxide poisoning delayed encephalopathy prediction model of the present invention.

Fig. 3 is a schematic diagram of the training and testing process of the carbon monoxide poisoning delayed encephalopathy prediction model of the present invention.

Fig. 4 is a schematic diagram of the overall process of the present invention.

FIG. 5 is a schematic diagram of a feature dependency graph of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only It is a part of the embodiments of this application, not all of them. The components of the embodiments of the application generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative efforts belong to the scope of protection of the present application.

Below in conjunction with accompanying drawing 1 and accompanying drawing 5 the present invention is described in further detail:

A method for predicting delayed encephalopathy in carbon monoxide poisoning based on multimodal learning, comprising the following steps:

Step 1: Obtain multimodal data of patients with carbon monoxide poisoning, and preprocess the multimodal data to obtain a data set of patients with carbon monoxide poisoning;

The multimodal data includes images, waveforms, and structured data;

The image data includes brain CT and brain MRI; the waveform image data includes EEG; and the structured data CO poisoning first visit information, auxiliary examination and treatment conditions.

Specifically, the CO poisoning first visit information includes: age, gender, occupation, poisoning method, poisoning raw material, poisoning place, type of work before poisoning, education level, past medical history, time of poisoning, time of visit, time of first hyperbaric oxygen treatment, Clinical manifestations at the time of poisoning (headache, vomiting, weakness of limbs, coma, palpitations, dyspnea, etc.), coma time, Glasgow score at the time of coma, state of consciousness at the first hyperbaric oxygenation; the auxiliary examination and treatment information includes: blood carbon dioxide Oxyhemoglobin, blood gas, blood routine, C-reactive protein, D-dimer, liver function, blood lipid, blood sugar and some specific biomarkers, related drug treatment (whether to use hormones, improve brain function, reduce intracranial pressure by dehydration, promote awakening, etc.) and other treatments (hyperbaric oxygen therapy program, rehabilitation).

In step 1, if there is missing data in a certain modality in the acquired patient, mark the missing data;

The step 1 also includes obtaining follow-up information of patients with carbon monoxide poisoning, and constructing a prediction label; if the patient develops delayed encephalopathy within a period of time after carbon monoxide poisoning, the prediction label is 1, otherwise the prediction label is 0.

The data set described in step 1 is divided into training set and test set, and described training set is divided into training set 1 and training set 2, and training set 1 is used for training feature extraction model and feature dependency graph; Training set 2 uses For training a carbon monoxide poisoning delayed encephalopathy prediction model, there is no missing data in the training set, and there may be missing brain CT, brain MRI or EEG data in the testing set.

Step 2: Based on the data set obtained in step 1, construct a feature dependency graph through the Bayesian network structure learning algorithm and doctor's advice, and delete the nodes representing the label of carbon monoxide poisoning delayed encephalopathy in the feature dependency graph, and keep the nodes representing the features ; Construct a set F of all the features appearing in the feature dependency graph, and obtain the corresponding adjacency matrix A based on the structure of the feature dependency graph;

Referring to Figure 5, the feature dependency graph is a directed acyclic graph, where nodes represent features, edges represent dependencies between nodes, and the nodes pointed by the directed edges depend on the starting nodes of the directed edges.

All the features appearing in the feature-dependent graph form a set F, for example, as shown in Figure 5, the feature set F of the feature-dependent graph = {occupation, age, poisoning method, poisoning raw material, past medical history-cardiovascular disease, coma time, hyperbaric oxygen treat}. According to the structure of the feature dependency graph, the corresponding adjacency matrix A can be obtained.

Step 3: Based on the data types in the data set, respectively establish feature extraction models, and extract multimodal data feature vectors based on the feature extraction models;

Referring to shown in accompanying drawing 1, the feature extraction model described in step 3 comprises: feature extraction model M1, feature extraction model M2 and feature extraction model M3;

The feature extraction model M1 is composed of a convolution module of EEGNet, and the convolution module is composed of three convolution layers, followed by Conv2D, DepthwiseConv2D and SeparableConv2D;

The feature extraction model M2 and the feature extraction model M3 are respectively composed of a first convolutional neural network and a second convolutional neural network.

The feature extraction model M1, the feature extraction model M2 and the feature extraction model M3 are all trained from the training set 1 divided in the data set; three classifiers are obtained by training the EEG, brain CT and brain MRI data in the data set respectively , adjust the parameters of the feature extraction model so that the classification performance of the corresponding classifier meets the predetermined requirements, then remove the sigmoid classification layer in the classifier, and extract the convolution structure of the classifier that meets the predetermined conditions under the three modes to form a feature extraction Models M1, M2, and M3; the feature extraction models M1, M2, and M3 are used to extract EEG feature vector E2, brain CT feature vector E3, and brain MRI feature vector E4, respectively.

Optionally, the present invention may first use a pre-training strategy to initialize the parameters of the feature extraction model, thereby speeding up the training speed and improving the accuracy of the feature extraction model.

Step 4: Construct a graph embedding feed-forward neural network based on feature-dependent graphs;

Referring to accompanying drawing 2, the graph embedding feed-forward neural network described in step 4 is defined as follows:

E1=MLP(σ(x(W·A)+b));

In the formula: A is the adjacency matrix of the feature dependency graph; x is the feature vector composed of the values of the patient's structural features, and the patient's structural features include the features that appear in the feature set F described in step 2; W and b are training parameters; · means Hadamard product operation; σ means activation function; MLP means multi-layer perceptron.

The graph embedding feedforward neural network of the present invention partially embeds the feature dependency graph described in step 2 to integrate the dependencies between features, and further constructs the feature vector of structured features.

Step 5: Fusion the multimodal data feature vectors to obtain fusion vectors C1 and C2;

Referring to accompanying drawing 2, described step 5 comprises the following steps:

Step 5.1: Use the multimodal data feature vector extracted in step 3 as the training sample [x, E2, E3, E4, y]; the structured feature vector x is encoded by the graph embedding neural network to obtain the feature vector E1;

Step 5.2: Convert the feature vector E1 to the EEG feature vector E2 based on the first layer of encoder-decoder, and generate the joint information for capturing the joint information between the feature vector E1 and the EEG feature vector E2 during the conversion process Fusion vector C:

C = f _ED (E1, E2);

In the formula: f _ED () represents the encoder-decoder unit of the first layer, and obtains the fusion vector representation C; wherein, the encoder-decoder unit can choose a neural network structure;

Step 5.3: The fused vector C is input to the encoder-decoder unit of the second layer to realize the conversion of the fused vector C and brain CT and brain MRI feature vectors E3 and E4. In the encoder-decoder unit of the second layer, the conversion of the fusion vector C to brain CT and brain MRI uses an encoder and two separate decoders. During the encoding and decoding process, the fusion vector representations C1 and C2 are generated respectively:

C1=f _E1-D1 (C, E3);

C2=f _E1-D2 (C, E4);

In the formula: f _E1-D1 (), f _E1-D2 () represent the encoder-decoder unit of the second layer, which are used to obtain fusion vector representations C1 and C2 respectively; among them, the encoder-decoder unit can choose neural network structure.

The feature fusion of the present invention designs a two-layer encoder-decoder unit, which fuses EEG features and brain image features (including brain CT and brain MRI) respectively, so that it can fully mine the delayed encephalopathy specific to carbon monoxide poisoning Fusion features of prediction tasks to enhance the applicability and stability of a carbon monoxide poisoning delayed encephalopathy prediction model.

Step 6: Based on the graph embedding feed-forward neural network and the fusion vectors C1 and C2, a prediction model for CO poisoning delayed encephalopathy is established;

See Figure 2, after the fusion vectors C1 and C2 are spliced, they are input to the fully connected layer, and after nonlinear fitting, the probability of delayed encephalopathy in patients with carbon monoxide poisoning is output:

表示是否具有迟发性脑病风险的预测输出；In the formula: || means vector splicing, W and b are training parameters, g means activation function,

Indicates the predicted output of whether there is a risk of delayed encephalopathy;

The loss function of the predictive model is defined as:

L＝λ ₁ L _E2 +λ ₂ L _E3 +λ ₃ L _E4 +λ _p L _p ;

In the formula: L _Ei (i=2,3,4) represents the error generated during the feature fusion process, and L _p represents the error generated by classification prediction. λ _i (i=1,2,3) is a hyperparameter, which can weigh the proportion of different parts of the loss function according to the actual situation;

Specifically, the calculation method of the loss L _p of the prediction part:

L _p =-y log p-(1-y)log(1-p);

In the formula: y is the real label of the sample of the patient with carbon monoxide poisoning, if delayed encephalopathy occurs, the label is 1, otherwise the label is 0; p represents the probability that the patient with carbon monoxide poisoning is predicted to develop delayed encephalopathy;

The calculation method of the loss L _Ei of the fusion part:

分别表示步骤5.2和步骤5.3中所述的解码器的输出，Ei(i＝2,3,4)分别表示步骤3所述的脑电图、脑CT和脑MRI的向量表示。In the formula,

represent the outputs of the decoders described in step 5.2 and step 5.3, respectively, and Ei (i=2, 3, 4) represent the vector representations of the EEG, brain CT and brain MRI described in step 3, respectively.

The carbon monoxide poisoning delayed encephalopathy prediction model of the present invention updates the parameters of the carbon monoxide poisoning delayed encephalopathy prediction model by optimizing the weighted sum of feature fusion loss and classification loss, thereby obtaining more accurate carbon monoxide poisoning delayed encephalopathy prediction Model.

Step 7: Test the CO poisoning delayed encephalopathy prediction model, and select parameters that meet the prediction performance requirements as parameters of the CO poisoning delayed encephalopathy prediction model.

Input the structured feature vector x in the test set into the carbon monoxide poisoning delayed encephalopathy prediction model. Based on the prediction results of whether the patients with carbon monoxide poisoning delayed encephalopathy prediction model will develop delayed encephalopathy, the accuracy, recall rate and The F1 index evaluates the predictive performance of the CO poisoning delayed encephalopathy prediction model.

The carbon monoxide poisoning delayed encephalopathy prediction model of the present invention fully integrates multimodal diagnosis and treatment information during training, and only needs structured data as model input during application and early prediction, thus reducing practical application to a certain extent The demand for multi-modal data integrity in the forecasting model solves the problem that accurate forecasting cannot be achieved when some inspection data is missing.

The above-mentioned embodiments only express the specific implementation manners of the present application, and the descriptions thereof are relatively specific and detailed, but should not be construed as limiting the protection scope of the present application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the technical solution of the present application, and these all belong to the protection scope of the present application.

Claims (6)

1. A method for predicting carbon monoxide poisoning delayed encephalopathy based on multi-modal learning is characterized by comprising the following steps:

step 1: acquiring multi-mode data of a patient suffering from carbon monoxide poisoning, and preprocessing the multi-mode data to obtain a data set of the patient suffering from carbon monoxide poisoning;

the multi-modal data includes an image, a waveform diagram, and structured data;

the images include brain CT and brain MRI; the waveform map comprises an electroencephalogram; the structured data comprises first diagnosis information, auxiliary examination and treatment conditions of CO poisoning;

step 2: based on the data set obtained in the step 1, constructing a feature dependency graph through a Bayesian network structure learning algorithm and doctor suggestions, wherein nodes represent features in the feature dependency graph, and edges represent the relationship among the features and represent the dependency relationship among the features; then deleting the nodes of the characteristic dependency graph, which represent the delayed encephalopathy labels of the carbon monoxide poisoning, and reserving the nodes representing the characteristics; all features appearing in the feature dependency graph form a set F, and a corresponding adjacency matrix A is obtained based on the structure of the feature dependency graph;

step 3: respectively establishing a feature extraction model based on the data types in the data set, and extracting multi-mode data feature vectors based on the feature extraction model;

step 4: constructing a graph embedded feedforward neural network based on a characteristic dependency graph, wherein the definition of the graph embedded feedforward neural network is as follows:

E1 = MLP(σ(x(W·A) + b)) ；

wherein: a is an adjacency matrix of the feature dependency graph; x is a feature vector formed by the values of the structural features of the patient, wherein the structural features of the patient comprise the features appearing in the feature set F in the step 2; w and b are training parameters; representing the Hadamard product operation; sigma represents an activation function; MLP is a multi-layer perceptron;

step 5: fusing the multi-mode data feature vectors to obtain fusion vectors C1 and C2, wherein the detailed steps comprise:

step 5.1: taking the multi-mode data feature vector extracted in the step 3 as a training sample [ x, E2, E3, E4, y ]; the structural feature vector x is embedded into a neural network code through a graph to obtain a feature vector E1;

step 5.2: the feature vector E1 is converted into an electroencephalogram feature vector E2 based on the first layer encoder-decoder, and a fusion vector C for capturing joint information between the feature vector E1 and the electroencephalogram feature vector E2 is generated in the conversion process:

C = fE-D (E1, E2) ；

wherein: fE-D () represents the encoder-decoder unit of the first layer, obtaining a fusion vector representation C; wherein the encoder-decoder unit may be of neural network architecture;

step 5.3: inputting the fusion vector C to a coder-decoder unit of a second layer to realize the conversion of the fusion vector C and brain CT and brain MRI eigenvectors E3 and E4; in the encoder-decoder unit of the second layer, the conversion of fusion vector C into brain CT and brain MRI uses one encoder and two separate decoders, which in the encoding and decoding process produce fusion vector representations C1 and C2, respectively:

C1 = fE1-D1 (C, E3)；

C2 = fE1-D2 (C, E4) ；

wherein: fE1-D1 (), fE1-D2 () represent encoder-decoder units of the second layer for obtaining fusion vector representations C1 and C2, respectively; wherein the encoder-decoder unit adopts a neural network structure;

step 6: based on the graph embedded feedforward neural network and fusion vectors C1 and C2, a carbon monoxide poisoning delayed encephalopathy prediction model is established, fusion vectors C1 and C2 are spliced, fusion vectors C1 and C2 are input into a full-connection layer after being spliced, and the probability of delayed encephalopathy occurrence of a carbon monoxide poisoning patient is output through nonlinear fitting:

yˆ = g(W *(C1 || C2) + b) ；

wherein: i represents vector stitching, W, b is a training parameter, g represents an activation function, y ˆ represents a predicted output of whether there is a risk of tardive encephalopathy;

the loss function of the predictive model of delayed encephalopathy in carbon monoxide poisoning is defined as:

L = λ1LE 2 + λ2 LE3 + λ3 LE 4 + λp Lp ；

wherein: LEi; i=2, 3, 4; representing errors generated in the feature fusion process, wherein Lp represents errors generated by classification prediction; λi; i=1, 2, 3; the duty ratio of the loss functions of different parts can be weighed according to actual conditions as super parameters;

specifically, the calculation method of the loss Lp of the predicted portion:

Lp = -y log p - (1- y) log(1- p) ；

wherein: y is the real label of a patient sample with carbon monoxide poisoning, delayed encephalopathy occurs, the label is 1, otherwise, the label is 0; p represents the probability that a patient suffering from carbon monoxide poisoning is predicted to develop a delayed encephalopathy;

calculation method of loss LEi of fusion part:

LEi =|| Ei - Eˆi || ² , i = 2, 3, 4 ；

wherein E ˆ i, i=2, 3, 4; the outputs of the decoders described in step 5.2 and step 5.3, ei, respectively; i=2, 3, 4;

vector representations of the electroencephalogram, brain CT and brain MRI described in step 3 are represented respectively;

step 7: and testing the carbon monoxide poisoning delayed encephalopathy prediction model, and selecting parameters meeting the prediction performance requirements as parameters of the carbon monoxide poisoning delayed encephalopathy prediction model.

2. The method for predicting carbon monoxide poisoning delayed encephalopathy based on multimodal learning according to claim 1, which

Is characterized in that in the step 1, if the acquired patient has data missing in a certain mode, the missing mark is carried out;

the step 1 further comprises the steps of obtaining follow-up information of the patient suffering from carbon monoxide poisoning and constructing a prediction label; if the patient suffers from delayed encephalopathy within a period of time after carbon monoxide poisoning, the predictive label is 1, otherwise, the predictive label is 0.

3. The method for predicting carbon monoxide poisoning delayed encephalopathy according to claim 1, wherein the feature extraction model in step 3 comprises: a feature extraction model M1, a feature extraction model M2, and a feature extraction model M3;

the feature extraction model M1 is composed of a convolution module of EEGNet, wherein the convolution module is composed of three convolution layers, namely Conv2D, depthwiseConv2D and SeparableConv2D in sequence; the feature extraction model M2 and the feature extraction model M3 are respectively composed of a first convolutional neural network and a second convolutional neural network.

4. The method for predicting carbon monoxide poisoning delayed encephalopathy based on multi-modal learning according to claim 3, wherein the feature extraction model M1, the feature extraction model M2 and the feature extraction model M3 are all obtained by training from a training set 1 divided from a dataset; respectively utilizing electroencephalogram, brain CT and brain MRI data in a data set to train to obtain three classifiers, adjusting parameters of a feature extraction model to enable classification performance of the corresponding classifier to meet preset requirements, removing a sigmoid classification layer in the classifier, extracting convolution structures of the classifier meeting preset conditions under three modes, and forming feature extraction models M1, M2 and M3; the feature extraction models M1, M2 and M3 are used to extract an electroencephalogram feature vector E2, a brain CT feature vector E3 and a brain MRI feature vector E4, respectively.

5. The method for predicting carbon monoxide poisoning late-onset encephalopathy based on multi-modal learning according to claim 1, wherein the structured feature vector x in the test set is input into a model for predicting carbon monoxide poisoning late-onset encephalopathy, and the prediction result of whether a patient based on the model for predicting carbon monoxide poisoning late-onset encephalopathy will develop late-onset encephalopathy.

6. The method for predicting carbon monoxide poisoning delayed encephalopathy based on multimodal learning according to claim 1, which

The method is characterized in that the data set in the step 1 is divided into a training set and a testing set, the training set is divided into a training set 1 and a training set 2, and the training set 1 is used for training a feature extraction model and a feature dependency graph; the training set 2 is used for training a carbon monoxide poisoning delayed encephalopathy prediction model, no data is missing in the training set, and the missing of brain CT, brain MRI or electroencephalogram data can exist in the test set.

Images