CN117137488A - An auxiliary identification method for depression symptoms based on EEG data and facial expression images - Google Patents

Abstract

The invention discloses an auxiliary identification method for depression symptoms based on brain electrical data and facial expression images, which is mainly designed in such a way that the degree of conflict treatment disorder is evaluated by organically combining brain electrical physiological indexes and facial image indexes around the conflict treatment disorder and core symptoms of negative bias of depression groups, so that objective and quantitative indexes are extracted for the identification and the pre-estimation of depression. Specifically, an electroencephalogram stimulation experiment is executed based on a preset experimental paradigm, facial expression image data and individual electroencephalogram data to be tested are synchronously collected, after N270 waveforms are obtained through analysis, multi-feature extraction is carried out on the facial expression image data and the N270 waveforms, and then multi-feature integration is carried out, and then the multi-feature integration is input into a trained neural network model, so that an auxiliary identification result of the depression symptoms is output. The invention not only can provide objective indexes for assisting doctors to identify depression by using experimental waveforms and image data, but also has the advantages of high accuracy, strong robustness and low error rate.

Description

An auxiliary identification method for depression symptoms based on EEG data and facial expression images

Technical field

The present invention relates to the application field of artificial intelligence technology, and in particular to an auxiliary identification method for depression symptoms based on EEG data and facial expression images.

Background technique

As a common affective disorder, depression seriously affects the physical and mental health of patients. How to better assist doctors in identifying, distinguishing, and predicting the relevant indications of depression and exploring the construction of objective indicators have become a long-term and important way to assist in the diagnosis and treatment of depression. Important goals.

The event-related potentials (ERP) paradigm is a neurophysiological experimental design and analysis method used to study the brain's electrophysiological response to specific stimuli or tasks, also known as "cognitive potentials", in revealing psychological-cognitive Great advantage in knowing the process. N270 is a negative wave recorded at 270ms when the two stimulations do not completely match. It reflects the brain's ability to process conflicting information. It can be used as an electrical activity indicator of conflict processing ability and has strong specificity. Patients with depression have obvious core symptoms of conflict handling impairment and negative bias, which are also the main reasons why patients with depression cannot socialize normally and restore social functions. Therefore, N270, as an electrical activity index to evaluate the ability to process conflict information, can be used as a specific objective index to predict depression independent of depressive syndrome. Previous imaging studies have shown that patients with depression have impaired conflict processing systems in work and social activities. Therefore, the present invention believes that N270 can be used as a sensitive objective indicator for predicting depression.

However, the previously known N270 task paradigm has the following problems: 1. The signal-to-noise ratio is low, requiring a large number of trial repetitions and data processing to extract reliable components. The experimental operation procedures are cumbersome and the subject's learning cost is high. This results in a relatively large amount of time and workload for the experiment; 2. In the stimulus presentation system, precise timing is not achieved between the event code and the stimulus code, and there is an error in time scale; 3. The materials used in the paradigm are mainly numbers, letters, graphics, etc. It cannot be well designed for the negative bias symptoms of depressed people; 4. Especially considering the single application of N270 EEG data, its reliability is relatively limited, which will have a certain impact on the subsequent generation of guidance reports.

Contents of the invention

In view of the above, the present invention aims to provide an auxiliary identification method for depression symptoms based on EEG data and facial expression images to solve the aforementioned technical problems.

The technical solutions adopted by the present invention are as follows:

The present invention provides an auxiliary identification method for depression symptoms based on EEG data and facial expression images, which includes:

Perform brain electrical stimulation experiments based on preset experimental paradigms, and simultaneously collect subjects' facial expression image data and individual EEG data;

Analyze the EEG data to obtain the N270 waveform;

Perform multi-feature extraction on facial expression image data and N270 waveform;

The multiple features are integrated and then input into the trained neural network model based on the self-attention mechanism to output the auxiliary identification results of depression symptoms.

In at least one of the possible implementations, the editing process of the preset experimental paradigm includes:

Based on Matlab and E-prime, grayscale photos with consistent physical properties were used; among them, the gender ratio in the grayscale photos was balanced, the subjects' expressions were balanced in neutral and negative proportions, and there were no facial marks, and half of them were The photo is partially blocked on the face;

Set a given number of trials, the duration of individual trials, and the total duration of the experiment.

In at least one possible implementation, the multi-feature extraction includes:

Using a pre-trained multi-task depth prediction model to extract facial features and physiological signal features from the facial expression image data;

Extract waveform features based on the N270 waveform;

Based on the facial expression image data and the N270 waveform, an emotional feature vector is obtained.

In at least one possible implementation, the multi-task depth prediction model includes a common feature extraction module and a multi-task feature fusion module;

The common feature extraction module is used to extract features of each task and restore rough depth maps, semantic segmentation maps, and surface vector maps;

The multi-task feature fusion module is used to perform multi-task fusion on the features extracted by the common feature extraction module, and can distinguish the common semantic features of each task and the unique semantic features of each task.

In at least one possible implementation, the common feature extraction module adopts a single-input multiple-output network and consists of at least the following four parts: an encoder, a multi-dimensional decoder, a multi-scale feature fusion sub-module and a refinement sub-module. module;

The encoder is used to extract features at multiple scales;

The multi-dimensional decoder is used to gradually expand the final features of the encoder through an upsampling module while reducing the number of channels;

The multi-scale feature fusion sub-module is used to combine different information at multiple scales into one;

The thinning sub-module is used to adjust the output size and number of channels of the image.

In at least one possible implementation, the multi-task feature fusion module adopts a multiple-input multiple-output network and consists of at least the following two parts: a multiple-input feature fusion module, which is used to combine the multi-task features output by the previous module. Features are fused; the feature decoding part is a multi-output decoder.

In at least one possible implementation, the output of the auxiliary identification result of depression symptoms includes:

Fusion of multiple features in time order to obtain spatio-temporal feature vectors;

The spatio-temporal feature vector is input into the Transformer encoder model, and then the auxiliary recognition result is obtained through softmax classification.

Compared with the existing technology, the main design concept of the present invention is to focus on the core symptoms of conflict processing disorder and negative bias in depressed people, organically combine brain electrophysiological indicators and facial imaging indicators to evaluate the degree of conflict processing disorder, and then provide Objective and quantitative indicators are extracted for the identification and prediction of depression. Specifically, the brain electrical stimulation experiment is performed based on the preset experimental paradigm, and the subject's facial expression image data and individual EEG data are simultaneously collected. After the N270 waveform is obtained through analysis, multi-feature extraction is performed on the facial expression image data and N270 waveform. The multiple features are then integrated and input into the trained neural network model to output the auxiliary identification results of depression symptoms. The present invention can not only provide objective indicators to assist doctors in identifying depression using experimental waveforms and image data, but also has the advantages of high accuracy, strong robustness and low error rate.

Furthermore, applying machine learning technology to the detection and feature extraction of facial expressions or micro-expressions of patients with depression, combined with the deep learning of multi-task modules, can significantly improve the sensitivity, accuracy and specificity of identification and prediction of depression indicators. .

Description of the drawings

In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described below in conjunction with the accompanying drawings, in which:

Figure 1 is a schematic flowchart of an auxiliary identification method for depression based on EEG data and facial expression images provided by an embodiment of the present invention.

Detailed ways

The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention and cannot be construed as limiting the present invention.

The present invention proposes an embodiment of an auxiliary identification method for depression based on EEG data and facial expression images. Specifically, as shown in Figure 1, it includes:

Step S1: Perform a brain electrical stimulation experiment based on a preset experimental paradigm, and simultaneously collect the facial expression image data of the subject (facial expressions can be collected and recorded close-up through a high-definition camera) and individual EEG data (specifically, brain electrogram time domain data);

In the preset paradigm mentioned here, the specific editing process can include the following parts: based on Matlab and E-prime, using grayscale photos with consistent physical properties; among them, the gender ratio in the grayscale photos is balanced, and the subject's expression is neutral Balanced with negative proportions, and without any facial marks (such as glasses, beards, skin moles, jewelry, etc.), half of the photos are partially occluded on the face; set a set number of trials and the duration of a single trial As well as the total duration of the experiment, for example, set at least 180 trials, each trial lasts 500ms, and the total duration of the interval between successively presented pictures and stimulation is about 8 minutes.

This new N270 task paradigm has the following advantages: 1. The experimental process is simple and the subjects are easy to operate; 2. The parameters are reasonably designed, the induced EEG waveforms are repeatable, the signal-to-noise ratio is high, the timing system is accurately calibrated, and the timing error range is controllable. ; 3. The picture material covers negative and neutral emotions, targeting the core symptoms of negative bias in depressed people; 4. It is task-sensitive and highly specific. It is designed for the core symptom of conflict processing disorder and can be used as an early predictor of depression. objective indicators.

In addition, the present invention believes that in addition to core symptoms such as depressive syndrome, conflict processing disorder, and negativity bias, depression also has changes in facial expressions or micro-expressions, such as an increase in negative expressions of sadness and a decrease in positive expressions of smiles. . Although facial expressions are produced by muscle contraction, their changes are consistent with internal psychological changes and can be used as a window for transmitting information such as emotions, intentions, and desires. When patients with depression are stimulated by conflicting information (conflicting information refers to information that conflicts with the patient's inner beliefs, values, or expectations), due to the weakening of conflict processing abilities, negative emotions such as uneasiness, anxiety, and stress will be triggered, and facial expressions will It will further transform into negative emotions, such as depression, despair, helplessness, etc. Based on this, when subjects perform the task of stimulating the brain's conflict processing system with the new N270 paradigm proposed by the present invention, they can simultaneously collect the patient's facial expressions as feature enhancements, and can preferably use AI-assisted algorithms to conduct in-depth learning of facial expression features. , thereby further deepening the identification of the conflict handling ability of depressed people, and extracting and quantifying facial features.

Step S2: Analyze the EEG data to obtain the N270 waveform;

In actual operation, it mainly includes:

After importing the EEG data and performing independent component analysis, set the N270 segmented time window;

The data segments of different stimuli and the same stimulus were superimposed and averaged, and the N270 waveform was obtained by waveform subtraction, and the N270 waveform was determined to be a negative wave between 220ms and 380ms after the stimulus presentation.

The complete waveform interpretation process can be found as follows:

a) Import data: Start Matlab and EEGlab, select the corresponding import format to import data;

b) Downsampling: adjust the sampling rate to 250Hz, compress data, and reduce the impact of high-frequency information;

c) Import channel information: determine the coordinates of each electrode; eliminate useless electrodes: filter leads according to actual conditions and needs;

d) Re-reference: use the bilateral mastoid processes as a reference;

e) Filtering: perform band-pass filtering from 0.5Hz to 45Hz;

f) Independent component analysis ICA: perform ICA analysis on the data in order to remove artifacts in the signal

Especially EOG artifacts and removing noise components;

g) Segmentation: The N270 segmentation time window is set to 200ms before stimulation and 800ms after stimulation;

h) Baseline correction: use 200ms before stimulation as the baseline for correction;

i) Superposition average: superimpose and average the data fragments of different stimuli and the same stimulus respectively, and subtract the waveforms of different face stimuli and the same face stimulus to obtain the N270 waveform;

j) ERP component identification: N270 is a negative wave between 220-380ms after stimulus presentation.

Step S3: Extract multiple features from facial expression image data and N270 waveform;

Specifically, the multi-feature extraction may include:

Extract facial features and physiological signal features based on the facial expression image data;

Extract waveform features based on the N270 waveform;

Based on the facial expression image data and the N270 waveform, an emotional feature vector is obtained.

What needs to be explained here is that the depth prediction algorithm of multi-task feature integration is an algorithm that can not only extract the same features of each task, but also use the mutual relationship between each task to promote each other. It is designed to perform feature fusion of task interaction. or parameter sharing. The main problem with the current multi-task feature integration model is that the feature fusion method only uses parameter sharing, and does not show the help of common features to tasks and the characteristics of each task. Therefore, the present invention introduces a multi-task based feature integration model. The deep learning method of the task module can effectively solve this problem.

For the above feature extraction process, the following implementation reference is given here:

(1) Facial features

Step S31: Process the facial expression image data into frames to obtain several image sequences;

Step S32: Use the pre-trained multi-task depth prediction model to extract facial features from the image sequence to obtain a facial frame sequence;

Step S33: Use the optical flow method to extract optical flow from adjacent frames of the facial frame sequence to obtain an optical flow sequence;

Step S34: Fusion and expansion of the face frame sequence and the optical flow sequence to obtain a one-dimensional face vector, and then linearly map the one-dimensional face vector to obtain a face embedding vector.

(2) Physiological signal characteristics

Step S310: Use the multi-task depth prediction model to extract the region of interest of the cheek from the image sequence to obtain the sequence of interest;

Step S320: Expand the sequence of interest to obtain a one-dimensional vector of interest, and then linearly map the one-dimensional vector of interest to obtain a physiological signal embedding vector.

(3) Waveform characteristics

Step S311: Extract the basic features of the N270 waveform in units of frames, combine the basic features into a one-dimensional waveform vector, and then linearly map the one-dimensional waveform vector to obtain a waveform embedding vector.

(4) Emotional characteristics

Step S301: Extract multi-dimensional facial feature vectors from the image sequence;

Step S302: Perform Fourier transform on the N270 waveform to convert it into frequency domain data, divide the frequency domain data matrix into blocks according to different frequency ranges based on frequency, obtain a block frequency domain matrix, and calculate the block After calculating the covariance matrix of each sub-block of the frequency domain matrix, calculate the LES of each covariance matrix to obtain the LES feature vector;

Step S303: Splice the multi-dimensional facial feature vector and the LES feature vector to obtain a fusion vector; input the fusion vector into a pre-trained emotional feature extraction model to obtain an emotional embedding vector corresponding to the image sequence.

Step S4: Integrate the multiple features and input them into the trained neural network model based on the self-attention mechanism, and output the auxiliary identification results of depression symptoms.

Specifically, this step can be, but is not limited to, the following ideas:

Fusion of multiple features in time order to obtain spatio-temporal feature vectors;

The spatio-temporal feature vector is input into the Transformer encoder model, and then the auxiliary recognition result is obtained through softmax classification.

Furthermore, regarding the multi-task depth prediction model mentioned above, network training can be carried out through two modules. One of them is a common feature extraction module, which is responsible for extracting the features of each task and can restore rough depth maps, semantic segmentation maps, and surface Vector diagram; the other module is the multi-task feature fusion module, which is responsible for multi-task fusion of the features extracted by the common feature extraction module. The network will distinguish the semantic features shared by each task and the semantic features unique to each task, so that The final recovered image is more structured. Specifically:

(1) About the shared feature extraction module

(1) Network structure

The total feature extraction module consists of four parts: encoder, multi-dimensional decoder, multi-scale feature fusion sub-module and refinement sub-module. Specifically, the encoder can be composed of four convolutional layers, responsible for extracting features at multiple scales of 1/4, 1/8, 1/16, and 1/32; the multidimensional decoder uses four upsampling modules to gradually expand the encoder final features while reducing the number of channels; the multi-scale feature fusion sub-module uses upsampling and channel connection to integrate four different scale features of the autoencoder: corresponding to the encoder, the four-layer output of the encoder (each (with 16 channels) are upsampled in the form of ×2, ×4, ×8 and ×16 respectively to have the same size as the final output. This upsampling is done in a channel-connected manner and then further transformed by convolutional layers. In order to obtain an output with 64 channels, the main purpose of the multi-scale feature fusion sub-module is to merge different information from multiple scales into one, so that the lower layer output of the encoder retains information with finer spatial resolution, which helps To restore the detailed information lost due to multiple downsampling; finally, the refinement sub-module is used to adjust the output size and channel number of the image, using three convolutional layers corresponding to three tasks, so that the output channel number is restored to depth 1 channel for images, 1 channel for semantic segmentation images and 3 channels for surface vectors to facilitate loss calculation and backpropagation. Taking the above embodiment as a schematic:

a) Process the facial expression image data and N270 waveform into frames respectively to obtain multiple image sequences;

b) The image is edited into an RGB image of 320×240×3 size;

c) Obtain four layers of features through the four convolutional layers of the encoder;

d) The four-layer features are decoded through the four sampling layers of the decoder;

e) After completion, the upsampling outputs of different scales are fused together through the multi-scale fusion sub-module, and finally the shapes of the depth image, semantic segmentation map, and surface vector map are restored through different deconvolution layers in the refinement sub-module.

(2) Training process

a) Establish a loss function between the RGB image, depth image, semantic segmentation map, surface vector map and the corresponding image predicted by the network, and the number of images processed at each time is 4;

b) Update network parameters by reducing the loss function;

c) After iterative updating until the loss function converges, set the number of iterations to 100.

The shared feature extraction module is a single-input multiple-output network. During the training process, the network needs to establish a loss function between the RGB image and depth image, semantic segmentation map, surface vector map and the corresponding image predicted by the network in the data set. When the network parameters are updated and iterated until the loss function converges, the trained network can be obtained. Since three tasks need to be processed at the same time and the correlation between the tasks must be ensured, the above loss function can be divided into three parts. The specific feature extraction loss function is:

L task = L depth + L seg + L normal

The functions and purposes of each part of the loss function are as follows:

(a) Depth map loss function, that is:

L depth = L 1 + L grad

This function is the sum of the L 1 loss function and the gradient loss function. For each pixel point i , the corresponding predicted depth and true depth are di and Di respectively. The L 1 loss function can constrain the difference between di and Di , which is The main part of the loss function provides the guarantee of accuracy.

(b) Semantic segmentation loss function, which is the cross-entropy function, is a commonly used loss function in semantic segmentation problems. It is often used to describe the distance between two probabilities. In this model, the probability of the predicted semantic category of the object in the image is constrained, Its expression is:

Among them, for each pixel in the predicted semantic segmentation image S and the true value image, s and s are elements that are either 0 or 1 in classes. There is only one sj value for each pixel, so the intersection Entropy loss only cares about the predicted probability of the correct category. As long as its value is large enough, it can ensure that the classification result is correct.

(c) Surface vector loss function, which measures the accuracy of the estimated depth map surface normal ( n _i ^d ) relative to its real data surface normal ( n _i ^g ). The surface vector loss is also calculated based on the depth gradient, but it measures the angle between the two surface normals. Therefore, the loss is very sensitive to the depth structure and can improve the consistency of the predicted structure. The expression is:

(d) The gradient loss function can constrain the gradient changes of points on the x- axis and y -axis ( gx ( ei ) and gv ( ei ), respectively, are the gradient losses on the x- and y -axes), and sensitively detect edge information, Depth is often discontinuous at the boundaries of an object. It should be noted that the gradient loss and the previous depth loss are different types of errors, so the network that needs weighting to train is expressed as:

During the training process of the common feature extraction module, the images of the three tasks are obtained directly at the output end of the network and the loss function is designed between the corresponding true values, and backpropagation is performed through gradient descent, thereby improving the parameters of the network. Updates are made and when the loss function converges, training can be completed. In an embodiment of training the shared feature extraction module algorithm, the number of iterations is set to 100, the number of images processed each time is 4, the initial learning rate is 10-4, and the learning rate is updated to the original 10 after every 20 iterations. 1/1, and the final convergence loss function value obtained after 100 iterations is 0.1245.

(2) Regarding the multi-task feature fusion module

(1) Network structure

The multi-task feature fusion module consists of two parts. The first part is the multi-input feature fusion module, which is responsible for fusing the multi-task features output by the previous module. The network used is densely connected U-net; the second part is feature decoding. The decoder part is similar to the decoder part in the previous part and is a multi-output decoder, so no further details will be given. Combined with the previous examples, specifically:

a) Create an encoding path consisting of multiple streams, each stream processing the image form of a different task in the previous module;

b) The three tasks are first encoded by the U-net encoder respectively. The pooled features of the image of task 1 after a convolution and pooling operation will be combined with the secondary pooling features of task 2. Through the convolution After layering, it is combined with the pooling features of Task 3 to ensure the commonality of features;

c) The obtained common features are first obtained through an upsampling operation to obtain a common upsampling feature;

d) Combine this upsampling feature with the previous pooling feature for decoding, and send it and the pooling features of different scales for the three tasks to the decoder respectively;

e) Connect with the features extracted from each previous task and restore the original shape of each task through the upsampling layer;

f) Compare the recovered depth image, semantic segmentation map, and surface vector map with the true values in the data set to update the parameters in the network.

The multi-task feature fusion module adds a dense connection method based on the original U-net network, which can effectively enhance the feature extraction capabilities of multi-input modes. In order to realize this dense connection mode, first create a multi-stream Composed of an encoding path, each stream handles a different task in the image form of the previous module. The main purpose of having separate streams for different image modalities is to fragment information that would otherwise be fused early on, thus limiting the network's ability to learn complex relationships between patterns. It can be seen from the network structure that the three tasks are first encoded by the U-net encoder, but the difference is that there will be interaction when passing through different convolutional layers. For example, the image of task 1 is passed through a convolution layer. After the pooling operation, the resulting pooled features will be combined with the secondary pooling features of Task 2, and then combined with the pooled features of Task 3 after passing through the convolution layer. This allows features to flow between tasks and ensures the commonality of features. In the decoder part, the obtained common features are first obtained through an upsampling operation to obtain a common upsampled feature, and then the upsampled feature is combined with the previous pooling feature for decoding, and it is combined with the pooling of different scales for the three tasks. The features are sent to the decoder, and are connected with the features extracted from each previous task and passed through the upsampling layer to restore the original shape of each task, and then combine the restored depth image, semantic segmentation map, and surface vector map with the ones in the data set. The true value is compared with the loss to update the parameters in the network. Through this combination of channel connection and downsampling connection, the features of different tasks can be further fused and the transformation between tasks can be promoted. This method is different from the shared features in the shared feature extraction module. While retaining the original features, It can better reflect the connection between different tasks and highlight the integration of multi-task features.

(2) Training process

a) Establish a loss function between the output and the corresponding depth image, semantic segmentation map, and surface vector map true value in the database, and the number of images processed each time is 4;

b) Update the network parameters by reducing the loss function. The loss function is the same as the common feature extraction module;

c) After iterative updating until the loss function converges, set the number of iterations to 100.

The multi-task feature fusion module belongs to a multi-input multi-output network. During training, it is necessary to establish a loss function based on the relationship between the output and the corresponding depth image, semantic segmentation map, and surface vector map true value in the database, and reduce the A small loss function is used to update the network parameters. After iterative updating until the loss function converges, a well-trained network is obtained. When training the network, this module can be trained together with the common feature extraction module to form an end-to-end neural network, that is, the input is a single RGB image, and the output is its corresponding depth image, semantic segmentation map, and surface vector map. Since the three tasks are the same as the common feature extraction module, the same loss function is used for constraints, so no detailed description is given. Regarding the training process of this module, this model directly obtains the images of the three tasks at the output end of the network and designs a loss function between them and their corresponding true values, and performs backpropagation through gradient descent, thereby performing the parameters of the network. Update, when the loss function converges, training can be completed. During the training process of this model algorithm, the number of iterations is set to 100, the number of images processed each time is 4, the initial learning rate is 10-4, and the learning rate is updated to one-tenth of the original value after every 20 iterations, and finally The convergence loss function value obtained after 100 iterations is 0.1159.

To sum up, the main design concept of the present invention is to focus on the core symptoms of conflict processing disorder and negativity bias in depressed people, organically combine electroencephalogram physiological indicators and facial imaging indicators to evaluate the degree of conflict processing disorder, and then provide the basis for depression. Identify and estimate to extract objective and quantitative indicators. Specifically, the brain electrical stimulation experiment is performed based on the preset experimental paradigm, and the subject's facial expression image data and individual EEG data are simultaneously collected. After the N270 waveform is obtained through analysis, multi-feature extraction is performed on the facial expression image data and N270 waveform. The multiple features are then integrated and input into the trained neural network model to output the auxiliary identification results of depression symptoms. The present invention can not only provide objective indicators to assist doctors in identifying depression using experimental waveforms and image data, but also has the advantages of high accuracy, strong robustness and low error rate.

In the embodiment of the present invention, "at least one" refers to one or more, and "multiple" refers to two or more. "And/or" describes the relationship between associated objects, indicating that there can be three relationships. For example, A and/or B can represent the existence of A alone, the existence of A and B at the same time, or the existence of B alone. Where A and B can be singular or plural. The character "/" generally indicates that the related objects are in an "or" relationship. "At least one of the following" and similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can mean: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single, also Can be multiple.

The structure, features and effects of the present invention have been described in detail based on the embodiments shown in the drawings. However, the above are only preferred embodiments of the present invention. It should be noted that the technologies involved in the above-mentioned embodiments and their preferred modes Characteristics, those skilled in the art can rationally combine and match them into various equivalent solutions without departing from or changing the design ideas and technical effects of the present invention; therefore, the scope of the present invention is not limited to what is shown in the drawings. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which do not exceed the spirit covered by the description and drawings, should be within the protection scope of the present invention.

Claims (7)

1. An auxiliary identification method for depression symptoms based on brain electrical data and facial expression images is characterized by comprising the following steps:

performing an electroencephalogram stimulation experiment based on a preset experimental paradigm, and synchronously collecting facial expression image data of a tested person and electroencephalogram data of an individual;

analyzing the electroencephalogram data to obtain an N270 waveform;

performing multi-feature extraction on the facial expression image data and the N270 waveform;

and inputting the integrated multi-feature into a trained neural network model based on a self-attention mechanism, and outputting an auxiliary identification result of the depression symptoms.

2. The method for assisting in identifying a depressive disorder based on electroencephalogram data and facial expression images according to claim 1, wherein the editing process of the preset experimental paradigm comprises:

based on Matlab and E-prime, using gray photos with consistent physical properties; the sex proportion in the gray photos is balanced, the tested list is neutral and the negative proportion is balanced, no facial mark exists, and partial facial shielding treatment is carried out on half of the photos;

a given number of trials, the duration of a single trial, and the total duration of the experiment are set.

3. The method for assisting in the identification of a depressive disorder based on electroencephalogram data and facial expression images according to claim 1, wherein the multi-feature extraction comprises:

extracting facial features and physiological signal features from the facial expression image data by utilizing a pre-trained multitasking depth prediction model;

extracting waveform features based on the N270 waveform;

and obtaining emotion feature vectors based on the facial expression image data and the N270 waveform.

4. The method for assisting in identifying a depression disorder based on electroencephalogram data and facial expression images according to claim 3, wherein the multi-task depth prediction model comprises a common feature extraction module and a multi-task feature fusion module;

the common feature extraction module is used for extracting features of each task and recovering a rough depth map, a semantic segmentation map and a surface vector map;

the multi-task feature fusion module is used for carrying out multi-task fusion on the features extracted by the common feature extraction module, and can distinguish the common semantic features of each task and the unique semantic features of each task.

5. The method for assisting in identifying a depressive disorder based on electroencephalogram data and facial expression images according to claim 4, wherein the common feature extraction module adopts a single-input multi-output network and is composed of at least four parts: the device comprises an encoder, a multi-dimensional decoder, a multi-scale feature fusion sub-module and a refinement sub-module;

the encoder is used for extracting characteristics of various scales;

the multi-dimensional decoder is used for gradually expanding the final characteristics of the encoder through an up-sampling module and reducing the number of channels at the same time;

the multi-scale feature fusion submodule is used for combining different information of a plurality of scales into one;

the refinement sub-module is used for adjusting the output size of the image and the channel number.

6. The method for assisting in identifying a depressive disorder based on electroencephalogram data and facial expression images according to claim 4, wherein the multi-task feature fusion module adopts a multi-input multi-output network and is composed of at least two parts: the multi-input feature fusion module is used for fusing the multi-task features output by the previous module; the feature decoding section is a multi-output decoder.

7. The auxiliary identification method for depression symptoms based on brain electrical data and facial expression images according to any one of claims 1 to 6, wherein the outputting the auxiliary identification result for depression symptoms comprises:

fusing the multiple features in time sequence to obtain space-time feature vectors;

and inputting the space-time feature vector into a Transformer encoder model, and classifying by softmax to obtain an auxiliary recognition result.