CN115732076A - Fusion analysis method for multi-modal depression data - Google Patents

Abstract

The invention discloses a fusion analysis method of multi-modal depression data in the field of depression data fusion, and aims to extract and fuse emotional characteristics of different types of data recorded in multiple stages. And aiming at different types of data, data characteristics are provided according to the characteristics of the data. And then, the data features of different modes respectively obtain K value, Q value and V value expressions through three linear layers, then the attention A of the data of each mode is calculated by utilizing K and Q according to the attention mechanism of fused depression data, and A.V is taken as the feature after fusion to serve downstream tasks. Compared with the existing technology for researching the depression through the single mode, the technical scheme is not easily influenced by factors such as individual difference and the like, can better utilize the screening characteristics of a case set on the individual difference of a patient, and then fuses images, actions and sounds of the patient according to the screening characteristics to realize comprehensive diagnosis.

Description

A Fusion Analysis Method for Multimodal Depression Data

technical field

The invention belongs to the field of multimodal data fusion, in particular to a multimodal fusion analysis method applied to emotion recognition.

Background technique

Due to its high incidence and great harm, depression has become an internationally recognized public health problem that seriously threatens human physical and mental health. Early identification and early intervention are crucial to reducing the risk of depression. The traditional diagnosis of depression is made by doctors based on clinical experience and scales. This method mainly relies on single-modal data, which has subjective bias, lag, passivity, and limitations. Jeffery et al. found that using multimodal technology to identify depression is better than unimodal.

Multimodal techniques refer to a method of simultaneously processing or fitting multiple modal data to enhance model performance. Data of different modalities is difficult to be aligned and fused due to different forms of expression and different meanings. For example, in image and audio recognition tasks, image data is usually presented as pictures, while language data is usually presented as text, and the two are difficult to integrate due to different forms of expression; Different and difficult to integrate.

Existing works also explore a lot of multimodal techniques. Dupont, S et al. used hidden Markov joint finite automata to align speech data with image data, and used bimodal data to recognize speech and images. This method fuses data in different forms to a certain extent, but it still has the disadvantages of low efficiency and poor generalizability. Another way of thinking is to use neural networks for multi-data fusion. Zeng, X et al. used multi-modal self-encoders to fuse 10 kinds of drug description information (such as side effects, action pathways, etc.), and input disease types to classify the types of diseases to achieve the matching of drug types. The second The first step is to segment the symptoms of the disease, and to increase or decrease the corresponding drug dosage according to the differences in the performance of each symptom on different individuals. This multimodal fusion method fails to fully consider the relationship between modalities, and there is no way to fuse data of different representations. In summary, although there have been many attempts in multimodal technology, there is still no method that can well integrate multimodal data.

Contents of the invention

In order to solve the above problems, the object of the present invention is to provide a fusion analysis method of multimodal depression data.

In order to achieve the above object, the technical solution of the present invention is as follows: a fusion analysis method of multi-modal depression data, data of different data categories is entered into multi-stage data, and at this time, the entered data is subjected to emotional feature extraction, after that, different The data characteristics of the modality are expressed through the three linear layers to obtain the K value, Q value and V value, and then according to the attention mechanism of the fusion depression data, use K and Q to calculate the attention A of each modality data, and A·V As a fused feature, it serves downstream tasks. Due to the attention mechanism of fused depression data, the fused data features will contain multimodal information and can assist downstream classification tasks.

Further, including the following steps,

S1 data preprocessing, the data group is divided into text data, image data and audio data;

S2 integrates the attention mechanism of depression data and calculates the preprocessed data to obtain features containing multimodal information;

S3 depression recognition, splicing features containing multi-modal information, and output a fused data feature through a linear layer, the neurons in the last layer use the softmax function as the activation function and output classification prediction results.

Further, the text data in S1 includes scales and electronic medical records, and the scales and electronic medical record data are subjected to feature preliminary screening, missing value processing, feature encoding and normalization.

Further, the video data in S1 is extracted at a frequency of 20 frames per second. After denoising and de-artifacting the obtained image data, the face position detection is performed on each frame of the image, and the images are aligned according to the position of the eyes, and then the video image is cropped as 256×256 pixel face image.

Further, after the audio data in S1 is aligned with the image set acquired by frame extraction, Mel-frequency cepstral coefficients are extracted for each aligned speech segment.

Further, in the said S2, the K value corresponding to the text data, the image data and the audio data are calculated respectively, the Q value, the V value and the K value, and the Q value calculates the auxiliary attention of the video, audio and text respectively, and the three auxiliary attention After force splicing and passing through the Softmax function, the attention of video, audio, and text is formed and multiplied by the V value calculated in the previous step.

Further, the predicted result in S3 uses a cross-entropy loss function to fit the difference between the predicted value and the real value.

After adopting the above scheme, the following beneficial effects have been achieved: 1. Compared with the prior art of studying depression through a single modality, the single modality will be affected by factors such as individual differences. The discriminative features are then used to fuse the patient's image, motion and voice for a comprehensive diagnosis.

2. Compared with the traditional splicing data fusion method, this technical solution produces the following effects: Combining information of different modalities, and combining information representations of different modalities on the media. Then there is the issue of alignment, aligning different modal messages and dealing with possible dependencies. Finally, there is the transformation problem, so that the information of multiple modalities can be unified.

Description of drawings

Figure 1 is the framework of the multimodal fusion assessment model for perinatal depression;

Figure 2 shows the attention mechanism method for fusing depression data.

Detailed ways

The following is further described in detail through specific implementation methods:

The embodiment is basically as shown in accompanying drawing 1 and accompanying drawing 2: a kind of fusion analysis method of multimodal depression data carries out multi-stage data entry of data of different data categories, at this moment, the emotional feature extraction is performed on the entered data, after that, The data characteristics of different modalities are expressed through three linear layers to obtain the K value, Q value and V value, and then according to the attention mechanism of fusion depression data, K and Q are used to calculate the attention A of each modal data, and A· V is used as a fused feature to serve downstream tasks. Due to the attention mechanism of fused depression data, the fused data features will contain multimodal information and can assist downstream classification tasks.

The specific implementation process is as follows: the input of the present invention is video, audio and text data. It is divided into three main stages: data preprocessing, integrated depression data attention mechanism (Integrated Depression Data Attention, IDDA), and depression recognition. Include the following steps,

S1 data preprocessing, the data group is divided into text data, image data and audio data, the text data includes scales and electronic medical records, and the scales and electronic medical record data are subjected to feature preliminary screening, missing value processing, feature encoding and normalization The video data is extracted at a frequency of 20 frames per second. After denoising and de-artifacting the obtained image data, the face position detection is performed on each frame of the image, and the image is aligned according to the eye position, and then the video image is cropped to 256× 256-pixel facial image, audio data is aligned with the image set obtained by frame extraction, and Mel frequency cepstral coefficients are extracted for each aligned speech segment;

S2 fuses the attention mechanism of depression data and calculates the preprocessed data to obtain features containing multimodal information. In order to better fuse data, we propose a new multimodal data fusion mechanism (IDDA). First, 1. Calculate the corresponding K value, Q value and V value for the three kinds of data respectively. After that, use the K value and Q value to calculate the auxiliary attention of video, audio, and text respectively;

The three kinds of auxiliary attention are spliced and passed through the Softmax function to form the attention of video, audio, and text, and multiplied by the V value calculated in the previous step to obtain features containing multimodal information.

The features containing multimodal information are concatenated and passed through a linear layer to output a fused data feature, which is used as the input of the downstream task.

S3 depression recognition, splicing features containing multimodal information, and output a fused data feature through a linear layer, we chose LSTM as the classifier for downstream tasks. The model is optimized using the Adam optimizer, and the neurons in the last layer use the softmax function as the activation function and output classification prediction results. The cross-entropy loss function was used to fit the difference between the predicted value and the real value, and the learning rate of the model was 0.001.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device.

What is described above is only an embodiment of the present invention, and the common knowledge such as the specific structure and characteristics known in the scheme is not described too much here, and those of ordinary skill in the art know all the common knowledge in the technical field to which the invention belongs before the filing date or the priority date Technical knowledge, being able to know all the existing technologies in this field, and having the ability to apply conventional experimental methods before this date, those of ordinary skill in the art can improve and implement this plan based on their own abilities under the inspiration given by this application, Some typical known structures or known methods should not be obstacles for those of ordinary skill in the art to implement the present application. It should be pointed out that for those skilled in the art, under the premise of not departing from the structure of the present invention, several modifications and improvements can also be made, and these should also be regarded as the protection scope of the present invention, and these will not affect the implementation of the present invention. Effects and utility of patents. The scope of protection required by this application shall be based on the content of the claims, and the specific implementation methods and other records in the specification may be used to interpret the content of the claims.

Claims (7)

1. A fusion analysis method of multi-modal depression data is characterized by comprising the following steps: the method comprises the steps of carrying out multi-stage data entry on data of different data types, extracting emotion characteristics of the entered data, then respectively obtaining K values, Q values and V value expressions through three linear layers according to the data characteristics of different modes, calculating attention A of data of each mode by using K and Q according to a fused depression data attention mechanism, serving a downstream task by taking A and V as fused characteristics, and assisting the downstream classification task due to the fact that the fused depression data attention mechanism is fused, wherein the fused data characteristics contain multi-mode information.

2. The fusion analysis method of multimodal depression data according to claim 1, characterized by: comprises the following steps of (a) carrying out,

s1, preprocessing data, namely dividing data components into text data, image data and audio data;

s2, integrating a depression data attention mechanism, and calculating preprocessed data to obtain features containing multi-modal information;

and S3, identifying the depression, namely splicing the characteristics containing the multi-modal information, outputting a fused data characteristic through a linear layer, and outputting a classification prediction result by using a softmax function as an activation function for the neuron of the last layer.

3. The fusion analysis method of multimodal depression data according to claim 2, characterized by: and the text data in the S1 comprises a scale and an electronic case, and the scale and the electronic case history data are subjected to characteristic primary screening, missing value processing, characteristic coding and normalization.

4. The method for fusion analysis of multimodal depression data according to claim 2, wherein: and in the S1, the video data is subjected to image extraction according to the frequency of 20 frames per second, after the noise and artifact of the obtained image data are removed, the face position of each frame of image is detected, the image is aligned according to the eye position, and then the video image is cut into face images of 256 multiplied by 256 pixels.

5. The fusion analysis method of multimodal depression data according to claim 2, characterized by: and after the audio data in the S1 are aligned with the image set obtained by frame extraction, extracting a Mel frequency cepstrum coefficient for each aligned voice segment.

6. The fusion analysis method of multimodal depression data according to claim 2, characterized by: and in the S2, the K value, the Q value, the V value and the K value which correspond to the text data, the image data and the audio data respectively are calculated, the Q value calculates the auxiliary attention of the video, the audio and the text respectively, the three auxiliary attention are spliced and form the attention of the video, the audio and the text through a Softmax function, and the attention is multiplied by the V value calculated in the previous step.

7. The fusion analysis method of multimodal depression data according to claim 2, characterized by: and in the S3, the difference between the predicted value and the true value is fitted by adopting a cross entropy loss function as a prediction result.

Images