CN110301920A

CN110301920A - Multi-modal fusion method and device for psychological pressure detection

Info

Publication number: CN110301920A
Application number: CN201910567398.XA
Authority: CN
Inventors: 冯铃; 张慧君; 曹檑
Original assignee: Tsinghua University
Current assignee: Tsinghua University
Priority date: 2019-06-27
Filing date: 2019-06-27
Publication date: 2019-10-08
Anticipated expiration: 2039-06-27
Also published as: CN110301920B

Abstract

The embodiment of the present invention provides a kind of multi-modal fusion method and device for psychological pressure detection, the present invention is based on physiological data -> text, physiological data -> picture, text -> physiological data, text -> picture, picture -> physiological data, picture -> text attentions to strengthen eigenmatrix, based on the full Connection Neural Network that feedovers, the fusion feature matrix of text, picture and physiological data is obtained；Then the fusion feature matrix based on text, picture, the weights of importance value of physiological data and text, picture and physiological data, obtains the fusion representing matrix of three kinds of mode；The pressure class vector of reflection psychological pressure problem is finally obtained based on the fusion representing matrix for stating three kinds of mode and the fully-connected network that feedovers.The present invention not only compensates for the subjectivity bring deficiency by text and image data, solves the problems, such as some intrinsic of physiology related data by fusing text image data and physiology related data.

Description

Multi-modal fusion method and device for psychological pressure detection

Technical field

The present invention relates to field of computer technology more particularly to a kind of multi-modal fusion methods for psychological pressure detection And device.

Background technique

With the increase of social competition's pressure, teen-age psychological pressure problem is increasingly becoming one and more serious asks Topic.Excessive psychological pressure will lead to many physiology and psychological problem, this makes psychological pressure detection more and more important.

The existing psychological pressure detection work focused in social media is only focused in text and image content, however literary This and image content have subjectivity, and be sometimes beyond expression out true psychological condition.

The relevant work of existing some physiological signals demonstrates validity of the physiological signal when detecting psychological pressure, than Such as heart rate variability, electrocardiogram, galvanic skin reaction, electroencephalogram, blood pressure and electromyogram etc..But physiological signal related data There is a problem of some intrinsic, for example in the physiology related data of the state of being on wires and extreme pressure state is closely similar , therefore, true psychological condition can not also be given expression to according to physiological signal related data completely sometimes.

According to being described above it is found that lacking a kind of effective psychological pressure detection method and device at present.

Summary of the invention

For the problems of the prior art, the embodiment of the present invention provides a kind of multi-modal fusion for psychological pressure detection Method and device.

In a first aspect, the embodiment of the invention provides the attentions that a kind of pair of both modalities which data carry out feature interaction fusion Weight corresponding method, comprising:

Reflection both modalities which data different characteristic is obtained using matrix multiplication based on the eigenmatrix of both modalities which data Between information relevance incidence relation matrix；

Based on the incidence relation matrix and feedforward fully-connected network model, the feature square of one of modal data is obtained Influence power weight matrix of the battle array to the eigenmatrix of another modal data；

Based on the eigenmatrix of the influence power weight matrix and described two modal datas, matrix dot product and residual error are utilized Connection, obtain comprising described two modal datas eigenmatrix influence each other weight attention strengthen eigenmatrix.

Second aspect, the embodiment of the invention provides a kind of based on as described in relation to the first aspect to the progress of both modalities which data The multi-modal fusion method for psychological pressure detection of the attention weight corresponding method of feature interaction fusion, comprising:

The physiological data correlation eigen matrix and reflection user psychology moving type of reflection user's physiological status are obtained respectively The text feature matrix and picture feature matrix of state；

Based on the physiological data correlation eigen matrix, the text feature matrix and the picture feature matrix, utilize The attention weight corresponding method obtains mutual to the text feature matrix comprising the physiological data correlation eigen matrix First attention of weighing factor strengthens eigenmatrix, includes the physiological data correlation eigen matrix to the picture feature square Influence each other the second attention of weight of battle array strengthens eigenmatrix, related to the physiological data comprising the text feature matrix The influence each other third attention of weight of eigenmatrix strengthens eigenmatrix, special to the picture comprising the text feature matrix Sign matrix influence each other weight the 4th attention strengthen eigenmatrix, comprising the picture feature matrix to the physiological data Correlation eigen matrix influence each other weight the 5th attention strengthen eigenmatrix and comprising the picture feature matrix to described Text feature matrix influence each other weight the 6th attention strengthen eigenmatrix；

Strengthen eigenmatrix based on first attention, second attention strengthens eigenmatrix, third note Power of anticipating strengthens eigenmatrix, the 4th attention strengthens eigenmatrix, the 5th attention strengthens eigenmatrix and described 6th attention strengthens eigenmatrix, and based on the full Connection Neural Network that feedovers, acquisition text fusion feature matrix, picture fusion are special Levy matrix and physiological data fusion feature matrix；

Based on the physiological data correlation eigen matrix, the text feature matrix and the picture feature matrix, it is based on Feedover full Connection Neural Network, obtains text, picture, physiological data characteristic value；

Text, figure are obtained based on vector splicing and attention mechanism based on the text, picture, physiological data characteristic value The weights of importance value of piece, physiological data；

Based on the text, picture, the weights of importance value of physiological data and the text fusion feature matrix, described Picture fusion feature matrix and the physiological data fusion feature matrix, obtain the fusion representing matrix of three kinds of mode；

Fusion representing matrix and feedforward fully-connected network based on three kinds of mode, obtain reflection psychological pressure problem Pressure class vector.

The third aspect, the embodiment of the invention also provides the attentions that a kind of pair of both modalities which data carry out feature interaction fusion Power weight corresponding intrument, comprising:

First obtains module, obtains two kinds of reflection using matrix multiplication for the eigenmatrix based on both modalities which data The incidence relation matrix of information relevance between modal data different characteristic；

Second obtains module, for obtaining wherein one based on the incidence relation matrix and feedforward fully-connected network model Influence power weight matrix of the eigenmatrix of kind modal data to the eigenmatrix of another modal data；

Third obtains module, for the eigenmatrix based on the influence power weight matrix and described two modal datas, It is connected using matrix dot product with residual error, obtains the eigenmatrix comprising described two modal datas and influence each other the attention of weight Strengthen eigenmatrix.

Fourth aspect, the embodiment of the invention also provides it is a kind of based on as described in the third aspect to both modalities which data into The multi-modal fusion device for psychological pressure detection of the attention weight corresponding intrument of row feature interaction fusion, comprising:

4th obtains module, for obtaining the physiological data correlation eigen matrix of reflection user's physiological status and anti-respectively Reflect the text feature matrix and picture feature matrix of user psychology active state；

5th obtains module, for based on the physiological data correlation eigen matrix, the text feature matrix and described Picture feature matrix is obtained comprising the physiological data correlation eigen matrix using the attention weight corresponding method to institute It states influence each other the first attention of weight of text feature matrix and strengthens eigenmatrix, comprising the physiological data correlated characteristic square Battle array strengthens eigenmatrix, comprising the text feature matrix to influence each other the second attention of weight of the picture feature matrix Eigenmatrix, special comprising the text is strengthened to the influence each other third attention of weight of the physiological data correlation eigen matrix Sign matrix strengthens eigenmatrix, comprising the picture feature to influence each other the 4th attention of weight of the picture feature matrix Matrix strengthens eigenmatrix and comprising described to influence each other the 5th attention of weight of the physiological data correlation eigen matrix Picture feature matrix to the text feature matrix influence each other weight the 6th attention strengthen eigenmatrix；

6th obtains module, special for being strengthened based on first attention reinforcing eigenmatrix, second attention It is strong to levy matrix, third attention reinforcing eigenmatrix, the 4th attention reinforcing eigenmatrix, the 5th attention Change eigenmatrix and the 6th attention strengthens eigenmatrix, based on the full Connection Neural Network that feedovers, it is special to obtain text fusion Levy matrix, picture fusion feature matrix and physiological data fusion feature matrix；

7th obtains module, for based on the physiological data correlation eigen matrix, the text feature matrix and described Picture feature matrix obtains text, picture, physiological data characteristic value based on the full Connection Neural Network that feedovers；

8th obtains module, for being based on the text, picture, physiological data characteristic value, is spliced based on vector and is paid attention to Power mechanism obtains the weights of importance value of text, picture, physiological data；

9th obtains module, for based on the text, picture, the weights of importance value of physiological data and the text Fusion feature matrix, the picture fusion feature matrix and the physiological data fusion feature matrix, obtain melting for three kinds of mode Close representing matrix；

Tenth obtains module, for fusion representing matrix and feedforward fully-connected network based on three kinds of mode, obtains Negate the pressure class vector for reflecting psychological pressure problem.

5th aspect the embodiment of the invention also provides a kind of electronic equipment, including memory, processor and is stored in On reservoir and the computer program that can run on a processor, the processor are realized when executing described program such as first aspect institute The step of attention weight corresponding method of feature interaction fusion is carried out to both modalities which data is stated, and/or, such as second aspect institute The step of stating the multi-modal fusion method for psychological pressure detection.

6th aspect, the embodiment of the invention also provides a kind of non-transient computer readable storage mediums, are stored thereon with Computer program is realized when the computer program is executed by processor and carries out feature to both modalities which data as described in relation to the first aspect The step of attention weight corresponding method of interaction fusion, and/or, the multimode as described in second aspect for psychological pressure detection The step of state fusion method.

By prior art scheme it is found that provided in an embodiment of the present invention carry out feature interaction fusion to both modalities which data Attention weight corresponding method and device obtain two kinds of reflection using matrix multiplication based on the eigenmatrix of both modalities which data The incidence relation matrix of information relevance between modal data different characteristic, and connected entirely based on the incidence relation matrix and feedforward Network model is connect, the eigenmatrix for obtaining one of modal data weighs the influence power of the eigenmatrix of another modal data Weight matrix, the eigenmatrix finally based on the influence power weight matrix and described two modal datas, using matrix dot product and Residual error connection, obtain comprising described two modal datas eigenmatrix influence each other weight attention strengthen eigenmatrix, The embodiment of the present invention realizes the attention weight that both modalities which data are carried out with feature interaction fusion by treatment process above Corresponding method, and it is based on this method, another embodiment of the present invention provides a kind of multi-modal fusions for psychological pressure detection Method and device is based on the physiological data correlation eigen matrix, the text feature matrix and the figure in this embodiment Piece eigenmatrix is obtained comprising the physiological data correlation eigen matrix using the attention weight corresponding method to described Influence each other the first attention of weight of text feature matrix strengthens eigenmatrix, comprising the physiological data correlation eigen matrix Eigenmatrix is strengthened, comprising the text feature matrix pair to influence each other the second attention of weight of the picture feature matrix The influence each other third attention of weight of the physiological data correlation eigen matrix strengthens eigenmatrix, comprising the text feature Matrix strengthens eigenmatrix, comprising the picture feature square to influence each other the 4th attention of weight of the picture feature matrix Battle array strengthens eigenmatrix and comprising the figure to influence each other the 5th attention of weight of the physiological data correlation eigen matrix Piece eigenmatrix strengthens eigenmatrix to influence each other the 6th attention of weight of the text feature matrix, is then based on described First attention strengthens eigenmatrix, second attention strengthens eigenmatrix, the third attention strengthens eigenmatrix, 4th attention strengthens eigenmatrix, the 5th attention strengthens eigenmatrix and the 6th attention strengthens feature Matrix is obtained text fusion feature matrix, picture fusion feature matrix and physiological data and is melted based on the full Connection Neural Network that feedovers Close eigenmatrix；Then the physiological data correlation eigen matrix, the text feature matrix and the picture feature square are based on Battle array obtains text, picture, physiological data characteristic value based on the full Connection Neural Network of feedovering, be then based on the text, picture, Physiological data characteristic value obtains the weights of importance value of text, picture, physiological data based on vector splicing and attention mechanism； Then based on the text, picture, the weights of importance value of physiological data and the text fusion feature matrix, the picture Fusion feature matrix and the physiological data fusion feature matrix, obtain the fusion representing matrix of three kinds of mode；Finally it is based on institute State three kinds of mode fusion representing matrix and feedforward fully-connected network, obtain reflection psychological pressure problem pressure classify to Amount.The embodiment of the present invention is not only compensated for by fusing text image data and physiology related data by text and image data Subjectivity bring it is insufficient, solve the problems, such as physiology related data is some intrinsic (such as in the state of being on wires and extreme The physiology related data of pressure state is very similar), certain shortage of data is also compensated for a certain extent and is generated Psychology detects the empty window phase.

Detailed description of the invention

In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is this hair Bright some embodiments for those of ordinary skill in the art without creative efforts, can be with root Other attached drawings are obtained according to these attached drawings.

Fig. 1 is the attention weight pair that both modalities which data are carried out with feature interaction fusion that one embodiment of the invention provides The flow chart of induction method；

Fig. 2 is the attention weight pair that both modalities which data are carried out with feature interaction fusion that one embodiment of the invention provides The model structure of induction method；

Fig. 3 is the flow chart for the multi-modal fusion method for psychological pressure detection that one embodiment of the invention provides；

Fig. 4 is the Text character extraction process schematic that one embodiment of the invention provides；

Fig. 5 is the physiological characteristic extraction process schematic diagram that one embodiment of the invention provides；

Fig. 6 is one embodiment of the invention offer to text, picture and the multi-modal detection psychological pressure of physiology related data The model structure of the fusion method of problem；

Fig. 7 is the structural representation for the multi-modal fusion device for psychological pressure detection that one embodiment of the invention provides Figure；

Fig. 8 is the attention weight pair that both modalities which data are carried out with feature interaction fusion that one embodiment of the invention provides Answer the structural schematic diagram of device；

Fig. 9 is the structural schematic diagram for the electronic equipment that one embodiment of the invention provides.

Specific embodiment

In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is A part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art Every other embodiment obtained without creative efforts, shall fall within the protection scope of the present invention.

Before introducing scheme provided in an embodiment of the present invention, first generation origin of the invention is briefly described.When When teenager is by psychological pressure, and sleep quality (difficulty falling asleep, early awakening etc.) is usually for daily routines amount (such as note step number) It will appear some exceptions.On the other hand, literal expression and picture expression can largely show teen-age psychological condition and Daily routines.The embodiment of the present invention is intended to through fusing text, image data and physiology related data, teen-age to detect Psychological pressure.Since the embodiment of the present invention needs to solve the problems, such as multi-modal fusion, the embodiment of the present invention be first proposed One attention weight corresponding method for making both modalities which data be able to carry out feature interaction fusion.In order to text, picture, Adolescent psychology pressure is detected in physiology related data, the embodiment of the present invention is proposed to text, picture and physiology related data The fusion method of multi-modal detection psychological pressure problem, above-mentioned attention weight corresponding method is applied on mode two-by-two, It is assigned again by weight, finally realizes psychological pressure detection.It below will be by specific embodiment to provided in an embodiment of the present invention The attention weight corresponding method of feature interaction fusion is carried out to both modalities which data and for the multi-modal of psychological pressure detection Fusion method and device are described in detail.

Fig. 1 shows the attention weight provided in an embodiment of the present invention that both modalities which data are carried out with feature interaction fusion The flow chart of corresponding method.As shown in Figure 1, provided in an embodiment of the present invention carry out feature interaction fusion to both modalities which data Attention weight corresponding method includes the following steps:

Step 101: the eigenmatrix based on both modalities which data obtains reflection both modalities which data using matrix multiplication The incidence relation matrix of information relevance between different characteristic.

In the present embodiment, modal data refers to that the text data for detecting adolescent psychology pressure (such as investigate by text Questionnaire, diary, caprice, composition etc.), image data for detecting adolescent psychology pressure (such as picture questionnaire, favorite Caricature, conveniently scribble etc.) or physiology related data for detecting adolescent psychology pressure, such as motion conditions and sleep quality Deng.

In the present embodiment, both modalities which data can refer to text data and image data both modalities data, can also With refer to text data and physiology related data both modalities data, can also refer to image data and physiology related data both Modal data.

Step 102: based on the incidence relation matrix and feedforward fully-connected network model, obtaining one of modal data Eigenmatrix to the influence power weight matrix of the eigenmatrix of another modal data.

Step 103: the eigenmatrix based on the influence power weight matrix and described two modal datas utilizes matrix dot Multiply and connected with residual error, obtain comprising described two modal datas eigenmatrix influence each other weight attention strengthen feature square Battle array.

In the present embodiment, main purpose is, to obtain the incidence relation between the eigenmatrix of both modalities which data, And this incidence relation is corresponded to back in former eigenmatrix, so that the eigenmatrix of treated each mode includes another One mode influences information to its incidence relation.As shown in Fig. 2, specific as follows:

Assuming that the eigenmatrix of two modal datas is A and B, whereinUsing matrix multiplication by A Turn order matrix multiple with B, to obtain comprising the incidence relation matrix in each feature in A and B between each feature

It, will by one layer of fully-connected networkIt maps backVector space, obtain A_B→A,

A_B→AIndicate influence power weight of the mode B for mode A, W₁Indicate the first default training parameter.

Using dot product operation come by A_B→AIt is multiplied with A, and the attention obtained after residual error connection strengthens eigenmatrix The influence that the information and B for containing B generate A.

With it is upper similarly, available attention strengthens eigenmatrix The shadow that the information and A for containing A generate B It rings:

This method is called in order to facilitate Examples hereinafter, uses f_AMMIndicate this method, i.e.,

It should be noted that the both modalities which data in the present embodiment can refer to the text data of detection psychological pressure With image data both modalities data, also can refer to detection psychological pressure text data and physiology related data this two It plants modal data, can also refer to the image data and physiology related data both modalities data for detecting psychological pressure, this Inventive embodiments are realized by treatment process above and carry out feature friendship to text data and image data both modalities data The attention weight corresponding method mutually merged realizes and carries out spy to text data and physiology related data both modalities data The attention weight corresponding method of sign interaction fusion, realize to image data and physiology related data both modalities data into The attention weight corresponding method namely the present embodiment of row feature interaction fusion make that treated, and text data includes picture Data and physiology related data influence its incidence relation, so that treated, image data includes text data and physiology Related data influences its incidence relation, so that treated, physiology related data includes text data and image data pair Its incidence relation influences.Namely the eigenmatrix of the present embodiment each mode that makes that treated includes another mode Information is influenced on its incidence relation and obtains multi-modal characteristic consequently facilitating multi-modal characteristic is merged The result of combined influence.It is subsequent that embodiment is based on this method, a kind of multi-modal fusion side for psychological pressure detection is provided The text data, image data and physiology related data that are used for psychological pressure detection are carried out multi-modal fusion by method and device, from And the subjectivity bring deficiency by text and image data can be not only made up, but also can solve physiology related data Some intrinsic problems (such as in the physiology related data of the state of being on wires and extreme pressure state be very similar), this Outside, the psychology for also compensating for certain shortage of data to a certain extent and generating detects the empty window phase.

By prior art scheme it is found that provided in an embodiment of the present invention carry out feature interaction fusion to both modalities which data Attention weight corresponding method, the purpose is to obtain the incidence relation between both modalities which eigenmatrix, and by this pass Connection relationship corresponds to back in former eigenmatrix, so that the eigenmatrix of treated each mode includes another mode to it Incidence relation influence information, use following processing means: the eigenmatrix based on both modalities which data utilizes Matrix Multiplication Method is obtained the incidence relation matrix of information relevance between reflection both modalities which data different characteristic, and is closed based on the association It is matrix and feedforward fully-connected network model, obtains spy of the eigenmatrix to another modal data of one of modal data The influence power weight matrix of matrix is levied, finally the feature square based on the influence power weight matrix and described two modal datas Battle array, connect using matrix dot product with residual error, and obtaining includes that the eigenmatrixes of described two modal datas influences each other the note of weight Power of anticipating strengthens eigenmatrix, and the embodiment of the present invention is realized by treatment process above and carries out feature interaction to both modalities which data The attention weight corresponding method of fusion, both modalities which data here can refer to text data and image data both modalities Data can also refer to text data and physiology related data both modalities data, can also refer to that image data is related to physiology Data both modalities data, the embodiment of the present invention by treatment process above realize to text data and image data this two Kind modal data carries out the attention weight corresponding method of feature interaction fusion, realizes to text data and physiology related data Both modalities data carry out the attention weight corresponding method of feature interaction fusion, realize related to physiology to image data After the attention weight corresponding method namely the present embodiment of data both modalities data progress feature interaction fusion make processing Text data to include image data and physiology related data influence its incidence relation so that treated image data It include that text data and physiology related data influence its incidence relation, so that treated, physiology related data includes Text data and image data influence its incidence relation.Namely the feature of the present embodiment each mode that makes that treated Matrix all includes that another mode influences information to its incidence relation, consequently facilitating multi-modal characteristic is melted It closes, obtains the result of multi-modal characteristic combined influence.It is subsequent that embodiment is based on this method, it provides a kind of for psychology pressure The multi-modal fusion method and device of power detection is related to physiology by the text data, the image data that are used for psychological pressure detection Data carry out multi-modal fusion, so that the subjectivity bring deficiency by text and image data can be not only made up, but also The some intrinsic problems that can solve physiology related data are (such as related to the physiology of extreme pressure state in the state of being on wires Data are very similar), in addition, the psychology for also compensating for certain shortage of data to a certain extent and generating detects empty window Phase.

Further, content based on the above embodiment, in the present embodiment, above-mentioned steps 101 can be in the following way It realizes:

The association of information relevance between reflection both modalities which data different characteristic is obtained using following first relational model Relational matrix:

Wherein,Indicate that incidence relation matrix, A indicate that the eigenmatrix of one of modal data, B indicate another The eigenmatrix of kind modal data, Indicate that real number space, k indicate the dimension of the both modalities which data Degree, B^TIt indicates that B's turns order matrix, eigenmatrix A and eigenmatrix B is turned into order matrix multiple using matrix multiplication, is wrapped Incidence relation matrix in A containing eigenmatrix in each feature and eigenmatrix B between each feature

Further, content based on the above embodiment, in the present embodiment, above-mentioned steps 102 can be in the following way It realizes:

Eigenmatrix B is obtained to the influence power weight matrix of eigenmatrix A using following second relational model:

And eigenmatrix A is obtained to the influence power weight matrix of eigenmatrix B using following third relational model:

Wherein, A_B→AIndicate influence power weight matrix of the eigenmatrix B to eigenmatrix A, B_A→BIndicate A pairs of eigenmatrix The influence power weight matrix of eigenmatrix B,Softmax indicates normalization exponential function, W₁ Indicate the first default training parameter in first kind training parameter, W₂

Indicate that the second default training parameter in first kind training parameter will be associated with by one layer of fully-connected network Relational matrixIt maps backVector space, obtain eigenmatrix B to the influence power weight matrix A of eigenmatrix A_B→A With eigenmatrix A to the influence power weight matrix B of eigenmatrix B_A→B。

Further, content based on the above embodiment, in the present embodiment, above-mentioned steps 103 can be in the following way It realizes:

Attention, which is obtained, using following 4th relational model strengthens eigenmatrix

And attention is obtained using following 5th relational model and strengthens eigenmatrix

Wherein, ⊙ indicates dot product operation, using dot product operation come by A_B→AIt is multiplied with A, and obtains the attention after residual error connection Power strengthens eigenmatrix In include the information of B and the influence that B generates A；Using dot product operation come by B_A→BWith B phase Multiply, and the attention obtained after residual error connection strengthens eigenmatrix In include the information of A and the shadow that A generates B It rings；

Wherein,f_AMMIt indicates to strengthen feature by eigenmatrix A and eigenmatrix B to attention MatrixStrengthen eigenmatrix with attentionTreatment process, specifically include: utilizing first relational model to described the Five relational models handle to eigenmatrix A and eigenmatrix B the power reinforcing eigenmatrix that gains attentionStrengthen with attention EigenmatrixTreatment process.

Fig. 3 shows the flow chart of the multi-modal fusion method provided in an embodiment of the present invention for psychological pressure detection. As shown in figure 3, the multi-modal fusion method institute based on the above embodiment provided in an embodiment of the present invention for psychological pressure detection The attention weight corresponding method for carrying out feature interaction fusion to both modalities which data stated is realized, provided in an embodiment of the present invention Multi-modal fusion method for psychological pressure detection includes the following steps:

Step 201: obtaining the physiological data correlation eigen matrix and reflection user's heart of reflection user's physiological status respectively Manage the text feature matrix and picture feature matrix of active state.

In this step, need to obtain the eigenmatrix of text, picture and physiology related data.For text feature matrix Acquisition process, reference can be made to acquisition process schematic diagram shown in Fig. 4.For the acquisition process of physiological data correlation eigen matrix, It can be found in acquisition process schematic diagram shown in fig. 5.Obtaining to the eigenmatrix of text, picture and physiology related data separately below It takes process to give to be discussed in detail.

1. each text is indicated with w, w={ w for text₁,w₂,···,w_n,w_iIndicate one Word.For example, initial word of the vector for 300 dimensions for selecting the pre-training of Chinese Word Vectors good as each word Text representation is thus X={ x by vector₁,x₂,···,x_n,x_iFor the 1*300 for indicating word meaning Vector.

The purpose of LSTM (Long Short-Term Memory, shot and long term memory network) network layer be calculate one can be with The text representation for expressing contextual information, because model cannot directly understand natural language, it is necessary to first calculate one The text representation that model is understood that, this text representation are specifically matrix form H.By text representation X={ x₁,x₂,···, x_nAs inputting into LSTM layers, wherein n indicates the quantity for the word for including in the text vocabulary, and n is in the present invention Take 20.The hidden layer output of two LSTM is respectively obtained by positive LSTM and reversed LSTMWithThe hidden layer of corresponding position is exported and is added, text representation matrix H is obtained:

The contribution degree distribution of weights of text representation matrix H is obtained using attention mechanism:

Attn_T=softmax (HW₃+b₁)

Wherein, Attn_TIt is contribution degree distribution of weights vector, indicates point of the contribution weight of the text representation of each word Cloth.By Attn_TIt is multiplied with H, and is connected by residual error, obtained the text representation matrix readjusted by weight distribution

It, will by one layer of fully-connected networkIt is mapped to the vector space of k × 1, has obtained text feature matrix F_T:

2. the image of boil down to 32*32 pixel can be unified every picture for picture, it is few in picture number in this way In the case of can be with the acquisition of speeding up picture feature.Due to being color image, so port number is 3, each picture is with 32*32*3's Vector indicates, obtains the feature vector of 4*4*512 by first three part-structure of the ResNet network of pre-training, next logical It crosses one layer of convolutional layer and obtains the feature vector of 4*4*32, the input of convolutional layer is the feature vector of 4*4*512, and convolution kernel size is 1*1 obtains the feature vector of 4*4*32 after convolution.It is the vector C that a length is 512 by the image spread of 4*4*32, indicates Preliminary characteristics of image C.Next the vector space of the dimension map of characteristics of image to n × 1 is obtained with a full articulamentum Picture feature matrix F_V:

F_V=ReLU (W₅C+b₃)。

3. for physiology related data relevant dormant data and exercise data can be acquired by bracelet, to dormant data Feature extraction is carried out with exercise data, relevant sleep characteristics vector sum motion feature vector is obtained and is stitched together as physiology phase Close data characteristics vector.For example, it is contemplated that teen-age daily schedule rule, for evening 8:00 to the next morning 10:00 Sleep quality.Be extracted 9 features, be respectively: sleep starts segment, sleep end fragment, sleep segment, deep sleep piece Section, deep sleep accounting, sleep total amount, unit-segment amount of sleep, sleep undulate quantity, number of regaining consciousness in sleep.In order to facilitate the time The metering of feature was used as a segment for every 15 minutes, such as 20:00-20:15 is segment 1, and 20:15-20:30 is segment 2, And so on, set of segments is indicated with T,

T={ t₁,t₂,···,t₅₆, t_i∈ T indicates the amount of sleep of i-th of segment.

Sleep starts segment: the continuous dormant data for generating at least continuous 4 segments earliest in sleep interval is all larger than 0 Start Fragment, as sleep start segment, that is, work as t_i*t_i+1*t_i+2*t_i+3When > 0, t_i,t_i+1,t_i+2,t_i+3∈ T, sleep start Segment is taken as the segment of the minimum value in i.

Sleep the end time: in sleep interval at least four continuously sleep segment amount of sleep be all larger than 0 piece the latest Section, i.e. t_i*t_i-1*t_i-2*t_i-3> 0 andt_i,t_i-1,t_i-2,t_i-3∈T。

Sleep segment: amount of sleep is greater than 0 segments in sleep metering section.

Deep sleep segment: when amount of sleep is higher than threshold θ in segment, for a deep sleep segment, general θ value 230, the threshold Value is bracelet parameter, and according to different bracelets, value is variable.

Deep sleep accounting: the ratio of deep sleep segment and sleep segment.

Sleep total amount: sleep starts segment to the sum of the amount of sleep between sleep end fragment.

Unit-segment amount of sleep: the ratio of sleep total amount and segment of sleeping is unit-segment amount of sleep.

Sleep undulate quantity: sleep starts segment to the standard deviation of the amount of sleep between sleep end fragment as fluctuation of sleeping Amount.

Awake number in sleep: sleep starts the segments that segment is less than threshold value beta to amount of sleep between sleep end fragment, β value 25, when sleep starts segment, between sleep end fragment, amount of sleep is less than 25, i.e., expression is waken up.

About motion feature vector, 5 motion features are extracted, are every daily motion step number, the consumption of daily calorie respectively Value, daily move distance, every daily motion duration, every daily motion enliven duration.Wherein every daily motion step number, the consumption of daily calorie Value, daily move distance, every daily motion duration can be directly obtained by bracelet.Every daily motion enlivens duration: by daily 24 A hourly average is divided into 96 segments, and it is high that step number, calorie consumption value, move distance, movement duration are moved in each segment It is that movement enlivens segment in the segment of its respective items mean value, every daily motion enlivens the sum of segment when being that every daily motion enlivens It is long.9 sleep characteristics and 5 motion features are stitched together, the physiology related data feature E of 14*1 is become_S。

Obtaining a better physiology related data by two layers of fully-connected network indicates matrix E:

E=ReLU (W₇(ReLU(W₆E_S+b₄)+b₅))

Obtaining physiology related data using attention mechanism indicates the contribution degree distribution of weights of matrix E:

Attn_E=softmax (W₈E+b₆)

Wherein, Attn_EIt is contribution degree distribution of weights vector, indicates point for the contribution weight that each physiological characteristic indicates Cloth.By Attn_EIt is multiplied with E, and is connected by residual error, obtained the text representation matrix readjusted by weight distribution

It, will by one layer of fully-connected networkIt is mapped to the vector space of k × 1, has obtained physiological data correlated characteristic square Battle array F_E:

It should be noted that above-mentioned example is a signal, the physiology related data is not limited to the sleep Data and the exercise data can also be blood pressure, pulse, galvanic skin reaction, electrocardiogram, electromyogram according to actual needs Etc. data, this is not limited by the present invention.

Step 202: being based on the physiological data correlation eigen matrix, the text feature matrix and the picture feature square Battle array is obtained comprising the physiological data correlation eigen matrix using the attention weight corresponding method to the text feature Matrix influence each other weight the first attention strengthen eigenmatrix, comprising the physiological data correlation eigen matrix to the figure Piece eigenmatrix influence each other weight the second attention strengthen eigenmatrix, comprising the text feature matrix to the physiology Data correlation eigen matrix influence each other weight third attention strengthen eigenmatrix, comprising the text feature matrix to institute State picture feature matrix influence each other weight the 4th attention strengthen eigenmatrix, comprising the picture feature matrix to described Influence each other the 5th attention of weight of physiological data correlation eigen matrix strengthens eigenmatrix and comprising the picture feature square Battle array to the text feature matrix influence each other weight the 6th attention strengthen eigenmatrix.

In this step, attention weight corresponding method is applied, all between feature vector two-by-two come the power that gains attention Strengthen eigenmatrix

Wherein, it is that physiological data -> text attention strengthens eigenmatrix that first attention, which strengthens eigenmatrix,It is that physiological data -> picture attention strengthens eigenmatrix that second attention, which strengthens eigenmatrix,Institute Stating third attention to strengthen eigenmatrix is that text -> physiological data attention strengthens eigenmatrix4th note It is that text -> picture attention strengthens eigenmatrix that power of anticipating, which strengthens eigenmatrix,5th attention strengthens feature Matrix is that picture -> physiological data attention strengthens eigenmatrixIt is figure that 6th attention, which strengthens eigenmatrix, Piece -> text attention strengthens eigenmatrixIn this way for each mode, it is strong two attentions have all been obtained Change eigenmatrix, all contains the related information of other both modalities which.

Step 203: eigenmatrix being strengthened based on first attention, second attention strengthens eigenmatrix, institute It states third attention and strengthens eigenmatrix, the 4th attention reinforcing eigenmatrix, the 5th attention reinforcing feature square Battle array and the 6th attention strengthen eigenmatrix, based on the full Connection Neural Network that feedovers, obtain text fusion feature matrix, figure Piece fusion feature matrix and physiological data fusion feature matrix.

In this step, further by one layer of fully-connected network, two attentions of each mode are strengthened into feature square Battle array merges into a fusion feature matrixEach fusion feature matrix includes the association of other both modalities which With influence force information:

Step 204: being based on the physiological data correlation eigen matrix, the text feature matrix and the picture feature square Battle array obtains text, picture, physiological data characteristic value based on the full Connection Neural Network that feedovers.

In this step, text, picture, physiology correlation eigen matrix are mapped between (0,1), it is then complete by one layer Connection obtains the relevant characteristic value S of text, picture, physiology_T, S_VAnd S_E:

S_T=ReLU (W₁₆softmax(F_T)+b₁₁)

S_V=ReLU (W₁₇softmax(F_V)+b₁₂)

S_E=ReLU (W₁₈softmax(F_E)+b₁₃)

Step 205: being based on the text, picture, physiological data characteristic value, based on vector splicing and attention mechanism, obtain Take the weights of importance value of text, picture, physiological data.

In this step, by the text, picture, physiological data characteristic value S_T, S_VAnd S_EIt is spliced together, passes through attention Power mechanism obtains the weights of importance value weight of text, picture, physiological data_T, weight_VAnd weight_E:

(weight_T,weight_V,weight_E)=softmax ([S_T,S_V,S_E]W₁₉)

Step 206: based on the text, picture, the weights of importance value of physiological data and the text fusion feature Matrix, the picture fusion feature matrix and the physiological data fusion feature matrix, the fusion for obtaining three kinds of mode indicate square Battle array.

In this step, three kinds of mode refer to text data mode, image data mode and physiology related data mode.This Step by the text, picture, physiological data weights of importance value weight_T, weight_VAnd weight_EMelt with the text Close eigenmatrixThe picture fusion feature matrixWith the physiological data fusion feature matrixThe corresponding phase again that is multiplied Add, obtains the fusion representing matrix R of three kinds of mode_W:

Step 207: fusion representing matrix and feedforward fully-connected network based on three kinds of mode obtain reflection psychology The pressure class vector of stress problems.

In this step, obtain indicating whether or not there is by a linear classifier pressure of the 1*2 of psychological pressure classify to Y is measured, two dimensions have respectively represented pressure and no pressure, wherein possessing meaning corresponding to the position of highest numerical value for conduct Final classification results.For example, specifically pressure class vector y can be obtained by such as drag:

Y=softmax (W₂₃R_W+b₁₄)

Wherein, W₁~W₂₃Indicate the first to the 23rd default training parameter in first kind training parameter, b₁~b₁₄Table Show the first to the 14th default training parameter in the second class training parameter.Wherein, the first kind training parameter and described Two class training parameters conform to normal distribution U (- 0.001,0.001), first to the 23rd in first kind training parameter The first to the 14th default training parameter in default training parameter and the second class training parameter is configured according to actual needs.

Fusion side to text, picture and the multi-modal detection psychological pressure problem of physiology related data shown in Figure 6 The model structure of method is it is found that the multi-modal fusion method provided in an embodiment of the present invention for psychological pressure detection, is based on institute Physiological data correlation eigen matrix, the text feature matrix and the picture feature matrix are stated, the attention weight is utilized Corresponding method obtains and influences each other the first of weight to the text feature matrix comprising the physiological data correlation eigen matrix Attention strengthens eigenmatrix, influences each other weight comprising the physiological data correlation eigen matrix to the picture feature matrix The second attention strengthen eigenmatrix, comprising the text feature matrix to the mutual shadow of physiological data correlation eigen matrix The third attention for ringing weight strengthens eigenmatrix, influences each other comprising the text feature matrix to the picture feature matrix 4th attention of weight strengthens eigenmatrix, includes the picture feature matrix to the physiological data correlation eigen matrix phase 5th attention of mutual weighing factor strengthens eigenmatrix and comprising the picture feature matrix to the text feature matrix phase 6th attention of mutual weighing factor strengthens eigenmatrix, is then based on first attention and strengthens eigenmatrix, described the Two attentions strengthen eigenmatrix, the third attention strengthens eigenmatrix, the 4th attention strengthens eigenmatrix, institute It states the 5th attention and strengthens eigenmatrix and the 6th attention reinforcing eigenmatrix, be based on the full Connection Neural Network that feedovers, Obtain text fusion feature matrix, picture fusion feature matrix and physiological data fusion feature matrix；Then it is based on the physiology Data correlation eigen matrix, the text feature matrix and the picture feature matrix are obtained based on the full Connection Neural Network that feedovers To text, picture, physiological data characteristic value, it is then based on the text, picture, physiological data characteristic value, is spliced based on vector With attention mechanism, the weights of importance value of text, picture, physiological data is obtained；Then the text, picture, physiology are based on The weights of importance value of data and the text fusion feature matrix, the picture fusion feature matrix and the physiological data Fusion feature matrix obtains the fusion representing matrix of three kinds of mode；Fusion representing matrix finally based on three kinds of mode with And feedforward fully-connected network, obtain the pressure class vector of reflection psychological pressure problem.The embodiment of the present invention passes through fusing text Image data and physiology related data not only compensate for the subjectivity bring deficiency by text and image data, solve life The some intrinsic problems for managing related data (for example in the physiology related data of the state of being on wires and extreme pressure state are very It is similar), the psychology for also compensating for certain shortage of data to a certain extent and generating detects the empty window phase.

Further, content based on the above embodiment, in the present embodiment, above-mentioned steps 201 can pass through such as lower section Formula is realized:

The text feature matrix of reflection user psychology active state is obtained using following 6th processing model:

Attn_T=softmax (HW₃+b₁)

Wherein, F_TIndicate that text feature matrix, H indicate text representation matrix,It indicates to readjust by weight distribution Text representation matrix, Attn_TThe contribution degree distribution of weights vector for indicating text representation matrix H, by text representation X={ x₁, x₂,···,x_nIt is used as input to enter shot and long term memory network

It LSTM layers, is exported by the hidden layer that positive LSTM and reversed LSTM respectively obtain two LSTMWithThe hidden layer of corresponding position is exported and is added, text representation matrix H is obtained；Text is obtained using attention mechanism The contribution degree distribution of weights vector Attn of this representing matrix H_T: Attn_T=softmax (HW₃+b₁), Attn_TIndicate each word Text representation contribution weight distribution, by Attn_TIt is multiplied with H, and is connected by residual error, obtained by weight distribution weight The text representation matrix newly adjusted It, will by one layer of fully-connected networkBe mapped to k × 1 to Quantity space has obtained text feature matrix F_T:Wherein, W₃It indicates in first kind training parameter Third preset training parameter, W₄Indicate the 4th default training parameter in first kind training parameter, b₁Indicate the second class training ginseng The first default training parameter in number, b₂Indicate that the second default training parameter in the second class training parameter, ReLU indicate activation Function；Softmax indicates normalization exponential function, text representationx_iIndicate word meaning Vector, n indicate text in include word quantity；

And the picture feature matrix of reflection user psychology active state is obtained using following 7th processing model:

F_V=ReLU (W₅C+b₃)

Wherein, F_VIndicate that picture feature matrix, C indicate picture feature, with a full articulamentum by the dimension of picture feature C It is mapped to the vector space of n × 1, obtains picture feature matrix F_V；Wherein, W₅Indicate that the in first kind training parameter the 5th presets Training parameter, b₃Indicate that the third in the second class training parameter presets training parameter；

And the physiological data correlation eigen matrix of reflection user's physiological status is obtained using following 8th processing model:

E=ReLU (W₇(ReLU(W₆E_S+b₄)+b₅))

Attn_E=softmax (W₈E+b₆)

Wherein, F_EIndicate physiological data correlation eigen matrix, E_SIndicate physiology related data eigenmatrix, E_SThe inside includes There are multiple preset physiological characteristics, E is indicated to E_SCarrying out the physiology related data that two layers of fully-connected network obtains indicates matrix E, Attn_EIndicate that physiology related data indicates the contribution degree distribution of weights vector of matrix E,It indicates to readjust by weight distribution Text representation matrixTo E_SCarrying out the physiology related data that two layers of fully-connected network obtains indicates matrix E:E=ReLU (W₇ (ReLU(W₆E_S+b₄)+b₅)), using attention mechanism obtain physiology related data indicate matrix E contribution degree distribution of weights to Measure Attn_E: Attn_E=softmax (W₈E+b₆), by Attn_EIt is multiplied with E, and is connected by residual error, obtained by weight distribution The text representation matrix of readjustment It, will by one layer of fully-connected networkBe mapped to k × 1 to Quantity space has obtained physiological data correlation eigen matrix F_E:Wherein, Attn_EIndicate each The distribution for the contribution weight that physiological characteristic indicates, W₆Indicate the 6th default training parameter in first kind training parameter, W₇Indicate the The 7th default training parameter in a kind of training parameter, W₈Indicate the 8th default training parameter in first kind training parameter, W₉Table Show the 9th default training parameter in first kind training parameter, b₄Indicate the 4th default training ginseng in the second class training parameter Number, b₅Indicate the 5th default training parameter in the second class training parameter, b₆Indicate the 6th default instruction in the second class training parameter Practice parameter, b₇Indicate the 7th default training parameter in the second class training parameter.

Further, content based on the above embodiment, in the present embodiment, above-mentioned steps 202 can pass through such as lower section Formula is realized:

Based on the physiological data correlation eigen matrix and the text feature matrix, using described in above-described embodiment pairs Both modalities which data carry out the attention weight corresponding method of feature interaction fusion, and obtaining includes the physiological data correlated characteristic Matrix to the text feature matrix influence each other weight the first attention strengthen eigenmatrix；First attention is strengthened Eigenmatrix is that physiological data -> text attention strengthens eigenmatrix

Based on the physiological data correlation eigen matrix and the picture feature matrix, using described in above-described embodiment pairs Both modalities which data carry out the attention weight corresponding method of feature interaction fusion, and obtaining includes the physiological data correlated characteristic Matrix to the picture feature matrix influence each other weight the second attention strengthen eigenmatrix；Second attention is strengthened Eigenmatrix is that physiological data -> picture attention strengthens eigenmatrix

Based on the physiological data correlation eigen matrix and the text feature matrix, using described in above-described embodiment pairs Both modalities which data carry out the attention weight corresponding method of feature interaction fusion, obtain comprising the text feature matrix to institute State physiological data correlation eigen matrix influence each other weight third attention strengthen eigenmatrix；The third attention is strengthened Eigenmatrix is that text -> physiological data attention strengthens eigenmatrix

Based on the text feature matrix and the picture feature matrix, using described in above-described embodiment to both modalities which Data carry out the attention weight corresponding method of feature interaction fusion, obtain special to the picture comprising the text feature matrix Sign matrix influence each other weight the 4th attention strengthen eigenmatrix；4th attention strengthen eigenmatrix be text -> The attention of picture strengthens eigenmatrix

Based on the physiological data correlation eigen matrix and the picture feature matrix, using described in above-described embodiment pairs Both modalities which data carry out the attention weight corresponding method of feature interaction fusion, obtain comprising the picture feature matrix to institute State physiological data correlation eigen matrix influence each other weight the 5th attention strengthen eigenmatrix；5th attention is strengthened Eigenmatrix is that picture -> physiological data attention strengthens eigenmatrix

Based on the text feature matrix and the picture feature matrix, using described in above-described embodiment to both modalities which Data carry out the attention weight corresponding method of feature interaction fusion, obtain special to the text comprising the picture feature matrix Sign matrix influence each other weight the 6th attention strengthen eigenmatrix；6th attention strengthen eigenmatrix be picture -> The attention of text strengthens eigenmatrix

Further, content based on the above embodiment, in the present embodiment, above-mentioned steps 203 can pass through such as lower section Formula is realized:

Using following 9th processing model, eigenmatrix is strengthened based on first attention and the 6th attention is strong Change eigenmatrix, by one layer of fully-connected network, obtains text fusion feature matrix

Using following tenth processing model, eigenmatrix is strengthened based on second attention and the 4th attention is strong Change eigenmatrix, by one layer of fully-connected network, obtains picture fusion feature matrix

Using following 11st processing model, eigenmatrix and the 5th attention are strengthened based on the third attention Strengthen eigenmatrix, by one layer of fully-connected network, obtains physiological data fusion feature matrix

Wherein, W₁₀~W₁₅Indicate the tenth to the 15th default training parameter in first kind training parameter, b₈~b₁₀It indicates The the 8th to the tenth default training parameter in second class training parameter.

Further, content based on the above embodiment, in the present embodiment, above-mentioned steps 204 can pass through such as lower section Formula is realized:

Text, picture, physiological data characteristic value are obtained using following 12nd processing model:

S_T=ReLU (W₁₆softmax(F_T)+b₁₁)

S_V=ReLU (W₁₇softmax(F_V)+b₁₂)

S_E=ReLU (W₁₈softmax(F_E)+b₁₃)

Wherein, by the physiological data correlation eigen matrix F_E, the text feature matrix F_TWith the picture feature square F_V Battle array is mapped between (0,1), then obtains text, picture, physiological data characteristic value S by one layer of full connection_T, S_VAnd S_E；Its In, W₁₆~W₁₈Indicate the 16th default training parameter to the 18th default training parameter in first kind training parameter, b₁₁~b₁₃ Indicate the 11st to the 13rd default training parameter in the second class training parameter.

Further, content based on the above embodiment, in the present embodiment, above-mentioned steps 205 can pass through such as lower section Formula is realized:

Using following 13rd processing model, by the text, picture, physiological data characteristic value S_T, S_VAnd S_EIt is spliced to one It rises, by attention mechanism, obtains the weights of importance value weight of text, picture, physiological data_T, weight_VWith weight_E:

(weight_T,weight_V,weight_E)=softmax ([S_T,S_V,S_E]W₁₉)

Wherein, W₁₁₉Indicate the 19th default training parameter in first kind training parameter.

Further, content based on the above embodiment, in the present embodiment, above-mentioned steps 206 can pass through such as lower section Formula is realized:

Using it is following 14th processing model, by the text, picture, physiological data weights of importance value weight_T, weight_VAnd weight_EWith the text fusion feature matrixThe picture fusion feature matrixWith the physiology number According to fusion feature matrixCorresponding be multiplied is added again, obtains the fusion representing matrix R of three kinds of mode_W:

Wherein, W₂₀~W₂₂Indicate the 20th default training parameter to the 22nd default instruction in first kind training parameter Practice parameter.

Fig. 7 shows the attention weight provided in an embodiment of the present invention that both modalities which data are carried out with feature interaction fusion The structural schematic diagram of corresponding intrument.As shown in fig. 7, provided in an embodiment of the present invention melt both modalities which data progress feature interaction The attention weight corresponding intrument of conjunction includes: that the first acquisition module 11, second obtains module 12 and third acquisition module 13, In:

First obtains module 11, obtains reflection two using matrix multiplication for the eigenmatrix based on both modalities which data The incidence relation matrix of information relevance between kind modal data different characteristic；

Second obtains module 12, for obtaining wherein based on the incidence relation matrix and feedforward fully-connected network model A kind of influence power weight matrix of the eigenmatrix of modal data to the eigenmatrix of another modal data；

Third obtains module 13, for the feature square based on the influence power weight matrix and described two modal datas Battle array, connect using matrix dot product with residual error, and obtaining includes that the eigenmatrixes of described two modal datas influences each other the note of weight Power of anticipating strengthens eigenmatrix.

Since the attention weight provided in an embodiment of the present invention for carrying out feature interaction fusion to both modalities which data is corresponding Device can be used for executing the attention weight pair that described in above-described embodiment both modalities which data are carried out with feature interaction fusion Induction method, working principle is similar with beneficial effect, therefore and will not be described here in detail, and particular content can be found in Jie of above-described embodiment It continues.

Fig. 8 shows the structural representation of the multi-modal fusion device provided in an embodiment of the present invention for psychological pressure detection Figure.As shown in figure 8, the multi-modal fusion device provided in an embodiment of the present invention for psychological pressure detection is based on above example The attention weight corresponding intrument for carrying out feature interaction fusion to both modalities which data realizes that the embodiment of the present invention provides For psychological pressure detection multi-modal fusion device, comprising: the 4th acquisition module the 21, the 5th obtain module the 22, the 6th obtains Modulus block the 23, the 7th obtains module the 24, the 8th and obtains the acquisition module 26 of module the 25, the 9th and the tenth acquisition module 27, in which:

4th obtain module 21, for obtain respectively reflection user's physiological status physiological data correlation eigen matrix and Reflect the text feature matrix and picture feature matrix of user psychology active state；

5th obtains module 22, for being based on the physiological data correlation eigen matrix, the text feature matrix and institute Picture feature matrix is stated, using the attention weight corresponding method, obtaining includes the physiological data correlation eigen matrix pair Influence each other the first attention of weight of the text feature matrix strengthens eigenmatrix, comprising the physiological data correlated characteristic Matrix strengthens eigenmatrix, comprising the text feature square to influence each other the second attention of weight of the picture feature matrix Battle array strengthens eigenmatrix, comprising the text to the influence each other third attention of weight of the physiological data correlation eigen matrix Eigenmatrix strengthens eigenmatrix, special comprising the picture to influence each other the 4th attention of weight of the picture feature matrix Sign matrix strengthens eigenmatrix and comprising institute to influence each other the 5th attention of weight of the physiological data correlation eigen matrix State picture feature matrix to the text feature matrix influence each other weight the 6th attention strengthen eigenmatrix；

6th obtains module 23, and for strengthening eigenmatrix based on first attention, second attention is strengthened Eigenmatrix, the third attention strengthen eigenmatrix, the 4th attention strengthens eigenmatrix, the 5th attention Strengthen eigenmatrix and the 6th attention strengthens eigenmatrix, based on the full Connection Neural Network that feedovers, obtains text fusion Eigenmatrix, picture fusion feature matrix and physiological data fusion feature matrix；

7th obtains module 24, for being based on the physiological data correlation eigen matrix, the text feature matrix and institute Picture feature matrix is stated, based on the full Connection Neural Network that feedovers, obtains text, picture, physiological data characteristic value；

8th obtains module 25, for being based on the text, picture, physiological data characteristic value, is spliced based on vector and is infused Meaning power mechanism, obtains the weights of importance value of text, picture, physiological data；

9th obtains module 26, for based on the text, picture, the weights of importance value of physiological data and the text This fusion feature matrix, the picture fusion feature matrix and the physiological data fusion feature matrix, obtain three kinds of mode Merge representing matrix；

Tenth obtain module 27, for based on three kinds of mode fusion representing matrix and feedforward fully-connected network, Obtain the pressure class vector of reflection psychological pressure problem.

Due to the multi-modal fusion device provided in an embodiment of the present invention for psychological pressure detection, can be used in execution The multi-modal fusion method described in embodiment for psychological pressure detection is stated, working principle is similar with beneficial effect, so Place is no longer described in detail, and particular content can be found in the introduction of above-described embodiment.

Based on identical inventive concept, further embodiment of this invention provides a kind of electronic equipment, referring to Fig. 9, the electricity Sub- equipment specifically includes following content: processor 301, memory 302, communication interface 303 and communication bus 304；

Wherein, the processor 301, memory 302, communication interface 303 are completed each other by the communication bus 304 Communication；

The processor 301 is used to call the computer program in the memory 302, and the processor executes the meter The above-mentioned attention weight corresponding method that both modalities which data are carried out with feature interaction fusion is realized when calculation machine program, and/or, it uses In the Overall Steps of the multi-modal fusion method of psychological pressure detection, for example, when the processor executes the computer program Realize following processes:

Reflection both modalities which data different characteristic is obtained using matrix multiplication based on the eigenmatrix of both modalities which data Between information relevance incidence relation matrix；Based on the incidence relation matrix and feedforward fully-connected network model, it is obtained A kind of influence power weight matrix of the eigenmatrix of middle modal data to the eigenmatrix of another modal data；Based on the shadow The eigenmatrix for ringing power weight matrix and described two modal datas, is connected using matrix dot product with residual error, is obtained comprising described The eigenmatrix of both modalities which data influence each other weight attention strengthen eigenmatrix.

For another example, following processes are realized when the processor executes the computer program:

The physiological data correlation eigen matrix and reflection user psychology moving type of reflection user's physiological status are obtained respectively The text feature matrix and picture feature matrix of state；Based on the physiological data correlation eigen matrix, the text feature matrix With the picture feature matrix, using the attention weight corresponding method, obtaining includes the physiological data correlated characteristic square Battle array strengthens eigenmatrix, related comprising the physiological data to influence each other the first attention of weight of the text feature matrix Eigenmatrix strengthens eigenmatrix, special comprising the text to influence each other the second attention of weight of the picture feature matrix Sign matrix strengthens eigenmatrix, comprising described to the influence each other third attention of weight of the physiological data correlation eigen matrix Text feature matrix strengthens eigenmatrix, comprising the figure to influence each other the 4th attention of weight of the picture feature matrix Piece eigenmatrix strengthens eigenmatrix and packet to influence each other the 5th attention of weight of the physiological data correlation eigen matrix Containing the picture feature matrix to the text feature matrix influence each other weight the 6th attention strengthen eigenmatrix；It is based on First attention strengthens eigenmatrix, second attention strengthens eigenmatrix, the third attention strengthens feature Matrix, the 4th attention strengthen eigenmatrix, the 5th attention strengthens eigenmatrix and the 6th attention is strong Change eigenmatrix, based on the full Connection Neural Network that feedovers, obtains text fusion feature matrix, picture fusion feature matrix and physiology Data fusion eigenmatrix；Based on the physiological data correlation eigen matrix, the text feature matrix and the picture feature Matrix obtains text, picture, physiological data characteristic value based on the full Connection Neural Network that feedovers；Based on the text, picture, life Data feature values are managed, based on vector splicing and attention mechanism, obtain the weights of importance value of text, picture, physiological data；Base It is special in the text, picture, the weights of importance value of physiological data and the text fusion feature matrix, picture fusion Matrix and the physiological data fusion feature matrix are levied, the fusion representing matrix of three kinds of mode is obtained；Based on three kinds of mode Fusion representing matrix and feedforward fully-connected network, obtain reflection psychological pressure problem pressure class vector.

Based on identical inventive concept, further embodiment of this invention provides a kind of non-transient computer readable storage medium Matter is stored with computer program in the non-transient computer readable storage medium, real when which is executed by processor The existing above-mentioned attention weight corresponding method that both modalities which data are carried out with feature interaction fusion, and/or, it is examined for psychological pressure The Overall Steps of the multi-modal fusion method of survey, for example, the processor realizes following processes when executing the computer program:

In addition, the logical order in above-mentioned memory can be realized and as independence by way of SFU software functional unit Product when selling or using, can store in a computer readable storage medium.Based on this understanding, of the invention Technical solution substantially the part of the part that contributes to existing technology or the technical solution can be with software in other words The form of product embodies, which is stored in a storage medium, including some instructions use so that One computer equipment (can be personal computer, server or the network equipment etc.) executes each embodiment institute of the present invention State all or part of the steps of method.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory (ROM, Read- Only Memory), random access memory (RAM, Random Access Memory), magnetic or disk etc. are various can be with Store the medium of program code.

The apparatus embodiments described above are merely exemplary, wherein described, unit can as illustrated by the separation member It is physically separated with being or may not be, component shown as a unit may or may not be physics list Member, it can it is in one place, or may be distributed over multiple network units.It can be selected according to the actual needs In some or all of the modules realize the purpose of the embodiment of the present invention.Those of ordinary skill in the art are not paying wound In the case where the labour for the property made, it can understand and implement.

Through the above description of the embodiments, those skilled in the art can be understood that each embodiment can It realizes by means of software and necessary general hardware platform, naturally it is also possible to pass through hardware.Based on this understanding, on Stating technical solution, substantially the part that contributes to existing technology can be embodied in the form of software products in other words, should Computer software product may be stored in a computer readable storage medium, such as ROM/RAM, magnetic disk, CD, including several fingers It enables and using so that a computer equipment (can be personal computer, server or the network equipment etc.) executes each implementation Multi-modal fusion method described in certain parts of example or embodiment for psychological pressure detection.

Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations；Although Present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that: it still may be used To modify the technical solutions described in the foregoing embodiments or equivalent replacement of some of the technical features； And these are modified or replaceed, technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution spirit and Range.

Claims

1. the attention weight corresponding method that a kind of pair of both modalities which data carry out feature interaction fusion characterized by comprising

It is obtained between reflection both modalities which data different characteristic based on the eigenmatrix of both modalities which data using matrix multiplication The incidence relation matrix of information relevance；

Based on the incidence relation matrix and feedforward fully-connected network model, the eigenmatrix pair of one of modal data is obtained Another influence power weight matrix of the eigenmatrix of modal data；

Based on the eigenmatrix of the influence power weight matrix and described two modal datas, connected using matrix dot product and residual error Connect, obtain comprising described two modal datas eigenmatrix influence each other weight attention strengthen eigenmatrix.

2. the attention weight corresponding method according to claim 1 that both modalities which data are carried out with feature interaction fusion, It is characterized in that, the eigenmatrix based on both modalities which data obtains reflection both modalities which data not using matrix multiplication With the incidence relation matrix of information relevance between feature, specifically include:

The incidence relation of information relevance between reflection both modalities which data different characteristic is obtained using following first relational model Matrix:

Wherein,Indicate that incidence relation matrix, A indicate that the eigenmatrix of one of modal data, B indicate another mode The eigenmatrix of data, Indicate that real number space, k indicate the dimension of the both modalities which data, B^TTable Show that B's turns order matrix, eigenmatrix A and eigenmatrix B is turned into order matrix multiple using matrix multiplication, is obtained comprising feature square Incidence relation matrix in battle array A in each feature and eigenmatrix B between each feature

3. the attention weight corresponding method according to claim 2 that both modalities which data are carried out with feature interaction fusion, It is characterized in that, it is described based on the incidence relation matrix and feedforward fully-connected network model, obtain one of modal data Eigenmatrix to the influence power weight matrix of the eigenmatrix of another modal data, specifically include:

Wherein, A_B→AIndicate influence power weight matrix of the eigenmatrix B to eigenmatrix A, B_A→BIndicate eigenmatrix A to feature square The influence power weight matrix of battle array B,Softmax indicates normalization exponential function, W₁Indicate the The first default training parameter in a kind of training parameter, W₂It indicates the second default training parameter in first kind training parameter, leads to One layer of fully-connected network is crossed, by incidence relation matrixIt maps backVector space, obtain eigenmatrix B to spy Levy the influence power weight matrix A of matrix A_B→AWith eigenmatrix A to the influence power weight matrix B of eigenmatrix B_A→B。

4. the attention weight corresponding method according to claim 3 that both modalities which data are carried out with feature interaction fusion, It is characterized in that, the eigenmatrix based on the influence power weight matrix and described two modal datas, utilizes matrix dot Multiply and connected with residual error, obtain comprising described two modal datas eigenmatrix influence each other weight attention strengthen feature square Battle array, specifically includes:

Wherein, ⊙ indicates dot product operation, using dot product operation come by A_B→AIt is multiplied with A, and the attention obtained after residual error connection is strong Change eigenmatrix In include the information of B and the influence that B generates A；Using dot product operation come by B_A→BIt is multiplied with B, And the attention obtained after residual error connection strengthens eigenmatrix In include the information of A and the influence that A generates B；

Wherein,f_AMMIt indicates to strengthen eigenmatrix by eigenmatrix A and eigenmatrix B to attentionStrengthen eigenmatrix with attentionTreatment process, specifically include: using first relational model to the 5th relationship Model handle to eigenmatrix A and eigenmatrix B the power reinforcing eigenmatrix that gains attentionStrengthen feature square with attention Battle arrayTreatment process.

5. a kind of attention for carrying out feature interaction fusion based on described in any item pairs of both modalities which data of such as Claims 1 to 4 The multi-modal fusion method for psychological pressure detection of power weight corresponding method characterized by comprising

The physiological data correlation eigen matrix of reflection user's physiological status is obtained respectively and reflects user psychology active state Text feature matrix and picture feature matrix；

Based on the physiological data correlation eigen matrix, the text feature matrix and the picture feature matrix, using described Attention weight corresponding method is obtained and is influenced each other comprising the physiological data correlation eigen matrix to the text feature matrix First attention of weight strengthens eigenmatrix, includes the physiological data correlation eigen matrix to the picture feature matrix phase Second attention of mutual weighing factor strengthens eigenmatrix, includes the text feature matrix to the physiological data correlated characteristic Matrix influence each other weight third attention strengthen eigenmatrix, comprising the text feature matrix to the picture feature square Influence each other the 4th attention of weight of battle array strengthens eigenmatrix, related to the physiological data comprising the picture feature matrix Eigenmatrix influence each other weight the 5th attention strengthen eigenmatrix and comprising the picture feature matrix to the text Eigenmatrix influence each other weight the 6th attention strengthen eigenmatrix；

Strengthen eigenmatrix based on first attention, second attention strengthens eigenmatrix, the third attention Strengthen eigenmatrix, the 4th attention strengthens eigenmatrix, the 5th attention strengthens eigenmatrix and the described 6th Attention strengthens eigenmatrix, based on the full Connection Neural Network that feedovers, obtains text fusion feature matrix, picture fusion feature square Battle array and physiological data fusion feature matrix；

Based on the physiological data correlation eigen matrix, the text feature matrix and the picture feature matrix, based on feedforward Full Connection Neural Network obtains text, picture, physiological data characteristic value；

Based on the text, picture, physiological data characteristic value, based on vector splicing and attention mechanism, obtain text, picture, The weights of importance value of physiological data；

Based on the text, picture, the weights of importance value of physiological data and the text fusion feature matrix, the picture Fusion feature matrix and the physiological data fusion feature matrix, obtain the fusion representing matrix of three kinds of mode；

Fusion representing matrix and feedforward fully-connected network based on three kinds of mode, obtain the pressure of reflection psychological pressure problem Power class vector.

6. the multi-modal fusion method according to claim 5 for psychological pressure detection, which is characterized in that the difference It obtains the physiological data correlation eigen matrix of reflection user's physiological status and reflects the text feature of user psychology active state Matrix and picture feature matrix, specifically include:

Attn_T=softmax (HW₃+b₁)

Wherein, F_TIndicate that text feature matrix, H indicate text representation matrix,Indicate the text readjusted by weight distribution Representing matrix, Attn_TThe contribution degree distribution of weights vector for indicating text representation matrix H, by text representation X={ x₁, x₂,···,x_nAs inputting into LSTM layers of shot and long term memory network, two are respectively obtained by positive LSTM and reversed LSTM The hidden layer of a LSTM exportsWithThe hidden layer of corresponding position is exported and is added, text is obtained Representing matrix H；The contribution degree distribution of weights vector Attn of text representation matrix H is obtained using attention mechanism_T: Attn_T= softmax(HW₃+b₁), Attn_TThe distribution for indicating the contribution weight of the text representation of each word, by Attn_TIt is multiplied with H, And connected by residual error, obtain the text representation matrix readjusted by weight distribution It is logical One layer of fully-connected network is crossed, it willIt is mapped to the vector space of k × 1, has obtained text feature matrix F_T:Wherein, W₃Indicate that the third in first kind training parameter presets training parameter, W₄Indicate first The 4th default training parameter in class training parameter, b₁Indicate the first default training parameter in the second class training parameter, b₂It indicates The second default training parameter in second class training parameter, ReLU indicate activation primitive；Softmax indicates normalization index letter Number, text representationx_iIndicate that the vector of word meaning, n indicate the word for including in text Quantity；

F_V=ReLU (W₅C+b₃)

Wherein, F_VIndicate that picture feature matrix, C indicate picture feature, with a full articulamentum by the dimension map of picture feature C To the vector space of n × 1, picture feature matrix F is obtained_V；Wherein, W₅Indicate the 5th default training in first kind training parameter Parameter, b₃Indicate that the third in the second class training parameter presets training parameter；

E=ReLU (W₇(ReLU(W₆E_S+b₄)+b₅))

Attn_E=softmax (W₈E+b₆)

Wherein, F_EIndicate physiological data correlation eigen matrix, E_SIndicate physiology related data eigenmatrix, E_SThe inside includes more A preset physiological characteristic, E are indicated to E_SCarrying out the physiology related data that two layers of fully-connected network obtains indicates matrix E, Attn_E Indicate that physiology related data indicates the contribution degree distribution of weights vector of matrix E,Indicate the text readjusted by weight distribution This representing matrixTo E_SCarrying out the physiology related data that two layers of fully-connected network obtains indicates matrix E:

E=ReLU (W₇(ReLU(W₆E_S+b₄)+b₅)), obtaining physiology related data using attention mechanism indicates the contribution of matrix E Spend distribution of weights vector Attn_E: Attn_E=softmax (W₈E+b₆), by Attn_EIt is multiplied with E, and is connected by residual error, obtained The text representation matrix readjusted by weight distributionIt, will by one layer of fully-connected network It is mapped to the vector space of k × 1, has obtained physiological data correlation eigen matrix F_E:Wherein, Attn_EIndicate the distribution for the contribution weight that each physiological characteristic indicates, W₆Indicate the 6th default instruction in first kind training parameter Practice parameter, W₇Indicate the 7th default training parameter in first kind training parameter, W₈The 8th in expression first kind training parameter is pre- If training parameter, W₉Indicate the 9th default training parameter in first kind training parameter, b₄Indicate in the second class training parameter Four default training parameters, b₅Indicate the 5th default training parameter in the second class training parameter, b₆It indicates in the second class training parameter The 6th default training parameter, b₇Indicate the 7th default training parameter in the second class training parameter.

7. the multi-modal fusion method according to claim 6 for psychological pressure detection, which is characterized in that described to be based on The physiological data correlation eigen matrix, the text feature matrix and the picture feature matrix are weighed using the attention Weight corresponding method obtains and influences each other the of weight to the text feature matrix comprising the physiological data correlation eigen matrix One attention strengthen eigenmatrix, comprising the physiological data correlation eigen matrix to the picture feature matrix power of influencing each other Second attention of weight strengthens eigenmatrix, includes that the text feature matrix is mutual to the physiological data correlation eigen matrix The third attention of weighing factor strengthens eigenmatrix, includes the text feature matrix to the mutual shadow of picture feature matrix The 4th attention for ringing weight strengthens eigenmatrix, includes the picture feature matrix to the physiological data correlation eigen matrix 5th attention of the weight that influences each other strengthens eigenmatrix and comprising the picture feature matrix to the text feature matrix 6th attention of the weight that influences each other strengthens eigenmatrix, specifically includes:

Based on the physiological data correlation eigen matrix and the text feature matrix, using any one of such as Claims 1 to 4 institute That states carries out the attention weight corresponding method of feature interaction fusion to both modalities which data, and obtaining includes the physiological data phase Close eigenmatrix to the text feature matrix influence each other weight the first attention strengthen eigenmatrix；Described first pays attention to It is that physiological data -> text attention strengthens eigenmatrix that power, which strengthens eigenmatrix,

Based on the physiological data correlation eigen matrix and the picture feature matrix, using any one of such as Claims 1 to 4 institute That states carries out the attention weight corresponding method of feature interaction fusion to both modalities which data, and obtaining includes the physiological data phase Close eigenmatrix to the picture feature matrix influence each other weight the second attention strengthen eigenmatrix；Described second pays attention to It is that physiological data -> picture attention strengthens eigenmatrix that power, which strengthens eigenmatrix,

Based on the physiological data correlation eigen matrix and the text feature matrix, using any one of such as Claims 1 to 4 institute That states carries out the attention weight corresponding method of feature interaction fusion to both modalities which data, and obtaining includes the text feature square Battle array to the physiological data correlation eigen matrix influence each other weight third attention strengthen eigenmatrix；The third pays attention to It is that text -> physiological data attention strengthens eigenmatrix that power, which strengthens eigenmatrix,

It is described in any item to two using such as Claims 1 to 4 based on the text feature matrix and the picture feature matrix Kind modal data carries out the attention weight corresponding method of feature interaction fusion, obtains comprising the text feature matrix to described Picture feature matrix influence each other weight the 4th attention strengthen eigenmatrix；4th attention strengthens eigenmatrix Text -> picture attention strengthens eigenmatrix

Based on the physiological data correlation eigen matrix and the picture feature matrix, using any one of such as Claims 1 to 4 institute That states carries out the attention weight corresponding method of feature interaction fusion to both modalities which data, and obtaining includes the picture feature square Battle array to the physiological data correlation eigen matrix influence each other weight the 5th attention strengthen eigenmatrix；Described 5th pays attention to It is that picture -> physiological data attention strengthens eigenmatrix that power, which strengthens eigenmatrix,

It is described in any item to two using such as Claims 1 to 4 based on the text feature matrix and the picture feature matrix Kind modal data carries out the attention weight corresponding method of feature interaction fusion, obtains comprising the picture feature matrix to described Text feature matrix influence each other weight the 6th attention strengthen eigenmatrix；6th attention strengthens eigenmatrix Picture -> text attention strengthens eigenmatrix

8. the multi-modal fusion method according to claim 7 for psychological pressure detection, which is characterized in that described to be based on First attention strengthens eigenmatrix, second attention strengthens eigenmatrix, the third attention strengthens feature Matrix, the 4th attention strengthen eigenmatrix, the 5th attention strengthens eigenmatrix and the 6th attention is strong Change eigenmatrix, based on the full Connection Neural Network that feedovers, obtains text fusion feature matrix, picture fusion feature matrix and physiology Data fusion eigenmatrix, specifically includes:

Using following 9th processing model, eigenmatrix is strengthened based on first attention and the 6th attention strengthens spy Matrix is levied, by one layer of fully-connected network, obtains text fusion feature matrix

Using following tenth processing model, eigenmatrix is strengthened based on second attention and the 4th attention strengthens spy Matrix is levied, by one layer of fully-connected network, obtains picture fusion feature matrix

Using following 11st processing model, eigenmatrix is strengthened based on the third attention and the 5th attention is strengthened Eigenmatrix obtains physiological data fusion feature matrix by one layer of fully-connected network

Wherein, W₁₀~W₁₅Indicate the tenth default training parameter to the 15th default training parameter in first kind training parameter, b₈ ~b₁₀Indicate the 8th to the tenth default training parameter in the second class training parameter.

9. the multi-modal fusion method according to claim 8 for psychological pressure detection, which is characterized in that described to be based on The physiological data correlation eigen matrix, the text feature matrix and the picture feature matrix, based on the full connection mind of feedforward Through networks, text, picture, physiological data characteristic value are obtained, is specifically included:

S_T=ReLU (W₁₆softmax(F_T)+b₁₁)

S_V=ReLU (W₁₇softmax(F_V)+b₁₂)

S_E=ReLU (W₁₈softmax(F_E)+b₁₃)

Wherein, by the physiological data correlation eigen matrix F_E, the text feature matrix F_TWith the picture feature square F_VBattle array is reflected It is mapped between (0,1), text, picture, physiological data characteristic value S is then obtained by one layer of full connection_T, S_VAnd S_E；Wherein, W₁₆ ~W₁₈Indicate the 16th to the 18th default training parameter in first kind training parameter, b₁₁~b₁₃Indicate the second class training ginseng The the 11st to the 13rd default training parameter in number.

10. the multi-modal fusion method according to claim 9 for psychological pressure detection, which is characterized in that the base Text, picture, physiology number are obtained based on vector splicing and attention mechanism in the text, picture, physiological data characteristic value According to weights of importance value, specifically include:

Using following 13rd processing model, by the text, picture, physiological data characteristic value S_T, S_VAnd S_EIt is spliced together, By attention mechanism, the weights of importance value weight of text, picture, physiological data is obtained_T, weight_VAnd weight_E:

(weight_T,weight_V,weight_E)=softmax ([S_T,S_V,S_E]W₁₉)

11. the multi-modal fusion method according to claim 10 for psychological pressure detection, which is characterized in that the base It is special in the text, picture, the weights of importance value of physiological data and the text fusion feature matrix, picture fusion Matrix and the physiological data fusion feature matrix are levied, the fusion representing matrix of three kinds of mode is obtained, specifically includes:

Wherein, W₂₀~W₂₂Indicate the 20th default training parameter to the 22nd default training ginseng in first kind training parameter Number.

12. the attention weight corresponding intrument that a kind of pair of both modalities which data carry out feature interaction fusion, which is characterized in that packet It includes:

First obtains module, obtains reflection both modalities which using matrix multiplication for the eigenmatrix based on both modalities which data The incidence relation matrix of information relevance between data different characteristic；

Second obtains module, for obtaining one of mould based on the incidence relation matrix and feedforward fully-connected network model Influence power weight matrix of the eigenmatrix of state data to the eigenmatrix of another modal data；

Third obtains module, for the eigenmatrix based on the influence power weight matrix and described two modal datas, utilizes Matrix dot product is connected with residual error, obtain comprising described two modal datas eigenmatrix influence each other weight attention strengthen Eigenmatrix.

13. a kind of based on the attention weight pair that as claimed in claim 12 both modalities which data are carried out with feature interaction fusion Answer the multi-modal fusion device for psychological pressure detection of device characterized by comprising

4th obtains module, and the physiological data correlation eigen matrix and reflection for obtaining reflection user's physiological status respectively are used The text feature matrix and picture feature matrix of family psychological activity state；

5th obtains module, for being based on the physiological data correlation eigen matrix, the text feature matrix and the picture Eigenmatrix is obtained comprising the physiological data correlation eigen matrix using the attention weight corresponding method to the text Influence each other the first attention of weight of eigen matrix strengthens eigenmatrix, comprising the physiological data correlation eigen matrix pair The picture feature matrix influence each other weight the second attention strengthen eigenmatrix, comprising the text feature matrix to institute It states the influence each other third attention of weight of physiological data correlation eigen matrix and strengthens eigenmatrix, comprising the text feature square Battle array strengthens eigenmatrix, comprising the picture feature matrix to influence each other the 4th attention of weight of the picture feature matrix Eigenmatrix and comprising the picture is strengthened to influence each other the 5th attention of weight of the physiological data correlation eigen matrix Eigenmatrix to the text feature matrix influence each other weight the 6th attention strengthen eigenmatrix；

6th obtains module, and for strengthening eigenmatrix based on first attention, second attention strengthens feature square Battle array, the third attention strengthen eigenmatrix, the 4th attention strengthens eigenmatrix, the 5th attention strengthens spy It levies matrix and the 6th attention strengthens eigenmatrix, based on the full Connection Neural Network that feedovers, obtain text fusion feature square Battle array, picture fusion feature matrix and physiological data fusion feature matrix；

7th obtains module, for being based on the physiological data correlation eigen matrix, the text feature matrix and the picture Eigenmatrix obtains text, picture, physiological data characteristic value based on the full Connection Neural Network that feedovers；

8th obtains module, for being based on the text, picture, physiological data characteristic value, based on vector splicing and attention machine System obtains the weights of importance value of text, picture, physiological data；

9th obtains module, for being merged based on the text, picture, the weights of importance value of physiological data and the text Eigenmatrix, the picture fusion feature matrix and the physiological data fusion feature matrix, obtain the fusion table of three kinds of mode Show matrix；

Tenth obtains module, for fusion representing matrix and feedforward fully-connected network based on three kinds of mode, obtains anti- Reflect the pressure class vector of psychological pressure problem.

14. a kind of electronic equipment including memory, processor and stores the calculating that can be run on a memory and on a processor Machine program, which is characterized in that the processor is realized as described in any one of Claims 1-4 when executing described program to two kinds Modal data carries out the step of attention weight corresponding method of feature interaction fusion, and/or, as claim 5 to 11 is any The step of multi-modal fusion method of psychological pressure detection is used for described in.

15. a kind of non-transient computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer It is realized when program is executed by processor and feature interaction fusion is carried out to both modalities which data as described in any one of Claims 1-4 Attention weight corresponding method the step of, and/or, as described in any one of claim 5 to 11 for psychological pressure detection The step of multi-modal fusion method.