CN114209319A - fNIRS emotion recognition method and system based on graph network and adaptive denoising - Google Patents

Abstract

The invention discloses an fNIRS emotion recognition method and system based on a graph network and self-adaptive denoising, which comprises the steps that fNIRS acquisition equipment continuously acquires the variation of light intensity before and after transmitting and receiving, converts the variation of the light intensity into the variation of absorbance, and further obtains the relative variation of the concentration of oxygenated hemoglobin and deoxygenated hemoglobin; denoising through a self-adaptive denoising network model to obtain a pure signal, wherein the input signal of the self-adaptive denoising network model is the relative variation of the concentrations of oxyhemoglobin and deoxyhemoglobin obtained in the last step, and the output signal is the relative variation data of the concentrations of pure oxyhemoglobin and deoxyhemoglobin; mapping graph nodes by combining probe and channel characteristics, restoring brain topology by using a graph network, recognizing network models by dynamic graph attention emotion, and classifying and outputting emotion labels. The invention solves the problems of complex wearing, difficult operation and the like of the brain-computer interface in practical application at present.

Description

fNIRS emotion recognition method and system based on graph network and adaptive denoising

technical field

The invention relates to the field of human-machine signal recognition, in particular to a fNIRS emotion recognition method and system based on graph network and adaptive denoising.

Background technique

Emotions affect people's cognitive and behavioral activities, and are also an important factor in mental health. As a research hotspot, emotion recognition can be divided into non-physiological signals and physiological signals. As traditional physiological index detection methods, EEG, EEG and fMRI have made certain progress in emotion recognition. At the same time, the limitations of such methods are gradually revealed, such as low temporal or spatial resolution, high cost of acquisition equipment, susceptibility to interference, and inconvenience to carry.

In recent years, with the development of near-infrared technology and the technical upgrade of acquisition equipment, functional near-infrared spectroscopy (fNIRS), as an emerging non-invasive brain detection method, has high compliance, With the advantages of strong anti-interference ability, portability, easy implementation and low cost, it is suitable for all possible test groups and experimental scenarios. With the continuous development of 5G technology, Internet of Things, human-computer interaction, machine learning and other technologies, sentiment analysis based on fNIRS is of great significance and broadness in the fields of healthcare, media entertainment, information retrieval, education, and smart wearable devices. application prospects. Therefore, an emotion recognition method and system based on functional near-infrared spectroscopy has wide demands and broad prospects.

SUMMARY OF THE INVENTION

In order to overcome the shortcomings and deficiencies of the prior art, the present invention provides an fNIRS emotion recognition method and system based on a graph network and adaptive denoising.

The present invention adopts following technical scheme:

As shown in Figure 1, a fNIRS emotion recognition method based on graph network and adaptive denoising includes the following steps:

The S1 fNIRS acquisition device continuously collects the change of light intensity before and after emission and reception, converts the change of light intensity into the change of absorbance, and uses the Beer-Lambert law to obtain the difference between the change of absorbance and the concentration of light-absorbing chromophore in brain tissue. The relationship equation mainly refers to oxyhemoglobin and deoxyhemoglobin, and the relative change of oxyhemoglobin and deoxyhemoglobin concentrations is further obtained by solving the equation.

The further process is as follows:

与

S1.1 Obtain the continuous wave original near-infrared light intensity changes of two different wavelengths from the acquisition device, denoted as

and

与

S1.2 Convert raw light intensity information to absorbance

and

S1.3. According to the Beer-Lambert law, the relationship equation between the change of absorbance and the relative change of the concentration of light-absorbing chromophore in brain tissue is obtained;

Among them, ε is the molar extinction coefficient, d is the detection depth, and DPF is the differential path factor;

S1.4. Through the solution of the equation, the relative changes of oxyhemoglobin and deoxyhemoglobin concentrations are obtained, which are recorded as ΔC _HbO (t) and ΔC _HbR (t).

S2 obtains a pure signal by denoising the adaptive denoising network model, the input signal of the adaptive denoising network model is the relative change of the concentration of oxyhemoglobin and deoxyhemoglobin obtained in the previous step, and the output signal is pure oxyhemoglobin and relative changes in deoxyhemoglobin concentration data.

More specifically:

和

As shown in FIG. 2 , the adaptive denoising network model includes a depthwise convolution confrontation formed by multiple convolution and deconvolution blocks of different sizes, denoted as G _p , where the input is the noisy data ΔC _HbO ( t) and ΔC _HbR (t), the output of the generation network is the relative change data of the pure oxyhemoglobin and deoxyhemoglobin concentrations, denoted as

and

The difference between the output of the denoising network model and the input is the generated pure noise signal, denoted as

与纯净的数据P_HbO(t)和P_HbR(t)相加，获得生成的带噪数据，记为

和

The pure noise signal that will be generated

Add to the clean data P _HbO (t) and P _HbR (t) to obtain the generated noisy data, denoted as

and

Denote ΔC _HbO (t), ΔC _HbR (t) as ΔC, P _HbO (t), and P _HbR (t) as P, then the total loss of the model is defined as:

为噪声判别式的损失：in,

is the loss of the noise discriminant:

为纯净判别式的损失：

is the loss of pure discriminant:

为循环一致性损失：

For cycle consistency loss:

After iterative training, the denoising algorithm is:

P _HbO (t), P _HbR (t) = G _p (ΔC _HbO (t),ΔC _HbR (t)) (8)

Relative change data P _HbO (t) and P _HbR (t) of pure oxyhemoglobin and deoxyhemoglobin concentrations can be obtained.

S3 combines the characteristics of probes and channels to map nodes in the graph, uses the graph network to restore the brain topology, and uses the dynamic graph attention emotion recognition network model to classify and output emotion labels. The dynamic graph attention emotion recognition network model includes graph convolution and attention mechanism, and the probe is the optode of fNIRS, including a transmitter and a receiver.

As shown in Figure 3, the specific process is:

S3.1 First define the graph, denoted as G(V,E,W), where V represents the combination of graph nodes, |V|=n represents a total of n nodes, corresponding to the data sequence ΔC of n channels of fNIRS _HbO (t), ΔC _HbR (t) is denoted as X; E represents the set of edges in the graph and the set of different channels in fNIRS; W is the adjacency matrix, which defines the connection relationship of nodes, that is, the correlation of different channels in fNIRS, The value in the adjacency matrix describes the importance of the relationship between nodes, and the initialization method of the value w _ij uses the Gaussian kernel function method:

In the formula, dist is the Gaussian distance between nodes, and θ and τ are fixed parameters in the Gaussian distance algorithm.

S3.2. In order to introduce graph attention, first calculate the similarity coefficient e _ij between each node and neighboring nodes:

和

分别为节点i和节点j的权重矩阵；where a is the global mapping matrix,

and

are the weight matrices of node i and node j, respectively;

S3.3 calculates the attention coefficient between graph nodes and normalizes it, denoted as α _ij :

In the formula, LeakyReLU() is a nonlinear activation function;

S3.4. In graph convolution, multi-head attention is used to perform weighted summation for parameter integration to obtain new features X′ _i :

In the formula, σ is the nonlinear mapping relationship, K is the number of attention heads, and || represents splicing;

S3.5, perform pooling dimension reduction, output emotion categories through the classifier after flattening and fully connected layers, the model framework is shown in Figure 2;

S3.6. The model uses cross entropy plus a regularization term as the loss function, which is expressed as:

分别为真实标签与预测值，

为学习率，

用于表示模型所有参数；Among them, CossEntropy() represents cross entropy calculation, l,

are the true label and the predicted value, respectively,

is the learning rate,

Used to represent all parameters of the model;

S3.7. Use the back-propagation algorithm to realize the dynamic change of the adjacency matrix, and calculate the loss function to iteratively update the network on the partial differential of the adjacency matrix:

The iterative update formula is:

A fNIRS emotion recognition system based on graph network and adaptive denoising, including:

fNIRS acquisition module: use the fNIRS acquisition equipment to continuously collect the change of light intensity before and after transmission and reception, convert the change of light intensity into the change of absorbance, and further obtain the relative change of the concentration of oxyhemoglobin and deoxyhemoglobin;

fNIRS adaptive denoising network module: used to obtain pure signals;

fNIRS dynamic graph attention emotion recognition network module: output emotion labels according to pure signals.

A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the fNIRS emotion recognition method when the processor executes the computer program.

A storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the fNIRS emotion recognition method.

Beneficial effects of the present invention:

(1) In the process of data denoising, an adaptive denoising model based on generative adversarial network is adopted. Compared with the traditional data denoising method based on machine learning, the present invention can largely avoid manual participation and experience analysis in the process of data denoising, overcome the disadvantage that the traditional method has strong task dependence, and has relatively high performance under multi-task conditions. High adaptability. At the same time, through the method of generative adversarial learning, the noise difference problem of fNIRS in "static tasks" and "dynamic tasks" can be solved without specific assumptions, and the denoising model has strong generalization ability.

(2) In the emotional representation extraction of fNIRS signals, a deep learning network model is used. Compared with the traditional manual feature extraction method, the present invention can realize the extraction of emotional features in fNIRS through network learning in a data-driven manner, overcoming the limited dimension and uncertainty of validity existing in the calculation of fixed features in the traditional method. And other issues. Learning and extracting features through a deep learning network can effectively obtain emotional representations of different dimensions, and enhance the extraction and utilization of emotional features in fNIRS data.

(3) The dynamic graph convolutional neural network is used to effectively model the fNIRS data with probe position information and channel signal. Compared with the traditional data that is simply regarded as a time series signal and analyzed using a machine learning method based on a support vector machine, a Bayesian classifier, or a recurrent neural network based on long short-term memory, the present invention proposes to use a graph method to perform topology on fNIRS data. Mapping, which maps different probes to nodes in the graph, the time series data is the feature of the node, and the relationship of different passes is represented as the edge in the graph through the adjacency matrix. The method makes full use of the characteristics of the data, has the restoration of the topology of the brain structure, and simultaneously characterizes the data correlation of different channels, which can improve the accuracy of the network model in the emotion recognition of fNIRS brain signal detection.

(4) This method introduces a graph attention mechanism, obtains the attention coefficients between different nodes by calculating the similarity coefficients between different probe nodes and adjacent nodes, and uses the weighted summation of the multi-head attention mechanism in the graph convolution process. Update node features. This method can introduce the feature relationship of adjacent nodes in the model training process, so that the model can better extract the correlation features of different channel data in fNIRS, and can also obtain the activation responses of different brain regions to different emotions , has a significant role in emotion recognition of brain signals.

Description of drawings

Fig. 1 is the working flow chart of the present invention;

Fig. 2 is the fNIRS adaptive denoising network model structural diagram of the present invention;

Fig. 3 is the fNIRS dynamic graph attention emotion recognition network model structure diagram of the present invention;

FIG. 4 is a schematic diagram of the fNIRS acquisition module of the present invention.

Detailed ways

The present invention will be described in further detail below with reference to the embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.

Example 1

An fNIRS emotion recognition method based on graph network and adaptive denoising is suitable for emotion recognition tasks of fNIRS acquisition equipment, and mainly includes an external emotional stimulation step, an fNIRS acquisition step, an fNIRS adaptive denoising step, and an fNIRS emotion recognition step.

In the external emotional stimulation step, the external emotional stimulation material adopts videos of six emotional label types of anger, disgust, fear, pleasure, sadness and surprise, and users can induce emotions by watching the videos.

As shown in Figure 4, the fNIRS acquisition step, the fNIRS acquisition equipment is composed of multi-channel dual-wavelength near-infrared continuous wave transmitter-receiver sources.

First, the user wears the fNIRS acquisition device to collect and record the change of light intensity of different optodes before and after emission and reception in real time.

与

Let the single-channel light intensity change at any time be:

and

The calculation is converted to absorbance as:

According to Beer-Lambert law: ΔA=ε×d×DPF, where ε is the molar extinction coefficient, d is the detection depth, and DPF is the differential path factor.

By solving the relationship equation between the change of absorbance and the relative change of the concentration of light-absorbing chromophore in brain tissue, the relative change of the concentration of oxyhemoglobin and deoxyhemoglobin was obtained, which was recorded as ΔC _HbO and ΔC _HbR .

fNIRS adaptive denoising step

The data is input into the fNIRS adaptive denoising module for denoising enhancement, and the trained generation network obtains the relative change data P _HbO and P _HbR of pure oxyhemoglobin and deoxyhemoglobin concentrations. P _HbO , PHbR=G _p (ΔC _HbO , ΔC _HbR ).

The fNIRS adaptive denoising module is based on a generative adversarial network, performs feature extraction on the fNIRS signal through convolution, and generates a pure signal through deconvolution. In the training process of the network, input pairs of noisy signals and pure signals are trained through the adversarial loss of the two discriminators, and at the same time, the cycle consistency loss is introduced to constrain the spatial mapping to improve the training efficiency of the model.

fNIRS emotion recognition steps

Map the pure data of all channels to graph nodes: P _HbO , P _HbR → V, V is the set of graph nodes. Use the Gaussian kernel function to initialize the adjacency matrix W, where the Gaussian kernel function is:

In the formula, dist is the Gaussian distance between nodes, and θ and τ are fixed parameters in the Gaussian distance algorithm.

a为全局映射矩阵，

和

分别为节点i和节点j的权重矩阵。Calculate the similarity coefficient between each node and neighboring nodes

a is the global mapping matrix,

and

are the weight matrices of node i and node j, respectively.

Using the LeakyReLU nonlinear activation function to normalize the graph node attention coefficient as α _ij ,

σ为非线性映射关系，K为注意力头的数量,||表示拼接。New features obtained by graph convolution weighted summation and splicing of multi-head attention parameters

σ is the nonlinear mapping relationship, K is the number of attention heads, and || represents splicing.

The pooling dimension is reduced, and the emotion recognition probability is output through the classifier after flattening and full connection layer.

In this embodiment, the emotion recognition probability is the probability value of the six Ekman emotion classification labels of anger, disgust, happiness, sadness, and surprise, and

Existing technologies are mostly based on physiological signal-based emotion recognition methods, and most of them are based on EEG signals and functional magnetic resonance. The role and potential in emotion research is not only of great significance in practical applications, but also creates a new method for emotional analysis of physiological signals.

The signal de-noising method adopted in this application can realize end-to-end adaptive de-noising of multi-channel fNIRS signals, and the algorithm has high generalization ability and universality.

The invention proposes a dynamic graph attention model based on the dynamic graph convolution method to construct the brain topology and extract features, and at the same time introduces an attention mechanism to extract the correlation features between fNIRS channels, so as to improve the learning ability of the model and obtain higher emotion recognition accuracy.

The invention adopts the method of deep learning, and extracts the learning of features through data driving, which improves the expression ability of emotional features and avoids human participation.

Example 2

A fNIRS emotion recognition system based on graph network and adaptive denoising, including:

fNIRS acquisition module: use the fNIRS acquisition equipment to continuously collect the change of light intensity before and after transmission and reception, convert the change of light intensity into the change of absorbance, and further obtain the relative change of the concentration of oxyhemoglobin and deoxyhemoglobin;

fNIRS adaptive denoising network module: used to obtain pure signals;

fNIRS dynamic graph attention emotion recognition network module: output emotion labels according to pure signals.

Example 3

A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the fNIRS emotion recognition method when the processor executes the computer program.

Example 4

A storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the fNIRS emotion recognition method.

The above-mentioned embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited by the described embodiments, and any other changes, modifications, substitutions, and combinations made without departing from the spirit and principle of the present invention , simplification, all should be equivalent replacement modes, and are all included in the protection scope of the present invention.

Claims (10)

1. A fNIRS emotion recognition method based on graph network and adaptive denoising is characterized by comprising the following steps:

continuously acquiring the variation of light intensity before and after transmitting and receiving by the fNIRS acquisition equipment, converting the variation of the light intensity into the variation of absorbance, and further obtaining the relative variation of the concentration of oxyhemoglobin and deoxyhemoglobin;

denoising through a self-adaptive denoising network model to obtain a pure signal, wherein the input signal of the self-adaptive denoising network model is the relative variation of the concentrations of oxyhemoglobin and deoxyhemoglobin obtained in the last step, and the output signal is the relative variation data of the concentrations of pure oxyhemoglobin and deoxyhemoglobin;

mapping graph nodes by combining probe and channel characteristics, restoring brain topology by using a graph network, recognizing network models by dynamic graph attention emotion, and classifying and outputting emotion labels.

2. The fNIRS emotion recognition method of claim 1, wherein the adaptive denoising network model comprises a plurality of deep convolution pairs of convolution and deconvolution blocks of different sizes.

3. The fNIRS emotion recognition method of claim 2, wherein the convolution performs feature extraction on the fNIRS signal and clean signal generation is performed by deconvolution blocks.

4. The fNIRS emotion recognition method of claim 1, wherein the dynamical graph attention emotion recognition network model comprises a graph convolution and attention mechanism.

5. The fNIRS emotion recognition method of claim 4, wherein the recognition process of the dynagram attention emotion recognition network model is as follows: and constructing a graph network, mapping data to a graph, mapping pure fNIRS signals to nodes of the graph, mapping characteristic features of probes and channels to edges of the graph, extracting the features by a dynamic graph convolution method, simultaneously introducing an attention mechanism to learn channel relevance, finally classifying and outputting through dimensionality reduction and flattening splicing of graph pooling, and realizing accurate emotion identification.

6. The fNIRS emotion recognition method of claim 5, wherein the emotion labels are output in a classified manner by the network model for attention emotion recognition of the dynamic graph, specifically:

the definition of the graph is denoted as G (V, E, W), where V represents the combination of graph nodes and | V | ═ n tableData sequence Δ C showing n nodes in total, corresponding to n channels of fNIRS_HbO(t)，ΔC_HbR(t) is denoted as X; e represents the set of edges in the graph, and the set of different channels in fNIRS; w is an adjacency matrix and defines the connection relation of the nodes, namely the relevance of different channels in the fNIRS, wherein the values in the adjacency matrix describe the importance of the relation between the nodes, and the value W_ijThe initialization method uses a Gaussian kernel function method;

drawing attention, and calculating similarity coefficients of each node and adjacent nodes;

calculating attention coefficients among the nodes of the graph and normalizing the attention coefficients;

in graph convolution, weighted summation is carried out by using multi-head attention to carry out parameter integration, and new features are obtained;

and performing pooling dimensionality reduction, and outputting emotion categories through a classifier after flattening and full connection layers.

7. The fNIRS emotion recognition method of any one of claims 1 to 6, wherein the equation relating the change in absorbance to the change in the concentration of the light-absorbing chromophore in the brain tissue is obtained using the beer-lambert law, and the relative change in the concentrations of oxyhemoglobin and deoxyhemoglobin is obtained by solving the equation.

8. A system for implementing the method of fNIRS emotion recognition of any of claims 1-6, comprising:

the fNIRS acquisition module: continuously acquiring the variation of light intensity before and after transmitting and receiving by using an fNIRS acquisition device, converting the variation of the light intensity into the variation of absorbance, and further obtaining the relative variation of the concentration of oxyhemoglobin and deoxyhemoglobin;

the fNIRS self-adaptive denoising network module: for obtaining a clean signal;

the fNIRS dynamics graph attention emotion recognition network module: and outputting the emotion label according to the pure signal.

9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the computer program implements the steps of the fNIRS emotion recognition method of any of claims 1 to 7.

10. A storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, performs the steps of the fNIRS emotion recognition method of any of claims 1 to 7.

Images