CN111881812B - Multi-modal emotion analysis method and system based on deep learning for acupuncture - Google Patents
Multi-modal emotion analysis method and system based on deep learning for acupuncture Download PDFInfo
- Publication number
- CN111881812B CN111881812B CN202010724202.6A CN202010724202A CN111881812B CN 111881812 B CN111881812 B CN 111881812B CN 202010724202 A CN202010724202 A CN 202010724202A CN 111881812 B CN111881812 B CN 111881812B
- Authority
- CN
- China
- Prior art keywords
- emotion
- signal
- acupuncture
- electroencephalogram
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
- G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/08—Feature extraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/467—Encoded features or binary features, e.g. local binary patterns [LBP]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Multimedia (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Human Computer Interaction (AREA)
- Biodiversity & Conservation Biology (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
The invention provides a multi-modal emotion analysis method and system based on deep learning for acupuncture and belongs to the technical field of emotion recognition. The method comprises the steps of extracting facial expression characteristics by Gabor wavelet transform, extracting electroencephalogram signals by UPLBP, reducing dimension, performing sparse linear fusion on the facial expression characteristics and the electroencephalogram signal characteristics to form a uniform and normalized feature vector, converting the feature vector into a tensor form, inputting a CNN-LSTM network for training, removing redundant information and obtaining predicted emotion classification information, and calculating a loss function and a correct rate of the network by comparing the predicted emotion classification information with actual emotion classification information. According to the invention, the expression characteristics and the electroencephalogram signal characteristics are fused, redundant information is removed, and the predicted emotion classification information is obtained through the training of the CNN-LSTM network, so that the emotion recognition accuracy is improved.
Description
Technical Field
The invention relates to the technical field of emotion recognition, in particular to a multi-modal emotion analysis method and system based on deep learning for acupuncture.
Background
Emotion recognition is an important branch of the field of image recognition, which recognizes the emotion of a person based on the external expression or internal physiological signals of the person. Emotion, although an internal subjective experience, is always accompanied by some external manifestation, i.e., some behavioral features that can be observed, that is, the manifestation of a person's face, posture and intonation. Meanwhile, besides the external expression, the emotion also generates physiological signals related to the emotion in the human body, and the heart rate and the brain electrical signals belong to the physiological signals caused by the emotion. Emotional analysis and recognition relate to a plurality of discipline fields, including neuroscience, psychology, cognitive science, computer science, artificial intelligence and the like, and are a cross discipline research.
Emotion recognition methods generally fall into two categories: extrinsic expression based recognition and physiological signal based recognition. The emotion recognition method based on the extrinsic expression mainly comprises the recognition of facial expressions and intonation expressions. The facial expression recognition method is used for recognizing different emotions according to the external expression of the emotions in the facial expressions; the tone recognition is performed based on the external expression of emotion in the language expression. The disadvantage of being based on external expressions is that the reliability of emotion recognition cannot be guaranteed, because the appearance of the external expressions is related to the character, and in addition, people can mask the real emotion of the people by disguising facial expressions and voice tones, and the disguise is not easy to find. In the emotion recognition method based on physiological signals, the recognition method based on the central nervous system is widely adopted at present, the principle is that different physiological signals, namely commonly-spoken electroencephalogram signals, are generated in the brain of a person under different emotions, different emotions are recognized by recognizing the electroencephalogram signals, the method is not easy to disguise, and the recognition rate is high. However, the recognition rate still needs to be improved compared with the recognition of emotion through external expression. Emotion recognition and analysis are increasingly applied to a wide range of fields, and people are also increasingly exploring emotion recognition methods.
In chinese patent application CN107463874A, a method for emotion recognition is disclosed, which comprises the following steps: acquiring a detected face image and an electroencephalogram signal at the same time or within the same time period; respectively processing a detected face image and an electroencephalogram signal, performing feature extraction on a face gray image to obtain expression original feature data, performing data dimension reduction on the expression original feature data to obtain an expression feature value, inputting the expression feature value into a pre-trained convolutional neural network, calculating to obtain an expression parameter value, sequentially performing feature standardization and feature normalization processing on the electroencephalogram signal to obtain an electroencephalogram feature value, and performing feature classification on the electroencephalogram feature value through a pre-trained SVM classifier to obtain an internal emotion parameter value; carrying out data fusion processing on the emotion parameter value and the internal emotion parameter value to obtain an emotion fusion parameter value; and identifying the current emotion of the detected face according to the comparison between the emotion fusion parameter value and a prestored emotion fusion database to obtain emotion information. The method combines the facial expression with the electroencephalogram signal for emotion recognition, and compares the fusion parameters with the fusion parameters prestored in the database to obtain emotion information. The method only performs common data analysis (such as mean, standard deviation, variance and the like) on the electroencephalogram image, and does not have the cognition of trend or curve morphology, so that the extracted features have certain influence on the classification accuracy; meanwhile, for the fusion of multi-modal features, the method cannot extract spatially sequenced features.
In chinese patent application document CN107220591A, a multimodal intelligent emotion sensing system is disclosed, which comprises an acquisition module, an identification module, and a fusion module, wherein the acquisition module acquires facial expressions, voices, actions, and physiological signals, the identification module includes emotion recognition units based on the expressions, the voices, the actions, and the physiological signals, each emotion recognition unit in the identification module identifies multimodal information, thereby obtaining each emotion component, and the fusion module fuses the emotion components of the identification module, thereby implementing accurate sensing of human emotion; the emotion recognition method comprises the steps that effective features are extracted from collected videos or images by each emotion recognition unit, mapping models of expressions, voices, actions and emotions are trained, and on the basis of the trained models, the emotion features are recognized through a classifier, so that emotion components are obtained; the emotion recognition unit based on the physiological signals is used for filtering noise in the physiological signals of a user acquired in a contact or non-contact mode, extracting the characteristics of the physiological signals of electrocardio, pulse, myoelectricity, electrodeionization, electroencephalogram and respiration signals by using a classical modal decomposition and Hilbert-Huang conversion algorithm, performing characteristic fusion on the characteristics by using a linear fusion method, selecting the characteristics by using an information gain rate, and finally recognizing the characteristics by using a classifier to obtain emotion components based on the physiological signals. The method considers the expression of various non-physiological factors of expression, voice and action and various physiological factors of electrocardio, pulse, myoelectricity, skin electricity, electroencephalogram and respiratory signals on emotion, but because the acquisition of various information, such as voice, is adopted, the emotion of people can be interfered and fluctuated, and the emotion recognition accuracy is influenced.
In the existing clinical acupuncture, the research on the emotion changes of patients or normal people before and after acupuncture is mostly a subjective evaluation mode such as scale evaluation, and the like, and the index of digitalization and quantification is lacked. And emotion recognition has not been applied to clinical acupuncture.
The prior art has at least the following disadvantages:
1. in the existing emotion recognition method, although non-physiological expressions such as facial expressions and voices are combined with physiological factors such as electroencephalogram and heart rate to perform emotion recognition, the problem of interference of electroencephalogram and heart rate and the like is caused when voice signals and the like are collected, so that the emotion recognition accuracy is influenced.
2. The existing emotion recognition method adopts PCA, LBP and other means to process characteristic values, has the problems of dimensionality randomization and experience of dimension reduction, incomplete image edge characteristic extraction information and the like.
3. The existing emotion recognition method adopts a linear fusion means for fusing characteristic values, has the problems of immobilization of different characteristic weights and the like, and does not consider the influence of each characteristic weight caused by character factors.
4. The existing research on the emotion changes of patients or normal people before and after acupuncture in clinical acupuncture is mostly a subjective evaluation mode such as scale evaluation and the like, and lacks of digitalized and quantized indexes.
Disclosure of Invention
The invention provides a multimode emotion analysis method and system for acupuncture based on deep learning, which are used for solving the technical problems in the prior art, and the method comprises the steps of combining facial expression of a human face and electroencephalogram characteristics, respectively extracting the characteristics, extracting the characteristics of the facial expression characteristics by Gabor wavelet transformation, extracting the characteristics of the electroencephalogram signals by UPLBP, reducing the dimension, performing sparse linear fusion on the facial expression characteristics and the electroencephalogram characteristics to form a uniform and normalized characteristic vector, converting the characteristic vector into a tensor form, inputting a CNN-LSTM network for training, removing redundant information, obtaining predicted emotion classification information, and calculating the reliability and accuracy of the network by comparing the predicted emotion classification information with actual emotion information. According to the invention, the expression characteristics and the electroencephalogram signal characteristics are fused, redundant information is removed, and the predicted emotion classification information is obtained through the training of the CNN-LSTM network, so that the emotion recognition accuracy is improved. Meanwhile, the method is combined with an acupuncture system, the change trend of the characteristics of the human face and the electroencephalogram is reflected by the emotion of a patient along with the progress of acupuncture treatment, and the effect evaluation of clinical treatment is facilitated.
The invention provides a multi-modal emotion analysis method based on deep learning for acupuncture, which comprises the following steps:
the method comprises the following steps: acquiring facial expression images of a patient during acupuncture, processing the acquired facial expression images, and extracting Gabor wavelet features of facial expressions through Gabor wavelet transformation;
step two: acquiring an electroencephalogram image of a patient during acupuncture, extracting texture features of the electroencephalogram image by adopting a uniform local binary pattern UPLBP, and performing dimension reduction processing on the image;
step three: performing multi-modal feature fusion on the features extracted in the first step and the second step by adopting a CNN-LSTM network, and performing emotion classification;
the invention adds LSTM network based on CNN network for fusion of multi-mode features, and aims to perform time-series processing on extracted image features. The emotion characteristics of the patient before, during and after treatment of acupuncture therapy are met. Because the LSTM network is a long-time memory network, the modeling of the problem containing the time sequence is facilitated.
Step four: and giving an acupuncture system operation suggestion according to the emotion classification result, and adjusting the acupuncture strength or frequency.
Preferably, the specific processing procedure of multi-modal feature fusion and emotion classification in step three comprises:
s001: fusing the Gabor wavelet characteristics of the facial expression image obtained in the step one and the characteristics of the electroencephalogram curve image obtained in the step two into a characteristic vector;
s002: converting the fused eigenvector into a tensor form, iterating by setting different values of the sample number batch _ size selected in one training, randomly taking out a training sample from each iteration training as input data of the CNN-LSTM network, and inputting the training sample into the CNN-LSTM network;
s003: adjusting the initial structure and network parameters of the CNN network, and performing iteration;
the iterative indexes are network precision and consumed time, and the initial structure and the network parameters of the CNN network which enable the network precision and the consumed time to be optimal are selected by adjusting the initial structure and the network parameters of the CNN network.
S004: extracting the characteristics of the picture in a CNN network through multilayer convolution pooling to obtain a five-dimensional tensor characteristic diagram;
s005: on the premise of not changing the numerical value in the characteristic diagram, converting the five-dimensional tensor characteristic diagram into a three-dimensional tensor characteristic diagram which meets the input requirement of the LSTM, and inputting the three-dimensional tensor characteristic diagram into an LSTM layer for processing;
s006: inputting the output of the LSTM layer into a full-connection layer and a function layer for SVM classification;
s007: obtaining emotion classification results of a one-dimensional array through SVM classification, and storing the trained neural network, wherein the one-dimensional array comprises predicted emotion classification information corresponding to samples after training;
training is carried out according to the emotion classification corresponding to the input feature vector of the SVM, and the loss function value of the emotion classification corresponding to the vector is the smallest, so that the feature is considered to be the class representing the emotion.
S008: and comparing the predicted emotion classification information with the actual emotion classification information to obtain the prediction accuracy of the trained neural network.
Preferably, the fusing into one feature vector specifically includes: weighting the extracted features of the images in different modes by adopting variable-weight sparse linear fusion to synthesize a feature vector, wherein a feature fusion weighting formula is expressed as follows:
O(x)＝γK(x)+(1-γ)F(x)………………………………(1)
wherein: k (x) represents the characteristics of the electroencephalogram curve image;
f (x) represents a facial expression feature;
gamma is an empirical weight coefficient of the influence of different characters on the electroencephalogram curve.
Preferably, in step S003, the adjusting the initial structure and the network parameter specifically includes: adjusting the number of layers of the convolutional layers in the CNN network initial structure and the network parameter learning rate;
preferably, the loss function adopted by the function layer is a softmax function;
the loss function here, using a Softmax function, also called normalized exponential function, is a generalization of the logistic function. It can "compress" a K-dimensional vector z containing arbitrary real numbers into another K-dimensional real vector σ (z) such that each element ranges between (0,1) and the sum of all elements is 1.
Preferably, in the first step, the facial expression feature extraction through Gabor wavelet transform includes:
extracting Gabor wavelet coefficients of facial expression feature points by convolution of the two-dimensional image to obtain a filter bank consisting of 2 scales and 2 directions;
the scale represents the number of effective pixel points extracted each time, and the direction represents whether to rotate by 0 degree or 90 degrees.
Taking the matching distance of the Gabor wavelet characteristics as a measurement standard of similarity to obtain facial expression characteristics;
the Gabor wavelet transform formula is as follows:
wherein:
s is the direction of the Gabor filter;
v is the dimension of the Gabor filter;
r ═ x, y represents the position of the extracted pixel;
σ is the ratio of the window width of the Gabor filter to the wavelength;
Ps,vis a wavelet vector defined as
Wherein:
wherein:
Pmaxis the maximum frequency;
f is a scale factor.
The invention provides a multi-modal emotion analysis system for acupuncture based on deep learning, which comprises:
the signal acquisition module is used for acquiring multi-modal emotion images of the person to be acupunctured, and the multi-modal emotion images comprise facial expression images and electroencephalogram images;
the signal characteristic extraction module is used for extracting a characteristic value of the acquired image;
the signal analysis module is used for analyzing the characteristic values extracted by the signal characteristic extraction module to obtain emotion changes and give operation suggestions;
and the signal execution module is used for executing the operation suggestion given by the signal analysis module.
Preferably, the signal feature extraction module extracts features of the acquired multi-modal emotion images, including Gabor bode features of facial expressions and texture features of electroencephalogram images, and performs dimension reduction.
Preferably, the signal analysis module analysis process is implemented by:
fusing the Gabor baud of the facial expression and the texture characteristics of the electroencephalogram curve image to synthesize a characteristic vector;
inputting the synthesized feature vector into a CNN-LSTM network to remove redundant information, and obtaining a classification result containing a trained predicted label value after training;
calculating the prediction accuracy of the CNN-LSTM network according to the classification result;
giving an acupuncture system operation suggestion according to the classification result, wherein the operation suggestion comprises the following steps: decreasing signal strength, decreasing signal frequency, increasing signal strength, or increasing signal frequency.
Preferably, the signal execution module performs the following operations according to the received operation suggestion of the signal analysis module:
when the operation suggestion is that the signal intensity is reduced, the signal execution module sends a signal with reduced intensity to the acupuncture system to control the acupuncture system to reduce the acupuncture intensity;
when the operation suggestion is that the signal frequency is reduced, the signal execution module sends a signal with the reduced frequency to the acupuncture system to control the acupuncture system to reduce the acupuncture frequency;
when the operation suggestion is that the signal intensity is increased, the signal execution module sends a signal with increased intensity to the acupuncture system to control the acupuncture system to increase the acupuncture intensity;
when the operation suggestion is to increase the signal frequency, the signal execution module sends a signal for increasing the frequency to the acupuncture system to control the acupuncture system to increase the acupuncture frequency.
Compared with the prior art, the invention has the following beneficial effects:
1. the method adopts the uniform local binary pattern UPLBP to extract the characteristics of the electroencephalogram signal, thereby solving the problems of other typesThe problem of excessive pattern binary patterns is that UPLBP reduces the dimension of the binary patterns, so that the types of the binary patterns are greatly reduced, but the stored texture feature data is not changed, but the total data amount is 2pThe number of seeds is reduced to p (p-1) +2, and the interference of noise is obviously weakened under the dimensionality reduction of the eigenvector.
2. The invention uses the CNN-LSTM network to perform characteristic level fusion on the facial expression image and the process electroencephalogram curve image, and fuses all characteristic data into a uniform and normalized characteristic vector in the fusion process, thereby eliminating the redundancy of information among all modes, and then the other characteristics form complementation, and reducing the interference of different characteristics.
3. According to the invention, through efficient sparse linear feature fusion and feedback of the recognition training accuracy, the weight ratio of the two types of features is continuously corrected, so that the extracted features can be combined more reasonably and efficiently, and the emotion classification recognition accuracy is improved.
4. The emotion recognition method is applied to the research on emotion changes of patients or normal people before and after acupuncture in clinical acupuncture, avoids interference of human factors, and is more intelligent and objective.
5. The invention directly extracts the visual characteristics of the curve, has the cognition of curve trend form, better accords with the cognition habit of people, and realizes the effect that the extracted characteristics are more beneficial to classification.
6. The invention realizes the advantage of extracting space sequence characteristics by using the LSTM by fusing multiple characteristics and inputting the fused characteristics into the LSTM for final classification after processing, thereby capturing the change trend of the characteristics of the human face and the brain electricity reflected by the emotion of a patient along with the progress of acupuncture treatment and being more beneficial to the effect evaluation of clinical treatment.
Drawings
FIG. 1 is a general design diagram of the multi-modal emotion analysis method of the present invention;
fig. 2a is a picture before UPLBP processing;
fig. 2b is a picture after UPLBP processing;
FIG. 3a is a graph of the contribution rate of each 10-dimensional information to feature extraction;
FIG. 3b is a graph of contribution rate versus dimensional change;
FIG. 4 is a schematic view of the multi-modal feature fusion of the present invention;
FIG. 5 is an example of a library of historical expression images;
FIG. 6 is a texture feature extracted for FIG. 5;
FIG. 7 is a historical electroencephalogram curve library;
FIG. 8a is a graph of accuracy in one training session, with the abscissa being the training step size;
FIG. 8b is a graph of error rate during one training session with training step size on the abscissa;
FIG. 9 is a comparison of different modality identification accuracy;
FIG. 10a is a confusion matrix of facial expressions versus emotion classification effect;
FIG. 10b is a confusion matrix of the emotion classification effect of the electroencephalogram signals;
FIG. 10c shows the effect of mixing the two modalities of facial expression and electroencephalogram on emotion classification;
fig. 11 is an overall flow chart of the present invention.
Detailed Description
The following detailed description of the present invention will be made with reference to the accompanying drawings 1-10.
The invention provides a multi-modal emotion analysis method based on deep learning for acupuncture, which comprises the following steps:
the method comprises the following steps: acquiring facial expression images of a patient during acupuncture, processing the acquired facial expression images, and extracting Gabor wavelet features of facial expressions through Gabor wavelet transformation;
step two: acquiring an electroencephalogram image of a patient during acupuncture, extracting texture features of the electroencephalogram image by adopting a uniform local binary pattern UPLBP, and performing dimension reduction processing on the image;
the brain electrical curve is the visual embodiment of human brain activity effect, and the image is the most direct visual stimulation input into the brain. Experts generally adjust their decisions according to the intuitive feeling of the electroencephalogram curve. Different from general image classification, the electroencephalogram curve has intuition, simplicity and clear texture characteristics.
The LBP principle is mainly to compare the difference between the pixel point and the adjacent pixel points, compare the gray characteristic values of the pixel points, perform thresholding processing, and perform a characteristic extraction process by using binary coding. In the range of 9 grids of 3 x 3, the central pixel point is set as a threshold value, the gray values of the surrounding eight pixel points are compared, binary coding is carried out, and the gray values are generally converted into decimal numbers, namely the LBP value of the central pixel point is obtained. The LBP has different values according to the central pixel point, and the corresponding LBP modes are also different. Considering the problems of calculation speed and storage capacity comprehensively, the invention uses a Uniform Local Binary Pattern (UPLBP) to process the picture and extract the texture characteristics of the electroencephalogram curve image. The uniform local binary pattern LBP characteristic is a typical method for extracting the texture characteristic of an image, and the computing method has the advantages of simple and understandable principle, high computing speed and strong classification capability; meanwhile, the lighting invariance is realized.
In order to solve the problem of excessive traditional LBP characteristic binary patterns and improve the statistics, the invention adopts a uniform local binary pattern UPLBP to reduce the dimension of the pattern types of an LBP operator, solves the problem of excessive jump types of pixel values from 0 to 1, also solves the sparsity of characteristic space, enables the network training process to be quicker, enables the dimension of a characteristic vector to be less by the UPLBP, and can reduce the influence caused by high-frequency noise. The image feature extraction effect is superior to LBP.
Step three: performing multi-modal feature fusion on the features extracted in the first step and the second step by adopting a CNN-LSTM network, and performing emotion classification;
the invention adds LSTM network based on CNN network for fusion of multi-mode features, and aims to perform time-series processing on extracted image features. The emotion characteristics of the patient before, during and after treatment of acupuncture therapy are met. Because the LSTM network is a long-time memory network, the modeling of the problem containing the time sequence is facilitated.
Step four: and giving an acupuncture system operation suggestion according to the emotion classification result, and adjusting the acupuncture strength or frequency.
As a preferred embodiment, the specific treatment process of step three includes:
the specific processing process of multi-modal feature fusion and emotion classification in the third step comprises the following steps:
s001: fusing the Gabor wavelet characteristics of the facial expression image obtained in the step one and the characteristics of the electroencephalogram curve image obtained in the step two into a characteristic vector;
s002: converting the fused eigenvector into a tensor form, iterating by setting different values of the sample number batch _ size selected in one training, randomly taking out a training sample from each iteration training as input data of the CNN-LSTM network, and inputting the training sample into the CNN-LSTM network;
s003: adjusting the initial structure and network parameters of the CNN network, and performing iteration;
the iterative indexes are network precision and consumed time, and the initial structure and the network parameters of the CNN network which enable the network precision and the consumed time to be optimal are selected by adjusting the initial structure and the network parameters of the CNN network.
S004: extracting the characteristics of the picture in a CNN network through multilayer convolution pooling to obtain a five-dimensional tensor characteristic diagram;
s005: on the premise of not changing the numerical value in the characteristic diagram, converting the five-dimensional tensor characteristic diagram into a three-dimensional tensor characteristic diagram which meets the input requirement of the LSTM, and inputting the three-dimensional tensor characteristic diagram into an LSTM layer for processing;
s006: inputting the output of the LSTM layer into a full-connection layer and a function layer for SVM classification;
s007: obtaining emotion classification results of a one-dimensional array through SVM classification, and storing the trained neural network, wherein the one-dimensional array comprises predicted emotion classification information corresponding to samples after training;
training is carried out according to the emotion classification corresponding to the input feature vector of the SVM, and the loss function value of the emotion classification corresponding to the vector is the smallest, so that the feature is considered to be the class representing the emotion.
S008: and comparing the predicted emotion classification information with the actual emotion classification information to obtain the prediction accuracy of the trained neural network.
As shown in figure 4, the invention uses the CNN-LSTM network to perform characteristic level fusion on the facial expression image and the electroencephalogram image. And respectively extracting uniform local binary pattern LBP characteristics from the electroencephalogram curve image, extracting Gabor characteristics from the facial expression image, and then performing multi-mode fusion. Because the features between each modality have redundancy problems, the redundancy of information between the modalities needs to be eliminated in the fusion process, so that the rest features form complementation, and the interference of different features is reduced. Meanwhile, the fusion mode can combine the extracted features more reasonably and efficiently, and the emotion classification and identification accuracy is improved. In the process of multi-modal information fusion, the fusion strategy based on the feature level is to extract emotional feature data under a single mode, then to adopt variable-weight sparse linear fusion to the feature data of all the modes to form a uniform and normalized feature vector, finally to carry out classification and identification of emotion through SVM classification, and the output result of SVM classification is used as the emotion type prediction result of the test sample.
As a preferred embodiment, the fusing into one feature vector specifically includes: weighting the extracted features of the images in different modes by adopting variable-weight sparse linear fusion to synthesize a feature vector, wherein a feature fusion weighting formula is expressed as follows:
O(x)＝γK(x)+(1-γ)F(x) (1)
wherein: k (x) represents the characteristics of the electroencephalogram curve image;
f (x) represents a facial expression feature;
gamma is an empirical coefficient.
As a preferred embodiment, in step S003, the adjusting the initial structure and the network parameter specifically includes: and adjusting the number of layers of the convolutional layers in the initial structure of the CNN network and the learning rate of network parameters.
In a preferred embodiment, the loss function adopted by the function layer is a softmax function;
the loss function here, using a Softmax function, also called normalized exponential function, is a generalization of the logistic function. It can "compress" a K-dimensional vector z containing arbitrary real numbers into another K-dimensional real vector σ (z) such that each element ranges between (0,1) and the sum of all elements is 1.
The Softmax function is defined as:
in a preferred embodiment, the facial expression feature extraction by Gabor wavelet transform in the first step includes:
extracting Gabor wavelet coefficients of facial expression feature points by convolution of the two-dimensional image to obtain a filter bank consisting of 2 scales and 2 directions;
taking the matching distance of the Gabor wavelet characteristics as a measurement standard of similarity to obtain facial expression characteristics;
the Gabor wavelet transform formula is as follows:
wherein:
s is the direction of the Gabor filter;
v is the dimension of the Gabor filter;
r ═ x, y represents the position of the extracted pixel;
σ is the ratio of the window width of the Gabor filter to the wavelength;
Ps,vis a wavelet vector defined as
Wherein:
wherein:
Pmaxis the maximum frequency;
f is a scale factor.
The purpose of feature extraction of facial images is to try to find the most appropriate representation of a human face, each person's facial features having unique information. The feature extraction of the facial expression image mainly extracts appearance features, and the main core is the texture features of the facial expression. Most typically based on Gabor wavelet characteristics or the like.
The Gabor wavelet transform is a typical appearance feature extraction method. Researches find that the two-dimensional Gabor wavelet has high conformity with human visual characteristics and high similarity to human visual field simulation. Moreover, the Gabor wavelet has wide adaptability, can enhance the local characteristics of key points in facial expressions, and can well describe the texture characteristics of images. And the extraction of the facial expression features of the human is mainly to extract the core and representative local texture features of each part, so that the Gabor features are adopted as the extracted features to be identified. And extracting Gabor wavelet coefficients of the feature points through convolution of the two-dimensional image, and taking the matching distance of the Gabor features as the measurement standard of the similarity.
The invention provides a multi-modal emotion analysis system for acupuncture based on deep learning, which comprises:
the signal acquisition module is used for acquiring a multi-modal emotion image of the person to be acupunctured;
the signal characteristic extraction module is used for extracting a characteristic value of the acquired image;
the signal analysis module is used for analyzing the characteristic values extracted by the signal characteristic extraction module to obtain emotion changes and give operation suggestions;
and the signal execution module is used for executing the operation suggestion given by the signal analysis module.
As a preferred embodiment, the signal feature extraction module extracts features of the acquired multi-modal emotion images, including Gabor bode features of facial expressions and texture features of electroencephalogram images, and performs dimension reduction.
As a preferred embodiment, the signal analysis module analysis process is implemented by:
fusing the Gabor baud of the facial expression and the texture characteristics of the electroencephalogram curve image to synthesize a characteristic vector;
inputting the synthesized feature vector into a CNN-LSTM network to remove redundant information, and obtaining a classification result containing a trained predicted label value after training;
calculating the prediction accuracy of the CNN-LSTM network according to the classification result;
giving an acupuncture system operation suggestion according to the classification result, wherein the operation suggestion comprises the following steps: decreasing signal strength, decreasing signal frequency, increasing signal strength, or increasing signal frequency.
As a preferred embodiment, the signal execution module performs the following operations according to the received operation suggestion of the signal analysis module:
when the operation suggestion is that the signal intensity is reduced, the signal execution module sends a signal with reduced intensity to the acupuncture system to control the acupuncture system to reduce the acupuncture intensity;
when the operation suggestion is that the signal frequency is reduced, the signal execution module sends a signal with the reduced frequency to the acupuncture system to control the acupuncture system to reduce the acupuncture frequency;
when the operation suggestion is that the signal intensity is increased, the signal execution module sends a signal with increased intensity to the acupuncture system to control the acupuncture system to increase the acupuncture intensity;
when the operation suggestion is to increase the signal frequency, the signal execution module sends a signal for increasing the frequency to the acupuncture system to control the acupuncture system to increase the acupuncture frequency.
Example 1
According to a specific embodiment of the present invention, the present invention provides a multi-modal emotion analysis method based on deep learning for acupuncture, which is described below by a set of experiments, wherein the experiments select 18 times of facial screenshots and electroencephalogram signal interception data segments for acupuncture treatment, and the method comprises the following steps:
the method comprises the following steps: acquiring facial expression images of a patient during acupuncture, processing the acquired facial expression images, and extracting Gabor wavelet features of facial expressions through Gabor wavelet transformation;
the facial expression images of the patient at the same time corresponding to the 18-time decision curve library are processed, and each picture is reduced to a uniform size, so that a historical facial expression image library is obtained, as shown in fig. 5.
The texture feature extraction is performed on the pictures in the historical facial expression image library, and the obtained atlas is shown in fig. 6.
Step two: extracting texture features of the electroencephalogram curve image by adopting a uniform local binary pattern UPLBP, and performing dimension reduction processing on the image;
the obtained electroencephalogram signal pictures are processed, and each picture is reduced to be in a uniform size, so that a historical electroencephalogram curve library is obtained, as shown in the attached figure 7.
And (3) extracting texture features after performing UPLBP processing on the pictures in the historical electroencephalogram curve library to obtain a texture feature atlas.
The above UPLBP processing includes performing dimension reduction and feature extraction on original image information, where the original image takes image information 1220, 620, and 3 as an example, and statistics of the image information after dimension reduction and feature extraction by the UPLBP are as follows (the contribution rate, i.e. the percentage of the features with different dimensions, covering the original information is shown in the table; for example, 3-dimensional features can cover 80% of the features of the original information, i.e. the contribution rate is 80%, 4-dimensional features can cover 95% of the features of the original information, i.e. the contribution rate is 95%, and finally, dimensions with high contribution rate are selected according to the total contribution rate and the total contribution rate of 10 dimensions):
TABLE 1 information contribution Rate per 10-D
The table data is plotted as shown in fig. 3a and 3 b. If we extract features of 60 dimensions, the first 10 dimensions cover 69.6% of the original information, the first 20 dimensions cover 84.6% of the original information, the first 30 dimensions cover 92.2% of the information, and the 60 dimensions cover 98.8% of the information. In order to save calculation overhead, the first 30 feature dimensions can be taken, and the features of the last 30 dimensions are ignored.
Step three: performing multi-modal feature fusion on the features extracted in the first step and the second step by adopting a CNN-LSTM network, and performing emotion classification;
the feature atlas is processed using a CNN-LSTM network. The picture is converted into a tensor, and the CNN can directly process the tensor of the picture and carry out convolution operation at the same time. The code execution environment is the Tensorflow1.14.0 version framework, the python version is 3.6.4, and the execution is based on a CPU.
The neural network structure CNN convolutional layer comprises 4 convolutional layers and 4 pooling layers, wherein the convolutional layers are alternated, and the size of the convolutional core is as follows: [3,3,1] and [1,1,1], the LSTM network portion consists of 25 network nodes, and the value of dropout is 1, and the choice of learning rate includes 0.01, 0.1 and 0.001. The size of the image input into the neural network CNN is 128 × 128 format, and the value of Batch _ size is 10, that is, 10 pictures are input in one training process.
Under the tenserflow framework, the flow of data is in the form of a tensor (tensor) whose data type is dtype, where the data type of each element in tensor is the same, such as tf.int8, tf.float32, etc. The process of circulating Data is called a Data Flow Graph, and includes Data (Data), Flow (Flow), and Graph (Graph).
The CNN-LSTM network used in the invention is trained by using emotional classification information, and parameters of neural network training are mainly referred to loss function (loss) and accuracy (accuracy). loss represents the difference between the predicted value and the accurate value, and accuracycacy represents the proportion of the predicted accurate value in the model predicted value. The larger the accuracy, the better, and the smaller the loss, the better.
The Softmax function is defined as:
wherein, thetaiAnd X is a column vector
X may be exchanged for the function of f (X).
In order to achieve the purpose that the loss function is continuously reduced, calculation is carried out from the output layer of the network layer by layer upwards through negative feedback adjustment of the network, the network is updated through parameters such as weight and offset of the loss function adjustment each time, and finally the loss function is stabilized through a certain number of iterations.
Through a plurality of times of experimental exploration, the following parameters are set: the batch size is 10, the Learning rate is 0.01, the iteration is performed 500 times, 128 × 128 format pictures are selected, and the test accuracy is 91.6%. The correct rate curve and error curve of a certain training in the neural network training process are shown in fig. 8a and fig. 8 b.
According to the accuracy and the error curve, when the iteration is carried out for about 100 times, the accuracy of the neural network reaches an expected value, and the error value is quickly reduced to a certain range, so that the training effectiveness of the neural network is reflected.
Example 2
The invention adopts a multi-mode feature fusion mode, the extracted features of different modal images are weighted by adopting variable-weight sparse linear fusion to synthesize a feature vector, and a feature fusion weighting formula is expressed as follows:
O(x)＝γK(x)+(1-γ)F(x) (1)
wherein: k (x) represents the characteristics of the electroencephalogram curve image;
f (x) represents a facial expression feature;
gamma is an empirical coefficient.
Table 2 shows the results of each single mode and the fusion of the two modes, and fig. 9 shows a graph of accuracy versus the use of different modes.
TABLE 2 Classification results of the modalities and after fusion
As can be seen from the attached figure 9, the effect of the single mode of facial expression on emotion classification is superior to that of an electroencephalogram signal curve, and the accuracy of emotion classification can be improved through the fusion of multi-mode images.
Example 3
In order to further study the complementary characteristics of the facial expression and the electroencephalogram signal curve, the embodiment analyzes the confusion matrix of each mode and reveals the advantages and the disadvantages of each mode.
Fig. 10a, 10b and 10c (each column in each figure represents predicted emotion classification information and each row represents actual emotion classification information.) present a confusion matrix based on facial expression and electroencephalogram signal graphs. Fig. 10a shows the emotion classification effect of facial expressions, fig. 10b shows the emotion classification effect of electroencephalogram signals, and fig. 10c shows the emotion classification effect of two mixed modalities. The experimental result shows that the facial expression and the electroencephalogram graph have different discriminative power on emotion recognition, the classification effect of the facial expression on the happy emotion is good, the classification effect of the two modes of the facial expression and the electroencephalogram signal on the calm emotion is poor, and classification errors are prone to occurring. However, the combination of the two modes realizes feature complementation, and the multi-mode fusion can obviously improve the classification precision of calm emotions.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (5)
1. A multi-modal emotion analysis method based on deep learning for acupuncture is characterized by comprising the following steps:
the method comprises the following steps: acquiring facial expression images of a patient during acupuncture, processing the acquired facial expression images, and extracting Gabor wavelet features of facial expressions through Gabor wavelet transformation;
step two: acquiring an electroencephalogram image of a patient during acupuncture, extracting texture features of the electroencephalogram image by adopting a uniform local binary pattern UPLBP, and performing dimension reduction processing on the image;
step three: performing multi-modal feature fusion on the features extracted in the first step and the second step by adopting a CNN-LSTM network, and performing emotion classification;
step four: giving an acupuncture system operation suggestion according to the emotion classification result, and adjusting acupuncture strength or frequency;
the specific processing process of multi-modal feature fusion and emotion classification in the third step comprises the following steps:
s001: fusing the Gabor wavelet characteristics of the facial expression image obtained in the step one and the characteristics of the electroencephalogram curve image obtained in the step two into a characteristic vector;
the merging into one feature vector specifically includes: weighting the extracted features of the images in different modes by adopting variable-weight sparse linear fusion to synthesize a feature vector, wherein a feature fusion weighting formula is expressed as follows:
O(x)＝γK(x)+(1-γ)F(x) (1)
wherein:
k (x) represents the characteristics of the electroencephalogram curve image;
f (x) represents a facial expression feature;
gamma is an empirical weight coefficient of the influence of different characters on the electroencephalogram curve;
s002: converting the fused eigenvector into a tensor form, iterating by setting different values of the sample number batch _ size selected in one training, randomly taking out a training sample from each iteration training as input data of the CNN-LSTM network, and inputting the training sample into the CNN-LSTM network;
s003: adjusting the initial structure and network parameters of the CNN network, and performing iteration; the adjusting of the initial structure and the network parameters specifically includes: adjusting the number of layers of the convolutional layers in the CNN network initial structure and the network parameter learning rate, and selecting the CNN network initial structure and the network parameters which enable the network precision and the consumed time to be optimal;
s004: extracting the characteristics of the picture in a CNN network through multilayer convolution pooling to obtain a five-dimensional tensor characteristic diagram;
s005: on the premise of not changing the numerical value in the characteristic diagram, converting the five-dimensional tensor characteristic diagram into a three-dimensional tensor characteristic diagram which meets the input requirement of the LSTM, and inputting the three-dimensional tensor characteristic diagram into an LSTM layer for processing;
s006: inputting the output of the LSTM layer into a full-connection layer and a function layer for SVM classification;
s007: through SVM classification, training is carried out according to emotion classification corresponding to an input feature vector of the SVM, the feature with the minimum loss function value of the emotion classification corresponding to the vector is selected to represent the emotion category, the emotion classification result of a one-dimensional array is obtained, and a trained neural network is stored, wherein the one-dimensional array comprises predicted emotion classification information corresponding to a sample after training;
s008: and comparing the predicted emotion classification information with the actual emotion classification information to obtain the prediction accuracy of the trained neural network, and continuously correcting the weight ratio of the characteristics of the electroencephalogram image and the facial expression characteristics in the characteristic fusion weighting formula according to the recognition accuracy.
2. The method of multimodal emotion analysis according to claim 1, wherein the loss function employed in the function layer is a softmax function.
3. The multimodal emotion analyzing method of claim 1, wherein in the first step, the facial expression feature extraction through Gabor wavelet transform comprises:
extracting Gabor wavelet coefficients of facial expression feature points by convolution of the two-dimensional image to obtain a filter bank consisting of 2 scales and 2 directions;
taking the matching distance of the Gabor wavelet characteristics as a measurement standard of similarity to obtain facial expression characteristics;
the Gabor wavelet transform formula is as follows:
wherein:
s is the direction of the Gabor filter;
v is the dimension of the Gabor filter;
r ═ x, y represents the position of the extracted pixel;
σ is the ratio of the window width of the Gabor filter to the wavelength;
Ps,vis a wavelet vector defined as
Wherein:
wherein:
Pmaxis the maximum frequency;
f is a scale factor.
4. A deep learning based multimodal emotion analysis system for acupuncture, comprising:
the signal acquisition module is used for acquiring multi-modal emotion images of the person to be acupunctured, and the multi-modal emotion images comprise facial expression images and electroencephalogram images;
the signal characteristic extraction module is used for extracting a characteristic value of the acquired image;
the signal feature extraction module is used for extracting the features of the collected multi-modal emotion images, including Gabor wavelet features of facial expressions and texture features of electroencephalogram images, and performing dimension reduction;
the signal analysis module is used for analyzing the characteristic values extracted by the signal characteristic extraction module to obtain emotion changes and give operation suggestions;
the signal analysis module analysis process is realized by the following operations:
fusing the Gabor baud of the facial expression and the texture characteristics of the electroencephalogram image to synthesize a characteristic vector, which specifically comprises the following steps: weighting the extracted features of the images in different modes by adopting variable-weight sparse linear fusion to synthesize a feature vector, wherein a feature fusion weighting formula is expressed as follows:
O(x)＝γK(x)+(1-γ)F(x) (1)
wherein:
k (x) represents the characteristics of the electroencephalogram curve image;
f (x) represents a facial expression feature;
gamma is an empirical weight coefficient of the influence of different characters on the electroencephalogram curve;
inputting the synthesized feature vector into a CNN-LSTM network to remove redundant information, performing training and SVM classification, performing training according to the emotion classification corresponding to the input feature vector of the SVM, selecting the feature with the minimum loss function value of the emotion classification corresponding to the vector to represent the class of the emotion, and obtaining a classification result containing a trained prediction label value;
calculating the prediction accuracy of the CNN-LSTM network according to the classification result, and continuously correcting the weight ratio of the characteristics of the electroencephalogram image and the facial expression characteristics in the characteristic fusion weighting formula according to the identified training accuracy;
giving an acupuncture system operation suggestion according to the classification result, wherein the operation suggestion comprises the following steps: decreasing signal strength, decreasing signal frequency, increasing signal strength, or increasing signal frequency;
and the signal execution module is used for executing the operation suggestion given by the signal analysis module.
5. The multimodal emotion analysis system of claim 4, wherein the signal execution module, based on the received operating recommendation from the signal analysis module, performs the following:
when the operation suggestion is that the signal intensity is reduced, the signal execution module sends a signal with reduced intensity to the acupuncture system to control the acupuncture system to reduce the acupuncture intensity;
when the operation suggestion is that the signal frequency is reduced, the signal execution module sends a signal with the reduced frequency to the acupuncture system to control the acupuncture system to reduce the acupuncture frequency;
when the operation suggestion is that the signal intensity is increased, the signal execution module sends a signal with increased intensity to the acupuncture system to control the acupuncture system to increase the acupuncture intensity;
when the operation suggestion is to increase the signal frequency, the signal execution module sends a signal for increasing the frequency to the acupuncture system to control the acupuncture system to increase the acupuncture frequency.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010724202.6A CN111881812B (en) | 2020-07-24 | 2020-07-24 | Multi-modal emotion analysis method and system based on deep learning for acupuncture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010724202.6A CN111881812B (en) | 2020-07-24 | 2020-07-24 | Multi-modal emotion analysis method and system based on deep learning for acupuncture |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111881812A CN111881812A (en) | 2020-11-03 |
| CN111881812B true CN111881812B (en) | 2021-10-08 |
Family
ID=73200641
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010724202.6A Active CN111881812B (en) | 2020-07-24 | 2020-07-24 | Multi-modal emotion analysis method and system based on deep learning for acupuncture |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111881812B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112244875A (en) * | 2020-11-10 | 2021-01-22 | 天津职业技术师范大学(中国职业培训指导教师进修中心) | Schizophrenia detection method based on electroencephalogram and deep learning |
| CN113143274B (en) * | 2021-03-31 | 2023-11-10 | 中国科学院心理研究所 | Emotion early warning method based on camera |
| CN113180696A (en) * | 2021-04-28 | 2021-07-30 | 北京邮电大学 | Intracranial electroencephalogram detection method and device, electronic equipment and storage medium |
| CN114305415B (en) * | 2021-11-25 | 2023-10-24 | 广东电网有限责任公司 | Cross-test and cross-mode multi-mode tension emotion recognition method and system |
| CN114366102B (en) * | 2022-01-05 | 2024-03-01 | 广东电网有限责任公司 | Multi-mode tension emotion recognition method, device, equipment and storage medium |
| CN114464326A (en) * | 2022-01-12 | 2022-05-10 | 山东大学 | Coronary heart disease prediction system based on multi-mode carotid artery data |
| CN114403877A (en) * | 2022-01-21 | 2022-04-29 | 中山大学 | Multi-physiological-signal emotion quantitative evaluation method based on two-dimensional continuous model |
| WO2023148471A1 (en) * | 2022-02-07 | 2023-08-10 | Cogitat Ltd. | Classification of brain activity signals |
| CN114565964A (en) * | 2022-03-03 | 2022-05-31 | 网易（杭州）网络有限公司 | Emotion recognition model generation method, recognition method, device, medium and equipment |
| CN114699080B (en) * | 2022-04-28 | 2023-04-25 | 电子科技大学 | Driver mental stress degree identification method based on fusion characteristics |
| CN115486820B (en) * | 2022-11-16 | 2023-03-17 | 安徽星辰智跃科技有限责任公司 | Method, system and device for detecting and quantifying acupuncture curative effect and physiological effect |
| CN116313029B (en) * | 2022-12-22 | 2023-08-08 | 安徽星辰智跃科技有限责任公司 | Method, system and device for dynamic control optimization of digital acupuncture |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106169073A (en) * | 2016-07-11 | 2016-11-30 | 北京科技大学 | A kind of expression recognition method and system |
| CN110623658A (en) * | 2019-09-24 | 2019-12-31 | 京东方科技集团股份有限公司 | Signal processing method, signal processing apparatus, medical device, and storage medium |
| CN110898326A (en) * | 2019-12-18 | 2020-03-24 | 深圳市中医院 | Control method for controlling output frequency of electro-acupuncture instrument by applying brain waves |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2001039664A1 (en) * | 1999-12-02 | 2001-06-07 | The General Hospital Corporation | Method and apparatus for measuring indices of brain activity |
-
2020
- 2020-07-24 CN CN202010724202.6A patent/CN111881812B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106169073A (en) * | 2016-07-11 | 2016-11-30 | 北京科技大学 | A kind of expression recognition method and system |
| CN110623658A (en) * | 2019-09-24 | 2019-12-31 | 京东方科技集团股份有限公司 | Signal processing method, signal processing apparatus, medical device, and storage medium |
| CN110898326A (en) * | 2019-12-18 | 2020-03-24 | 深圳市中医院 | Control method for controlling output frequency of electro-acupuncture instrument by applying brain waves |
Non-Patent Citations (2)
| Title |
|---|
| 基于加权融合策略的情感识别建模方法研究;魏薇;《中国博士学位论文全文数据库 信息科技辑》;20190815(第8期);第I136-14页 * |
| 基于脑电特征纹理与SVM分类的癫痫发作检测研究;孙成法 等;《电脑知识与技术》;20190531;第15卷(第14期);第166-168页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111881812A (en) | 2020-11-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111881812B (en) | Multi-modal emotion analysis method and system based on deep learning for acupuncture | |
| CN106682616B (en) | Method for recognizing neonatal pain expression based on two-channel feature deep learning | |
| Zhang et al. | Multi-person brain activity recognition via comprehensive EEG signal analysis | |
| CN113627518A (en) | Method for realizing multichannel convolution-recurrent neural network electroencephalogram emotion recognition model by utilizing transfer learning | |
| CN114052735B (en) | Deep field self-adaption-based electroencephalogram emotion recognition method and system | |
| Abbas et al. | DeepMI: Deep learning for multiclass motor imagery classification | |
| CN112656427A (en) | Electroencephalogram emotion recognition method based on dimension model | |
| CN110555468A (en) | Electroencephalogram signal identification method and system combining recursion graph and CNN | |
| Prasetio et al. | The facial stress recognition based on multi-histogram features and convolutional neural network | |
| CN111797674B (en) | MI electroencephalogram signal identification method based on feature fusion and particle swarm optimization algorithm | |
| Li et al. | EEG signal classification method based on feature priority analysis and CNN | |
| CN112990008B (en) | Emotion recognition method and system based on three-dimensional characteristic diagram and convolutional neural network | |
| CN114533086A (en) | Motor imagery electroencephalogram decoding method based on spatial domain characteristic time-frequency transformation | |
| Eslahi et al. | A GA-based feature selection of the EEG signals by classification evaluation: Application in BCI systems | |
| Song et al. | Common spatial generative adversarial networks based EEG data augmentation for cross-subject brain-computer interface | |
| Djamal et al. | Brain-computer interface of focus and motor imagery using wavelet and recurrent neural networks | |
| Subasi et al. | The Ensemble Machine Learning‐Based Classification of Motor Imagery Tasks in Brain‐Computer Interface | |
| Huang et al. | Robust multi-feature collective non-negative matrix factorization for ECG biometrics | |
| CN115422973A (en) | Electroencephalogram emotion recognition method of space-time network based on attention | |
| CN117473303B (en) | Personalized dynamic intention feature extraction method and related device based on electroencephalogram signals | |
| CN114220164A (en) | Gesture recognition method based on variational modal decomposition and support vector machine | |
| Hu et al. | MSATNet: multi-scale adaptive transformer network for motor imagery classification | |
| Tang et al. | Dynamic pruning group equivariant network for motor imagery EEG recognition | |
| CN114419657A (en) | ECG identity recognition method based on local segment sparse representation | |
| Sokhal et al. | Classification of EEG signals using empirical mode decomposition and lifting wavelet transforms |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |