CN110956981B - Speech emotion recognition method, device, equipment and storage medium - Google Patents
Speech emotion recognition method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN110956981B CN110956981B CN201911246544.5A CN201911246544A CN110956981B CN 110956981 B CN110956981 B CN 110956981B CN 201911246544 A CN201911246544 A CN 201911246544A CN 110956981 B CN110956981 B CN 110956981B
- Authority
- CN
- China
- Prior art keywords
- feature
- preset
- data
- voice
- emotion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 230000008909 emotion recognition Effects 0.000 title claims abstract description 77
- 238000000034 method Methods 0.000 title claims abstract description 74
- 230000008451 emotion Effects 0.000 claims abstract description 143
- 238000012360 testing method Methods 0.000 claims abstract description 43
- 238000013145 classification model Methods 0.000 claims abstract description 30
- 238000012545 processing Methods 0.000 claims abstract description 28
- 230000011218 segmentation Effects 0.000 claims abstract description 23
- 239000012634 fragment Substances 0.000 claims abstract description 22
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 20
- 238000005457 optimization Methods 0.000 claims abstract description 20
- 238000012549 training Methods 0.000 claims description 110
- 230000006870 function Effects 0.000 claims description 30
- 238000000605 extraction Methods 0.000 claims description 24
- 238000004364 calculation method Methods 0.000 claims description 11
- 238000012216 screening Methods 0.000 claims description 5
- 230000002093 peripheral effect Effects 0.000 claims 2
- 230000008569 process Effects 0.000 description 12
- 239000011159 matrix material Substances 0.000 description 7
- 230000001149 cognitive effect Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 5
- 238000013075 data extraction Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000013507 mapping Methods 0.000 description 4
- 230000007935 neutral effect Effects 0.000 description 4
- 230000003595 spectral effect Effects 0.000 description 4
- 239000013598 vector Substances 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000002996 emotional effect Effects 0.000 description 3
- 206010022998 Irritability Diseases 0.000 description 2
- 241001014642 Rasta Species 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003472 neutralizing effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
- G10L25/63—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for estimating an emotional state
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Computational Linguistics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- General Health & Medical Sciences (AREA)
- Child & Adolescent Psychology (AREA)
- Psychiatry (AREA)
- Hospice & Palliative Care (AREA)
- Signal Processing (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Machine Translation (AREA)
Abstract
The invention belongs to the technical field of voice signal processing and mode recognition, and discloses a voice emotion recognition method, a device, equipment and a storage medium. The method comprises the following steps: acquiring a test voice sample with a preset dimension, and performing segmentation processing on the test voice sample through a preset rule to obtain a plurality of initial voice samples; extracting signal characteristic data of the initial voice sample to obtain voice signal characteristic data to be processed; performing feature statistics on the feature data of the voice signal to be processed through a preset statistical function to obtain a feature statistical result to be confirmed; obtaining feature target data through a preset multi-target optimization algorithm according to the feature statistical result to be confirmed; and inputting the characteristic target data into a preset Softmax classification model to obtain a speech emotion recognition result. Through the method, the speech emotion fragments form speech emotion data, and the speech emotion data are input into the preset Softmax classification model, so that speech emotion can be better recognized.
Description
Technical Field
The present invention relates to the field of speech signal processing and pattern recognition technologies, and in particular, to a speech emotion recognition method, apparatus, device, and storage medium.
Background
There are many speech emotion recognition methods, but these methods do not notice that human speech emotion expressions have short-term and local characteristics. For example, in speech emotion recognition, the first half of a sentence, one word is angry and the whole sentence is angry. Several problems arise: first, identifying emotion using whole sentence often dilutes the characteristic changes of emotion. For example, "do we know to Beijing tomorrow and you feel feasible? "this sentence is often the second half of the sentence showing a large emotional difference. Feature variations that cause the use of mean pooling for time, convolution and fully connected layers for all features in deep learning to dilute emotion; secondly, when the parts are combined into sentences, the emotional characteristic changes are neutralized frequently. As is well known, chinese tones have one to four tones, wherein two and four tones have completely opposite characteristics in terms of time variation. Leading to the use of mean pooling for time in deep learning, neutralizing emotional feature changes for time series attention layers, and the like; thirdly, the positions of the words composing the emotion in the sentence are not fixed, which causes great difference of characteristics with the emotion. For example, "do this work? Is "and" feasible? Thus! "express the same meaning, but the output characteristics of the prior convolutional neural network are completely different.
The above is only for the purpose of assisting understanding of the technical aspects of the present invention, and does not represent an admission that the above is prior art.
Disclosure of Invention
The invention mainly aims to provide a speech emotion recognition method, a speech emotion recognition device, speech emotion recognition equipment and a storage medium, and aims to solve the technical problem of how to accurately recognize speech emotion.
In order to achieve the above object, the present invention provides a speech emotion recognition method, which comprises the following steps:
acquiring a test voice sample with a preset dimension, and performing segmentation processing on the test voice sample through a preset rule to obtain a plurality of initial voice samples;
extracting signal characteristic data of the initial voice sample to obtain voice signal characteristic data to be processed;
performing feature statistics on the voice signal feature data to be processed through a preset statistical function to obtain a feature statistical result to be confirmed;
obtaining feature target data through a preset multi-target optimization algorithm according to the feature statistical result to be confirmed;
and inputting the characteristic target data into a preset Softmax classification model to obtain a speech emotion recognition result.
Preferably, before the step of obtaining a test speech sample with a preset dimension and performing segmentation processing on the test speech sample through a preset rule to obtain a plurality of initial speech samples, the method further includes:
acquiring training voice samples with preset dimensions, and performing segmentation processing on the test voice samples through preset rules to obtain a plurality of initial training voice samples;
performing feature extraction on the initial training voice sample to obtain the features of a training voice signal to be processed;
performing feature statistics on the training voice signal features to be processed through a preset statistical function to obtain a training feature statistical result to be confirmed;
obtaining target training characteristic data through a preset multi-objective optimization algorithm according to the statistical result of the training characteristics to be confirmed;
acquiring emotion types corresponding to the target training characteristic data according to the target training characteristic data;
and establishing a preset Softmax classification model according to the emotion classes and target training characteristic data corresponding to the emotion classes.
Preferably, the step of obtaining target training feature data through a preset multi-objective optimization algorithm according to the statistical result of the training features to be confirmed includes:
carrying out emotion category division on the statistical result of the training features to be confirmed to obtain training feature data to be optimized corresponding to different emotion categories;
and obtaining target training characteristic data through a preset multi-target optimization algorithm according to the training characteristic data to be optimized.
Preferably, the step of inputting the feature target data into a preset Softmax classification model to obtain a speech emotion recognition result includes:
inputting the characteristic target data into the preset Softmax classification model to obtain speech emotion category data;
performing data statistics on the voice emotion type data to obtain a voice emotion type data value;
and obtaining a voice emotion recognition result according to the voice emotion category data value.
Preferably, the step of obtaining the speech emotion recognition result according to the speech emotion category data value includes:
judging whether the voice emotion type data value belongs to a preset voice emotion type threshold range or not;
and if the voice emotion type data value belongs to the preset voice emotion type threshold range, acquiring a voice emotion recognition result according to the voice emotion type data value.
Preferably, after the step of determining whether the speech emotion category data value belongs to a preset speech emotion category threshold range, the method further includes:
and if the voice emotion category data value does not belong to the preset voice emotion category threshold range, returning to the step of inputting the feature target data into the preset Softmax classification model to obtain voice emotion category data.
Preferably, the step of performing feature statistics on the to-be-processed speech signal feature data through a preset statistical function to obtain a to-be-confirmed feature statistical result includes:
screening the voice signal characteristic data to be processed to obtain label sample characteristic data;
and carrying out feature statistics on the tag sample feature data through a preset statistical function to obtain a feature statistical result to be confirmed.
In addition, to achieve the above object, the present invention further provides a speech emotion recognition apparatus, including: the system comprises an acquisition module, a processing module and a control module, wherein the acquisition module is used for acquiring a test voice sample with a preset dimensionality and performing segmentation processing on the test voice sample through a preset rule to obtain a plurality of initial voice samples;
the extraction module is used for extracting signal characteristic data of the initial voice sample to obtain voice signal characteristic data to be processed;
the statistical module is used for carrying out feature statistics on the voice signal feature data to be processed through a preset statistical function to obtain a feature statistical result to be confirmed;
the calculation module is used for obtaining feature target data through a preset multi-target optimization algorithm according to the feature statistical result to be confirmed;
and the determining module is used for inputting the characteristic target data into a preset Softmax classification model to obtain a speech emotion recognition result.
In addition, to achieve the above object, the present invention also provides an electronic device, including: a memory, a processor and a speech emotion recognition program stored on the memory and executable on the processor, the speech emotion recognition program being configured to implement the steps of the speech emotion recognition method as described in any of the above.
In addition, to achieve the above object, the present invention further provides a storage medium, on which a speech emotion recognition program is stored, and the speech emotion recognition program implements the steps of the speech emotion recognition method as described in any one of the above when executed by a processor.
The method comprises the steps of firstly obtaining a test voice sample with a preset dimensionality, carrying out segmentation processing on the test voice sample through a preset rule to obtain a plurality of initial voice samples, then carrying out signal characteristic data extraction on the initial voice samples to obtain voice signal characteristic data to be processed, screening the voice signal characteristic data to be processed to obtain tag sample characteristic data, carrying out characteristic statistics on the tag sample characteristic data through a preset statistical function to obtain a characteristic statistical result to be confirmed, then obtaining characteristic target data through a preset multi-target optimization algorithm according to the characteristic statistical result to be confirmed, and finally inputting the characteristic target data into a preset Softmax classification model to obtain a voice emotion recognition result. By the method, the voice emotion fragments and the emotion relation between the sentences and the fragments can be fully utilized and converted into the voice emotion data, so that the voice emotion recognition effect is improved.
Drawings
Fig. 1 is a schematic structural diagram of an electronic device in a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a first embodiment of a speech emotion recognition method according to the present invention;
FIG. 3 is a flowchart illustrating a speech emotion recognition method according to a second embodiment of the present invention;
FIG. 4 is a block diagram of a speech emotion recognition apparatus according to a first embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Referring to fig. 1, fig. 1 is a schematic structural diagram of an electronic device in a hardware operating environment according to an embodiment of the present invention.
As shown in fig. 1, the electronic device may include: a processor 1001, such as a Central Processing Unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a WIreless interface (e.g., a WIreless-FIdelity (WI-FI) interface). The Memory 1005 may be a Random Access Memory (RAM) Memory, or may be a Non-Volatile Memory (NVM), such as a disk Memory. The memory 1005 may alternatively be a storage device separate from the processor 1001.
Those skilled in the art will appreciate that the configuration shown in fig. 1 does not constitute a limitation of the electronic device and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a storage medium, may include an operating system, a network communication module, a user interface module, and a speech emotion recognition program.
In the electronic apparatus shown in fig. 1, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 of the electronic device according to the present invention may be disposed in the electronic device, and the electronic device calls the speech emotion recognition program stored in the memory 1005 through the processor 1001 and executes the speech emotion recognition method according to the embodiment of the present invention.
An embodiment of the present invention provides a speech emotion recognition method, and referring to fig. 2, fig. 2 is a flowchart illustrating a first embodiment of the speech emotion recognition method according to the present invention.
In this embodiment, the speech emotion recognition method includes the following steps:
step S10: the method comprises the steps of obtaining a test voice sample with preset dimensionality, and conducting segmentation processing on the test voice sample through a preset rule to obtain a plurality of initial voice samples.
Before the step of obtaining a test voice sample with a preset dimension, and performing segmentation processing on the test voice sample through a preset rule to obtain a plurality of initial voice samples, obtaining a training voice sample with a preset dimension, performing segmentation processing on the test voice sample through a preset rule to obtain a plurality of initial training voice samples, performing feature extraction on the initial training voice samples to obtain training voice signal features to be processed, performing feature statistics on the training voice signal features to be processed through a preset statistical function to obtain training feature statistical results to be confirmed, obtaining target training feature data through a preset multi-objective optimization algorithm according to the training feature statistical results to be confirmed, and obtaining an emotion category corresponding to the target training feature data according to the target training feature data, and establishing a preset Softmax classification model according to the emotion classes and target training characteristic data corresponding to the emotion classes.
In addition, it should be understood that the preset rule is a user-defined sample division rule, that is, if the duration of the obtained test voice sample of the preset dimension is 5s, and the preset rule is set to 0.2s, 25 segments of 0.2s initial voice samples are obtained after division according to the preset rule.
In addition, it should be noted that the preset dimension may be a time dimension, or may be a non-time dimension, and the present embodiment is not limited thereto.
Step S20: and extracting signal characteristic data of the initial voice sample to obtain voice signal characteristic data to be processed.
Furthermore, it should be understood that performing signal feature data on the initial speech sample extracts Mel Frequency Cepstral Coefficients (MFCC), Log Frequency Power Coefficients (Log Frequency coeffients, LFPC), Linear Predictive Cepstral Coefficients (LPCC), Zero Crossing with Peak Amplitude (zca), Perceptual Linear Prediction (PLP), Rasta filter Perceptual Linear prediction (R-PLP).
It should be understood that the above feature extraction result of each type of feature is a two-dimensional matrix, wherein one dimension is a time dimension, and then each type of feature F is calculatediFirst derivative in the time dimension Δ FiSecond derivative Δ Δ FiConnecting the original features, the first derivative result and the second derivative result in series in a non-time dimension to form a final feature extraction result of each type of features; and (4) connecting the final feature extraction results of the features of all the classes in series on a non-time dimension to obtain the feature extraction result of the sample.
Further, for ease of understanding, the following is exemplified:
suppose, MFCC corresponds to FMFCC∈R39×z,ΔFMFCC∈R39×z,ΔΔFi∈R39×zWherein z is the number of frames, i.e. the number of degrees of the time dimension, the concatenation result in the non-time dimension
Furthermore, it should be understood that at each time of speech signal feature extraction, MFCC, LFPC, LPCC, ZCPA, PLP, R-PLP features are extracted, where the number of Mel filters of MFCC, LFPC is 40; the linear prediction orders of the LPCC, the PLP and the R-PLP are respectively 12, 16 and 16; the frequency segmentation of the ZCAP is as follows: 0,106, 223, 352, 495, 655, 829, 1022, 1236, 1473, 1734, 2024, 2344, 2689, 3089, 3522, 4000. So that the dimensions of each class of features of each statement are respectively: ti 39, ti 40, ti 12, ti 16, where ti is the number of frames in the i-th sentence, and the number following the multiplier is the dimension of each frame feature. To obtain the change of the speech signal in the time dimension, a first derivative, a second derivative, is also calculated for the above features in the time dimension. Finally, the dimensionality of each type of feature is respectively as follows: ti 117, ti 140, ti 36, ti 48. The extracted speech signal features of the ith sample are combined from all the above features, and have a dimension ti (117+140+36+48+48+48).
Step S30: and carrying out feature statistics on the voice signal feature data to be processed through a preset statistical function to obtain a feature statistical result to be confirmed.
The statistical results of the above features in the time dimension are obtained using a statistical function using a mean (mean), a standard deviation (standard deviation), a minimum (min), a maximum (max), a kurtosis (kurtosis), and a skewness (skewness).
In addition, it should be understood that the tag sample feature data is obtained by screening the statistical results obtained above, feature statistics is performed on the tag sample feature data through a preset statistical function to obtain a feature statistical result to be confirmed, and the feature statistical result of the tag sample is recorded as { x }1,x2,...,xnAnd n is the number of labeled specimens.
Step S40: and obtaining characteristic target data through a preset multi-target optimization algorithm according to the statistical result of the characteristics to be confirmed.
In addition, it should be noted that { x in the above step1,x2,...,xnDivide by statement label into XA=[x1,x2,…,xm],XB=[xm+1,xm+2,…,xn]Wherein X isAIs a fragment of class A emotion, XBIs a segment of B-type emotion, and the training of the sentence segment emotion classification method based on tendency cognitive learning comprises the following steps:
(1) for X ∈ XAIs mixing XAThe angle of the sample within the X-centered Parzen window from the center sample X is divided into bins and X is then calculated at X using the following equationAThe distribution characteristics of the surrounding data.
bx=[b1,b2,…,bk]
In the formula bjDenotes the jth box, 1 (x)i∈Xj) At xiBelong to XjValue of time is 1 otherwise 0, XjIs XAA subset of (2), XjThe angle between the sample inside and x is distributed in the jth bin.
(2) For X ∈ XAIs mixing XBThe angle of the sample within the X-centered Parzen window from the center sample X is divided into bins and X is then calculated at X using the following equationAThe distribution characteristics of the surrounding data.
In the formulaIs shown asA box, 1 (x)i∈Xj) At xiBelong to XjValue of time is 1 otherwise 0, XjIs XBA subset of (2), XjThe angle between the sample inside and x is distributed in the jth bin.
(3) The difference in data distribution of the two data sets around point x is calculated using the following equation:
in the formulaThe distance between two vectors is represented, where the euclidean distance is used.
(4) According to the calculation result of the last step, the fragment set which is inclined to the A feeling can be obtainedSegment set inclined to B emotionAnd a collection of segments that are prone to neutral emotionWhereinIs dx>X of T.Is dx<-x of T.Is T>dx>-T. T is a threshold value set autonomously. For each set, clustering into a plurality of regions by using a spectral clustering method to obtain a region label of each segment xi
(5) Definition ofL=[LA,LB,LC]Wherein L isA∈Rp、LB∈Rq、LC∈RuP, q and u are each independentlyAndnumber of samples, LA、LBAnd LCThe element values in (1) are 1, 2 and 3 respectively. The feature subspace of the segment is learned using the following objective equation:
J=J1(oi,oj)+β*J2(oi,oj)
beta is an equilibrium parameter. Wherein J1(oi,oj) Can realizeAndthe intra-class distance between three classes is small, and the inter-class distance is large, and is defined as follows:
in the formula oiAnd ojIs composed ofAndresults after mapping to subspace. liAnd ljCorresponds to oiAnd ojThe value in L. m is a threshold value, the range of the distance between classes is adjusted, and the value of 1 is taken in the invention. GijIs xi and xjThe gaussian distance between. The calculation formula is as follows:
J2(oi,oj) It is possible to try to keep the relative relationship within each region constant and the regions belonging to the same class relatively close but not overlapping. The definition is as follows:
in the formulaAndare the region labels for xi and xj. GliIs aiClass all GijMaximum value of (2). The method can keep the relationship between two segments when the two segments belong to the same area, and minimize the distance between the two segments with a small weight when the two segments do not belong to the same area but belong to the same category, so that the two areas can be prevented from overlapping as much as possible.
To optimize the objective equation J, we define oi=φ(Wqφ(…W3φ(W2φ(W1xi+b1)+b2)+b3)+b4) Where phi (·) is a sigmoid function, W1,W2,…,WqTo map the matrix, b1,b2,…,bqIs an offset. By calculatingAndcan obtain W1,W2,…,WqAnd b1,b2,…,bqThe value of (a) is,is to take the derivative of J with respect to W,is the derivative of J over b.
Step S50: and inputting the characteristic target data into a preset Softmax classification model to obtain a speech emotion recognition result.
Further, it is understood that W is obtained according to the above steps1,W2,…,WqAnd b1,b2,…,bqCalculating { x1,x2,...,xmFeature selection result z.
In addition, W is defined as1,W2,…,WqAnd b1,b2,…,bqIs the characteristic target data in the present application.
Further, it should be understood that { x } is obtained separately using a preset Softmax classifier obtained during training1,x2,...,xmSpeech emotion classification of { l }1,l2,...,lm}. Then according to { l1,l2,...,lmAnd voting to obtain the emotion of the statement.
In addition, it should be noted that the feature target data is input into the preset Softmax classification model to obtain speech emotion category data, data statistics is performed on the speech emotion category data to obtain a speech emotion category data value, and a speech emotion recognition result is obtained according to the speech emotion category data value.
In addition, the step of obtaining the speech emotion recognition result according to the speech emotion category data value is to determine whether the speech emotion category data value belongs to a preset speech emotion category threshold range, and if the speech emotion category data value belongs to the preset speech emotion category threshold range, obtain the speech emotion recognition result according to the speech emotion category data value; and if the voice emotion category data value does not belong to the preset voice emotion category threshold range, returning to the step of inputting the feature target data into the preset Softmax classification model to obtain voice emotion category data.
In addition, the corpus used for emotion recognition effect evaluation according to the present invention is a standard database in the speech emotion recognition field. The training process is first completed and then the recognition test is performed. The test mode was performed in a 5-fold crossover fashion. 7 emotions of anger, fear, irritability, disgust, happiness, neutrality and sadness can be identified, the average classification accuracy is 94.65% under the condition that the speaker depends on the emotion recognition method, and the distinction degree between other emotions is better except that the emotion recognition method is easier to confuse with anger and anger. The average classification accuracy was 89.30% with speaker independence.
In the embodiment, a test voice sample with a preset dimension is obtained, the test voice sample is subjected to sectional processing through a preset rule to obtain a plurality of initial voice samples, then signal characteristic data extraction is carried out on the initial voice samples to obtain voice signal characteristic data to be processed, the voice signal characteristic data to be processed is screened to obtain label sample characteristic data, characteristic statistics is carried out on the label sample characteristic data through a preset statistical function to obtain a characteristic statistical result to be confirmed, then emotion classification is carried out on the training characteristic statistical result to be confirmed to obtain training characteristic data to be optimized corresponding to different emotion classifications, target training characteristic data is obtained through a preset multi-objective optimization algorithm according to the training characteristic data to be optimized, and finally the characteristic target data is input into a preset Softmax classification model, and obtaining a speech emotion recognition result. By the method, the voice emotion fragments and the emotion relation between the sentences and the fragments can be fully utilized to form the data with a tendency, so that the process of processing the tendency of a human can be simulated, the unbalanced information of the data is utilized, the data are compared with each other, the fragments with different emotions are separated under constraint conditions, and the sample scale is increased and the sample diversity is improved.
Referring to fig. 3, fig. 3 is a flowchart illustrating a speech emotion recognition method according to a second embodiment of the present invention.
Based on the first embodiment, before the step S10, the speech emotion recognition method in this embodiment further includes:
step S000: and acquiring training voice samples with preset dimensions, and performing segmentation processing on the test voice samples through preset rules to obtain a plurality of initial training voice samples.
Step S001: and extracting the characteristics of the initial training voice sample to obtain the characteristics of the training voice signal to be processed.
Step S002: and carrying out feature statistics on the training voice signal features to be processed through a preset statistical function to obtain a training feature statistical result to be confirmed.
Step S003: and obtaining target training characteristic data through a preset multi-objective optimization algorithm according to the statistical result of the training characteristics to be confirmed.
Step S004: and acquiring the emotion types corresponding to the target training characteristic data according to the target training characteristic data.
Step S005: and establishing a preset Softmax classification model according to the emotion classes and target training characteristic data corresponding to the emotion classes.
In addition, it should be noted that, the step of obtaining the target training feature data through the preset multi-objective optimization algorithm according to the to-be-confirmed training feature statistical result includes performing emotion class division on the to-be-confirmed training feature statistical result to obtain the to-be-optimized training feature data corresponding to different emotion classes, and obtaining the target training feature data through the preset multi-objective optimization algorithm according to the to-be-optimized training feature data.
In addition, it should be noted that, in the above step, a preset Softmax classification model is established, and in this stage, training is performed on all speakers respectively to obtain a classifier corresponding to each speaker, and the specific process is as follows:
step (1-1) segmenting each statement;
extracting the characteristics of each segment;
step (1-3) performing feature statistics on all features;
step (1-4) training a sentence fragment emotion classification method based on tendency cognitive learning;
step (1-5) training a support vector machine for each feature subspace;
the classification result of the step (1-6) is obtained by voting the results of all the support vector machines;
in addition, in the step (1-1), the speech signal is segmented at intervals of 0.2 seconds.
In the step (1-2), the extracting the speech signal feature for each segment includes: MFCC (Mel Frequency Cepstrum Coefficient), LFPC (Log Frequency Power Coefficient), LPCC (Linear Predictive Cepstral coding), ZCPA (Zero Crossing with Peak Amplitude), PLP (Percentual Linear Predictive), R-PLP (Ras ta Percentual Linear Predictive), the feature extraction result of each type of feature is a two-dimensional matrix, wherein one dimension is a time dimension; then calculate each class of features FiFirst derivative in the time dimension Δ FiSecond derivative Δ Δ FiConnecting the original features, the first derivative result and the second derivative result in series in a non-time dimension to form a final feature extraction result of each type of features; and (4) connecting the final feature extraction results of the features of all the classes in series on a non-time dimension to obtain the feature extraction result of the sample.
The characteristic statistics of the characteristics in the step (1-3) is as follows: obtaining the statistical results of mean, standard variance, minimum, maximum, kurtosis and skewness of the features in the time dimensionThe feature statistics are noted as { x1,x2,...,xnThe corresponding label is denoted as Y ═ Y1,y2,...,yn]∈Rn。
In the step (1-4), a data set X is givenA=[x1,x2,…,xm],XB=[xm+1,xm+2,…,xn]Wherein X isAIs a fragment of class A emotion, XBIs a segment of B-type emotion, and the training of the sentence segment emotion classification method based on tendency cognitive learning comprises the following steps:
step (1-4-1) for X ∈ XAIs mixing XAThe angle of the sample within the X-centered Parzen window from the center sample X is divided into bins and X is then calculated at X using the following equationAThe distribution characteristics of the surrounding data.
bx=[b1,b2,…,bk]
In the formula bjDenotes the jth box, 1 (x)i∈Xj) At xiBelong to XjValue of time is 1 otherwise 0, XjIs XAA subset of (2), XjThe angle between the sample inside and x is distributed in the jth bin.
Step (1-4-2) for X ∈ XAIs mixing XBThe angle of the sample within the X-centered Parzen window from the center sample X is divided into bins and X is then calculated at X using the following equationAThe distribution characteristics of the surrounding data.
In the formulaIs shown asA box, 1 (x)i∈Xj) At xiBelong to XjValue of time is 1 otherwise 0, XjIs XBA subset of (2), XjThe angle between the sample inside and x is distributed in the jth bin.
Step (1-4-3) calculates the difference in data distribution of the two data sets around point x using the following formula:
in the formulaRepresenting the distance between two vectors, a variety of distance calculation methods may be used.
Step (1-4-4) according to the calculation result of step (1-4-3), a fragment set prone to A emotion can be obtainedSegment set inclined to B emotionAnd a collection of segments that are prone to neutral emotionWhereinIs dx>X of T.Is dx<-x of T.Is T>dx>-T. T is a threshold value set autonomously. For each set, clustering into a plurality of regions by using a spectral clustering method to obtain a region label of each segment xi
Step (1-4-5) definitionL=[LA,LB,LC]Wherein L isA∈Rp、LB∈Rq、LC∈RuP, q and u are each independentlyAndnumber of samples, LA、LBAnd LCThe element values in (1) are 1, 2 and 3 respectively. The feature subspace of the segment is learned using the following objective equation:
J=J1(oi,oj)+β*J2(oi,oj)
beta is an equilibrium parameter. Wherein J1(oi,oj) Can realizeAndthe intra-class distance between three classes is small, and the inter-class distance is large, and is defined as follows:
in the formula oiAnd ojIs composed ofAndresults after mapping to subspace. liAnd ljCorresponds to oiAnd ojThe value in L. m is a threshold value, adjusting the range of inter-class distances. GijIs xiAnd xjThe gaussian distance between. The calculation formula is as follows:
J2(oi,oj) It is possible to try to keep the relative relationship within each region constant and the regions belonging to the same class relatively close but not overlapping. The definition is as follows:
in the formulaAndare the region labels for xi and xj. GliIs aiClass all GijMaximum value of (2). The method can keep the relationship between two segments when the two segments belong to the same area, and minimize the distance between the two segments with a small weight when the two segments do not belong to the same area but belong to the same category, so that the two areas can be prevented from overlapping as much as possible.
To optimize the objective equation J, define oi=φ(Wqφ(…W3φ(W2φ(W1xi+b1)+b2)+b3)+b4) Where phi (·) is a sigmoid function, W1,W2,…,WqTo map the matrix, b1,b2,…,bqIs an offset. By calculatingAndcan obtain W1,W2,…,WqAnd b1,b2,…,bqThe value of (a) is,is to take the derivative of J with respect to W,is the derivative of J over b.
Step (1-4-6) for the product obtained in step (1-4-5)Andtraining a Softmax classifier to separate emotion a, emotion B and emotion C.
And (1-4-7) training a softmax classifier capable of recognizing all emotion pairs according to the operation processes of the steps (1-4-5) and the steps (1-4-6).
Further, it should be understood that the following is a summary of the above:
the first step is as follows: all training sample voices were segmented at 0.2 second intervals.
The second step is that: extracting the characteristics of MFCC, LFPC, LPCC, ZCAP, PLP and R-PLP from all the speech segment training signals, wherein the number of Mel filters of the MFCC and the LFPC is 40; the linear prediction orders of the LPCC, the PLP and the R-PLP are respectively 12, 16 and 16; the frequency segmentation of the ZCAP is as follows: 0,106, 223, 352, 495, 655, 829, 1022, 1236, 1473, 1734, 2024, 2344, 2689, 3089, 3522, 4000. So that the dimensions of each class of features of each statement are respectively: ti 39, ti 40, ti 12, ti 16, where ti is the number of frames in the i-th sentence, and the number following the multiplier is the dimension of each frame feature. To obtain the change of the speech signal in the time dimension, a first derivative, a second derivative, is also calculated for the above features in the time dimension. Finally, the dimensionality of each type of feature is respectively as follows: ti 117, ti 140, ti 36, ti 48. The extracted speech signal features of the ith sample are combined from all the above features, and have a dimension ti (117+140+36+48+48+48).
The third step: the following statistical function was used: the mean (mean), standard deviation (standard deviation), minimum (min), maximum (max), kurtosis (kurtosis), skewness (skewness) are obtained as statistics of the above features in the time dimension. The feature statistics of the labeled samples are noted as { x1,x2,...,xnAnd n is the number of labeled specimens.
The fourth step: will { x in the previous step1,x2,...,xnDivide by statement label into XA=[x1,x2,…,xm],XB=[xm+1,xm+2,…,xn]Wherein X isAIs a fragment of class A emotion, XBIs a segment of B-type emotion, and the training of the sentence segment emotion classification method based on tendency cognitive learning comprises the following steps:
(1) for X ∈ XAIs mixing XAThe angle of the sample within the X-centered Parzen window from the center sample X is divided into bins and X is then calculated at X using the following equationAThe distribution characteristics of the surrounding data.
bx=[b1,b2,…,bk]
In the formula bjDenotes the jth box, 1 (x)i∈Xj) At xiBelong to XjValue of time is 1 otherwise 0, XjIs XAA subset of (2), XjThe angle between the sample inside and x is distributed in the jth bin.
(2) For X ∈ XAIs mixing XBThe angle of the sample within the X-centered Parzen window from the center sample X is divided into bins and X is then calculated at X using the following equationAThe distribution characteristics of the surrounding data.
In the formulaIs shown asA box, 1 (x)i∈Xj) At xiBelong to XjValue of time is 1 otherwise 0, XjIs XBA subset of (2), XjThe angle between the sample inside and x is distributed in the jth bin.
(3) The difference in data distribution of the two data sets around point x is calculated using the following equation:
in the formulaThe distance between two vectors is represented, where the euclidean distance is used.
(4) According to the calculation results of the steps (1-4-3), the fragment set which is inclined to the A feeling can be obtainedSegment set inclined to B emotionAnd a collection of segments that are prone to neutral emotionWhereinIs dx>X of T.Is dx<-x of T.Is T>dx>-T. T is a threshold value set autonomously. For each set, clustering into a plurality of regions by using a spectral clustering method to obtain a region label of each segment xi
(5) Definition ofL=[LA,LB,LC]Wherein L isA∈Rp、LB∈Rq、LC∈RuP, q and u are each independentlyAndnumber of samples, LA、LBAnd LCThe element values in (1) are 1, 2 and 3 respectively. The feature subspace of the segment is learned using the following objective equation:
J=J1(oi,oj)+β*J2(oi,oj)
beta is an equilibrium parameter. Wherein J1(oi,oj) Can realizeAndthe intra-class distance between three classes is small, and the inter-class distance is large, and is defined as follows:
in the formula oiAnd ojIs composed ofAndresults after mapping to subspace. liAnd ljCorresponds to oiAnd ojThe value in L. m is a threshold value, the range of the distance between classes is adjusted, and the value of 1 is taken in the invention. GijIs xiAnd xjThe gaussian distance between. The calculation formula is as follows:
J2(oi,oj) It is possible to try to keep the relative relationship within each region constant and the regions belonging to the same class relatively close but not overlapping. The definition is as follows:
in the formulaAndare the region labels for xi and xj. GliIs aiClass all GijMaximum value of (2). The method can keep the relationship between two segments when the two segments belong to the same area, and minimize the distance between the two segments with a small weight when the two segments do not belong to the same area but belong to the same category, so that the two areas can be prevented from overlapping as much as possible.
To optimize the objective equation J, define oi=φ(Wqφ(…W3φ(W2φ(W1xi+b1)+b2)+b3)+b4) Where phi (·) is a sigmoid function, W1,W2,…,WqTo map the matrix, b1,b2,…,bqIs an offset. By calculatingAndcan obtain W1,W2,…,WqAnd b1,b2,…,bqThe value of (a) is,is to take the derivative of J with respect to W,is the derivative of J over b.
(6) For those obtained in the above step (1-4-5)Andtraining the Sof tmax classifier to separate emotion A, emotion B and emotion C.
(7) And (4) training a Softmax classifier capable of recognizing all emotion pairs according to the operation process of the step (1-4-5) and the step (1-4-6).
In the embodiment, a plurality of initial training voice samples are obtained by obtaining training voice samples with preset dimensions and performing segmentation processing on the test voice samples through preset rules, then feature extraction is performed on the initial training voice samples to obtain training voice signal features to be processed, feature statistics is performed on the training voice signal features to be processed through a preset statistical function to obtain training feature statistical results to be confirmed, target training feature data are obtained through a preset multi-target optimization algorithm according to the training feature statistical results to be confirmed, then emotion classes corresponding to the target training feature data are obtained according to the target training feature data, and a preset Softmax classification model is established according to the emotion classes and the target training feature data corresponding to the emotion classes. By the method, the model can be trained aiming at the sentence local segments, different local segments in a sentence can be prevented from containing different emotions, or different local segments of the same emotion are prevented from conflicting with each other, and therefore the difference between deep learning physical meanings and speech emotion recognition characteristics is reduced.
In addition, an embodiment of the present invention further provides a storage medium, where a speech emotion recognition program is stored on the storage medium, and the speech emotion recognition program, when executed by a processor, implements the steps of the speech emotion recognition method described above.
Referring to FIG. 4, FIG. 4 is a block diagram illustrating a first embodiment of a speech emotion recognition apparatus according to the present invention.
As shown in fig. 4, the speech emotion recognition apparatus according to the embodiment of the present invention includes: the acquisition module 4001 is configured to acquire a test voice sample with a preset dimension, and perform segmentation processing on the test voice sample according to a preset rule to obtain a plurality of initial voice samples; an extraction module 4002, configured to perform signal feature data extraction on the initial voice sample to obtain to-be-processed voice signal feature data; the statistic module 4003 is configured to perform feature statistics on the to-be-processed speech signal feature data through a preset statistic function to obtain a feature statistical result to be confirmed; the calculating module 4004 is configured to obtain feature target data through a preset multi-objective optimization algorithm according to the feature statistical result to be confirmed; the determining module 4005 is configured to input the feature target data into a preset Softmax classification model, and obtain a speech emotion recognition result.
The obtaining module 4001 obtains a test voice sample with a preset dimension, and performs a segmentation process on the test voice sample according to a preset rule to obtain a plurality of initial voice samples.
Before the step of obtaining a test voice sample with a preset dimension, and performing segmentation processing on the test voice sample through a preset rule to obtain a plurality of initial voice samples, obtaining a training voice sample with a preset dimension, performing segmentation processing on the test voice sample through a preset rule to obtain a plurality of initial training voice samples, performing feature extraction on the initial training voice samples to obtain training voice signal features to be processed, performing feature statistics on the training voice signal features to be processed through a preset statistical function to obtain training feature statistical results to be confirmed, obtaining target training feature data through a preset multi-objective optimization algorithm according to the training feature statistical results to be confirmed, and obtaining an emotion category corresponding to the target training feature data according to the target training feature data, and establishing a preset Softmax classification model according to the emotion classes and target training characteristic data corresponding to the emotion classes.
In addition, it should be understood that the preset rule is a user-defined sample division rule, that is, if the duration of the obtained test voice sample of the preset dimension is 5s, and the preset rule is set to 0.2s, 25 segments of 0.2s initial voice samples are obtained after division according to the preset rule.
In addition, it should be noted that the preset dimension may be a time dimension, or may be a non-time dimension, and the present embodiment is not limited thereto.
The extraction module 4002 performs signal feature data extraction on the initial voice sample to obtain the operation of voice signal feature data to be processed.
Furthermore, it should be understood that performing signal feature data on the initial speech sample extracts Mel Frequency Cepstral Coefficients (MFCC), Log Frequency Power Coefficients (Log Frequency coeffients, LFPC), Linear Predictive Cepstral Coefficients (LPCC), Zero Crossing with Peak Amplitude (zca), Perceptual Linear Prediction (PLP), Rasta filter Perceptual Linear prediction (R-PLP).
It should be understood that the above feature extraction result of each type of feature is a two-dimensional matrix, wherein one dimension is a time dimension, and then each type of feature F is calculatediFirst derivative in the time dimension Δ FiThe original features, the first derivative results and the second derivative results are connected in series in a non-time dimension to form a final feature extraction result of each type of features; and (4) connecting the final feature extraction results of the features of all the classes in series on a non-time dimension to obtain the feature extraction result of the sample.
Further, for ease of understanding, the following is exemplified:
suppose, MFCC corresponds to FMFCC∈R39×z,ΔFMFCC∈R39×z,ΔΔFi∈R39×zWherein z is the number of frames, i.e. the number of degrees of the time dimension, the concatenation result in the non-time dimension
Furthermore, it should be understood that at each time of speech signal feature extraction, MFCC, LFPC, LPCC, ZCPA, PLP, R-PLP features are extracted, where the number of Mel filters of MFCC, LFPC is 40; the linear prediction orders of the LPCC, the PLP and the R-PLP are respectively 12, 16 and 16; the frequency segmentation of the ZCAP is as follows: 0,106, 223, 352, 495, 655, 829, 1022, 1236, 1473, 1734, 2024, 2344, 2689, 3089, 3522, 4000. So that the dimensions of each class of features of each statement are respectively: ti 39, ti 40, ti 12, ti 16, where ti is the number of frames in the i-th sentence, and the number following the multiplier is the dimension of each frame feature. To obtain the change of the speech signal in the time dimension, a first derivative, a second derivative, is also calculated for the above features in the time dimension. Finally, the dimensionality of each type of feature is respectively as follows: ti 117, ti 140, ti 36, ti 48. The extracted speech signal features of the ith sample are combined from all the above features, and have a dimension ti (117+140+36+48+48+48).
And the statistic module 4003 performs feature statistics on the feature data of the voice signal to be processed through a preset statistic function to obtain a feature statistic result to be confirmed.
The statistical results of the above features in the time dimension are obtained using a statistical function using a mean (mean), a standard deviation (standard deviation), a minimum (min), a maximum (max), a kurtosis (kurtosis), and a skewness (skewness).
In addition, it should be understood that the tag sample feature data is obtained by screening the statistical results obtained above, feature statistics is performed on the tag sample feature data through a preset statistical function to obtain a feature statistical result to be confirmed, and the feature statistical result of the tag sample is recorded as { x }1,x2,...,xnAnd n is the number of labeled specimens.
And the calculating module 4004 obtains the characteristic target data through a preset multi-objective optimization algorithm according to the statistical result of the characteristics to be confirmed.
In addition, it should be noted that { x in the above step1,x2,...,xnDivide by statement label into XA=[x1,x2,…,xm],XB=[xm+1,xm+2,…,xn]Wherein X isAIs a fragment of class A emotion, XBIs a segment of B-type emotion, and the training of the sentence segment emotion classification method based on tendency cognitive learning comprises the following steps:
(1) for X ∈ XAIs mixing XAThe angle of the sample within the X-centered Parzen window from the center sample X is divided into bins and X is then calculated at X using the following equationAThe distribution characteristics of the surrounding data.
bx=[b1,b2,…,bk]
In the formula bjDenotes the jth box, 1 (x)i∈Xj) At xiBelong to XjValue of time is 1 otherwise 0, XjIs XAA subset of (2), XjThe angle between the sample inside and x is distributed in the jth bin.
(2) For X ∈ XAIs mixing XBThe angle of the sample within the X-centered Parzen window from the center sample X is divided into bins and X is then calculated at X using the following equationAThe distribution characteristics of the surrounding data.
In the formulaIs shown asA box, 1 (x)i∈Xj) At xiBelong to XjValue of time is 1 otherwise 0, XjIs XBA subset of (2), XjThe angle between the sample inside and x is distributed in the jth bin.
(3) The difference in data distribution of the two data sets around point x is calculated using the following equation:
in the formulaThe distance between two vectors is represented, where the euclidean distance is used.
(4) According to the calculation result of the last step, the fragment set which is inclined to the A feeling can be obtainedSegment set inclined to B emotionAnd a collection of segments that are prone to neutral emotionWhereinIs dx>X of T.Is dx<-x of T.Is T>dx>-T. T is a threshold value set autonomously. For each set, clustering into a plurality of regions by using a spectral clustering method to obtain a region label of each segment xi
(5) Definition ofL=[LA,LB,LC]Wherein L isA∈Rp、LB∈Rq、LC∈RuP, q and u are each independentlyAndnumber of samples, LA、LBAnd LCThe element values in (1) are 1, 2 and 3 respectively. The feature subspace of the segment is learned using the following objective equation:
J=J1(oi,oj)+β*J2(oi,oj)
beta is an equilibrium parameter. Wherein J1(oi,oj) Can realizeAndthe intra-class distance between three classes is small, and the inter-class distance is large, and is defined as follows:
in the formula oiAnd ojIs composed ofAndresults after mapping to subspace. liAnd ljCorresponds to oiAnd ojThe value in L. m is a threshold value, the range of the distance between classes is adjusted, and the value of 1 is taken in the invention. GijIs xiAnd xjThe gaussian distance between. The calculation formula is as follows:
J2(oi,oj) It is possible to try to keep the relative relationship within each region constant and the regions belonging to the same class relatively close but not overlapping. The definition is as follows:
in the formulaAndare the region labels for xi and xj. GliIs aiClass all GijMaximum value of (2). The method can keep the relationship between two segments when the two segments belong to the same area, and minimize the distance between the two segments with a small weight when the two segments do not belong to the same area but belong to the same category, so that the two areas can be prevented from overlapping as much as possible.
To optimize the objective equation J, we define oi=φ(Wqφ(…W3φ(W2φ(W1xi+b1)+b2)+b3)+b4) Where phi (·) is a sigmoid function, W1,W2,…,WqTo map the matrix, b1,b2,…,bqIs an offset. By calculatingAndcan obtain W1,W2,…,WqAnd b1,b2,…,bqThe value of (a) is,is to find the conductance of J to WThe number of the first and second groups is,is the derivative of J over b.
And the determining module 4005 inputs the feature target data into a preset Softmax classification model to obtain a speech emotion recognition result.
Further, it is understood that W is obtained according to the above steps1,W2,…,WqAnd b1,b2,…,bqCalculating { x1,x2,...,xmFeature selection result z.
In addition, W is defined as1,W2,…,WqAnd b1,b2,…,bqIs the characteristic target data in the present application.
Further, it should be understood that { x } is obtained separately using a preset Softmax classifier obtained during training1,x2,...,xmSpeech emotion classification of { l }1,l2,...,lm}. Then according to { l1,l2,...,lmAnd voting to obtain the emotion of the statement.
In addition, it should be noted that the feature target data is input into the preset Softmax classification model to obtain speech emotion category data, data statistics is performed on the speech emotion category data to obtain a speech emotion category data value, and a speech emotion recognition result is obtained according to the speech emotion category data value.
In addition, the step of obtaining the speech emotion recognition result according to the speech emotion category data value is to determine whether the speech emotion category data value belongs to a preset speech emotion category threshold range, and if the speech emotion category data value belongs to the preset speech emotion category threshold range, obtain the speech emotion recognition result according to the speech emotion category data value; and if the voice emotion category data value does not belong to the preset voice emotion category threshold range, returning to the step of inputting the feature target data into the preset Softmax classification model to obtain voice emotion category data.
In addition, the corpus used for emotion recognition effect evaluation according to the present invention is a standard database in the speech emotion recognition field. The training process is first completed and then the recognition test is performed. The test mode was performed in a 5-fold crossover fashion. 7 emotions of anger, fear, irritability, disgust, happiness, neutrality and sadness can be identified, the average classification accuracy is 94.65% under the condition that the speaker depends on the emotion recognition method, and the distinction degree between other emotions is better except that the emotion recognition method is easier to confuse with anger and anger. The average classification accuracy was 89.30% with speaker independence.
It should be understood that the above is only an example, and the technical solution of the present invention is not limited in any way, and in a specific application, a person skilled in the art may set the technical solution as needed, and the present invention is not limited thereto.
In the embodiment, a test voice sample with a preset dimension is obtained, the test voice sample is subjected to sectional processing through a preset rule to obtain a plurality of initial voice samples, then signal characteristic data extraction is carried out on the initial voice samples to obtain voice signal characteristic data to be processed, the voice signal characteristic data to be processed is screened to obtain label sample characteristic data, characteristic statistics is carried out on the label sample characteristic data through a preset statistical function to obtain a characteristic statistical result to be confirmed, then emotion classification is carried out on the training characteristic statistical result to be confirmed to obtain training characteristic data to be optimized corresponding to different emotion classifications, target training characteristic data is obtained through a preset multi-objective optimization algorithm according to the training characteristic data to be optimized, and finally the characteristic target data is input into a preset Softmax classification model, and obtaining a speech emotion recognition result. By the method, the voice emotion fragments and the emotion relation between the sentences and the fragments can be fully utilized to form the data with a tendency, so that the process of processing the tendency of a human can be simulated, the unbalanced information of the data is utilized, the data are compared with each other, the fragments with different emotions are separated under constraint conditions, and the sample scale is increased and the sample diversity is improved.
It should be noted that the above-described work flows are only exemplary, and do not limit the scope of the present invention, and in practical applications, a person skilled in the art may select some or all of them to achieve the purpose of the solution of the embodiment according to actual needs, and the present invention is not limited herein.
In addition, the technical details that are not described in detail in this embodiment may refer to the speech emotion recognition method provided in any embodiment of the present invention, and are not described herein again.
Further, it is to be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention or portions thereof that contribute to the prior art may be embodied in the form of a software product, where the computer software product is stored in a storage medium (e.g. Read Only Memory (ROM)/RAM, magnetic disk, optical disk), and includes several instructions for enabling a terminal device (e.g. a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (8)
1. A speech emotion recognition method, characterized in that the method comprises:
acquiring a test voice sample with a preset dimension, and performing segmentation processing on the test voice sample through a preset rule to obtain a plurality of initial voice samples;
extracting signal characteristic data of the initial voice sample to obtain voice signal characteristic data to be processed;
performing feature statistics on the voice signal feature data to be processed through a preset statistical function to obtain a feature statistical result to be confirmed;
obtaining feature target data through a preset multi-target optimization algorithm according to the feature statistical result to be confirmed;
inputting the characteristic target data into a preset Softmax classification model to obtain a speech emotion recognition result;
the method for obtaining the initial voice samples comprises the following steps of obtaining a test voice sample with a preset dimensionality, carrying out segmentation processing on the test voice sample through a preset rule, and obtaining a plurality of initial voice samples, wherein before the step of obtaining the initial voice samples, the method further comprises the following steps:
acquiring training voice samples with preset dimensions, and performing segmentation processing on the test voice samples through preset rules to obtain a plurality of initial training voice samples;
performing feature extraction on the initial training voice sample to obtain the features of a training voice signal to be processed;
performing feature statistics on the training voice signal features to be processed through a preset statistical function to obtain a training feature statistical result to be confirmed;
carrying out emotion category division on the statistical result of the training features to be confirmed to obtain training feature data to be optimized corresponding to different emotion categories;
determining a peripheral data distribution characteristic set corresponding to each training characteristic data to be optimized;
determining data distribution differences corresponding to different emotion classes according to the surrounding data distribution feature set;
obtaining emotion fragment sets corresponding to different emotion types according to the data distribution difference;
determining a feature subspace corresponding to each emotion fragment set;
and establishing a preset Softmax classification model based on the plurality of feature subspaces.
2. The method of claim 1, wherein the step of inputting the feature target data into a preset Softmax classification model to obtain the speech emotion recognition result comprises:
inputting the characteristic target data into the preset Softmax classification model to obtain speech emotion category data;
performing data statistics on the voice emotion type data to obtain a voice emotion type data value;
and obtaining a voice emotion recognition result according to the voice emotion category data value.
3. The method of claim 2, wherein the step of obtaining speech emotion recognition results according to the speech emotion classification data values comprises:
judging whether the voice emotion type data value belongs to a preset voice emotion type threshold range or not;
and if the voice emotion type data value belongs to the preset voice emotion type threshold range, acquiring a voice emotion recognition result according to the voice emotion type data value.
4. The method of claim 3, wherein the step of determining whether the speech emotion classification data value falls within a preset speech emotion classification threshold range further comprises:
and if the voice emotion category data value does not belong to the preset voice emotion category threshold range, returning to the step of inputting the feature target data into the preset Softmax classification model to obtain voice emotion category data.
5. The method according to claim 1, wherein the step of performing feature statistics on the feature data of the speech signal to be processed by using a preset statistical function to obtain a feature statistical result to be confirmed comprises:
screening the voice signal characteristic data to be processed to obtain label sample characteristic data;
and carrying out feature statistics on the tag sample feature data through a preset statistical function to obtain a feature statistical result to be confirmed.
6. An apparatus for speech emotion recognition, the apparatus comprising:
the system comprises an acquisition module, a processing module and a control module, wherein the acquisition module is used for acquiring a test voice sample with a preset dimensionality and performing segmentation processing on the test voice sample through a preset rule to obtain a plurality of initial voice samples;
the extraction module is used for extracting signal characteristic data of the initial voice sample to obtain voice signal characteristic data to be processed;
the statistical module is used for carrying out feature statistics on the voice signal feature data to be processed through a preset statistical function to obtain a feature statistical result to be confirmed;
the calculation module is used for obtaining feature target data through a preset multi-target optimization algorithm according to the feature statistical result to be confirmed;
the determining module is used for inputting the characteristic target data into a preset Softmax classification model to obtain a speech emotion recognition result;
the speech emotion recognition apparatus further includes: acquiring training voice samples with preset dimensions, and performing segmentation processing on the test voice samples through preset rules to obtain a plurality of initial training voice samples; performing feature extraction on the initial training voice sample to obtain the features of a training voice signal to be processed; performing feature statistics on the training voice signal features to be processed through a preset statistical function to obtain a training feature statistical result to be confirmed; carrying out emotion category division on the statistical result of the training features to be confirmed to obtain training feature data to be optimized corresponding to different emotion categories; determining a peripheral data distribution characteristic set corresponding to each training characteristic data to be optimized; determining data distribution differences corresponding to different emotion classes according to the surrounding data distribution feature set; obtaining emotion fragment sets corresponding to different emotion types according to the data distribution difference; determining a feature subspace corresponding to each emotion fragment set; and establishing a preset Softmax classification model based on the plurality of feature subspaces.
7. An electronic device, characterized in that the device comprises: a memory, a processor and a speech emotion recognition program stored on the memory and executable on the processor, the speech emotion recognition program being configured to implement the steps of the speech emotion recognition method as claimed in any of claims 1 to 5.
8. A storage medium having stored thereon a speech emotion recognition program, which when executed by a processor implements the steps of the speech emotion recognition method as claimed in any one of claims 1 to 5.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911246544.5A CN110956981B (en) | 2019-12-06 | 2019-12-06 | Speech emotion recognition method, device, equipment and storage medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911246544.5A CN110956981B (en) | 2019-12-06 | 2019-12-06 | Speech emotion recognition method, device, equipment and storage medium |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110956981A CN110956981A (en) | 2020-04-03 |
CN110956981B true CN110956981B (en) | 2022-04-26 |
Family
ID=69980269
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911246544.5A Expired - Fee Related CN110956981B (en) | 2019-12-06 | 2019-12-06 | Speech emotion recognition method, device, equipment and storage medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110956981B (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112466324A (en) * | 2020-11-13 | 2021-03-09 | 上海听见信息科技有限公司 | Emotion analysis method, system, equipment and readable storage medium |
CN113326678B (en) * | 2021-06-24 | 2024-08-06 | 深圳前海微众银行股份有限公司 | Conference summary generation method and device, terminal equipment and computer storage medium |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102663413A (en) * | 2012-03-09 | 2012-09-12 | 中盾信安科技(江苏)有限公司 | Multi-gesture and cross-age oriented face image authentication method |
CN105488456A (en) * | 2015-11-23 | 2016-04-13 | 中国科学院自动化研究所 | Adaptive rejection threshold adjustment subspace learning based human face detection method |
CN105913025A (en) * | 2016-04-12 | 2016-08-31 | 湖北工业大学 | Deep learning face identification method based on multiple-characteristic fusion |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100250477A1 (en) * | 2009-03-31 | 2010-09-30 | Shekhar Yadav | Systems and methods for optimizing a campaign |
CA2786380C (en) * | 2010-01-18 | 2020-07-14 | Elminda Ltd. | Method and system for weighted analysis of neurophysiological data |
CN104008754B (en) * | 2014-05-21 | 2017-01-18 | 华南理工大学 | Speech emotion recognition method based on semi-supervised feature selection |
CN107977651B (en) * | 2017-12-21 | 2019-12-24 | 西安交通大学 | Common spatial mode spatial domain feature extraction method based on quantization minimum error entropy |
-
2019
- 2019-12-06 CN CN201911246544.5A patent/CN110956981B/en not_active Expired - Fee Related
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102663413A (en) * | 2012-03-09 | 2012-09-12 | 中盾信安科技(江苏)有限公司 | Multi-gesture and cross-age oriented face image authentication method |
CN105488456A (en) * | 2015-11-23 | 2016-04-13 | 中国科学院自动化研究所 | Adaptive rejection threshold adjustment subspace learning based human face detection method |
CN105913025A (en) * | 2016-04-12 | 2016-08-31 | 湖北工业大学 | Deep learning face identification method based on multiple-characteristic fusion |
Non-Patent Citations (4)
Title |
---|
The Role of Principal Angles in Subspace Classification;Jiaji Huang;《IEEE Transactions on Signal Processing》;20151117;全文 * |
Transferable Representation Learning with Deep Adaptation Networks;Mingsheng Long;<IEEE Transactions on Pattern Analysis and Machine Intelligence>;20180905;第3071-3085页 * |
人机情感交互的方法与技术研究;王国江;《中国博士学位论文全文数据库》;20080415(第4期);全文 * |
面向局部特征和特征表达的图像分类算法研究;张旭;《中国博士学位论文全文数据库》;20170215(第2期);全文 * |
Also Published As
Publication number | Publication date |
---|---|
CN110956981A (en) | 2020-04-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Dai et al. | Learning discriminative features from spectrograms using center loss for speech emotion recognition | |
Bahari et al. | Speaker age estimation and gender detection based on supervised non-negative matrix factorization | |
Lan et al. | An extreme learning machine approach for speaker recognition | |
CN111696557A (en) | Method, device and equipment for calibrating voice recognition result and storage medium | |
CN108346436A (en) | Speech emotional detection method, device, computer equipment and storage medium | |
Parthasarathy et al. | Convolutional neural network techniques for speech emotion recognition | |
Wang et al. | Discriminative neural embedding learning for short-duration text-independent speaker verification | |
CN111916111A (en) | Intelligent voice outbound method and device with emotion, server and storage medium | |
JP6787770B2 (en) | Language mnemonic and language dialogue system | |
CN107767881B (en) | Method and device for acquiring satisfaction degree of voice information | |
CN110956981B (en) | Speech emotion recognition method, device, equipment and storage medium | |
CN104538036A (en) | Speaker recognition method based on semantic cell mixing model | |
Bahari | Speaker age estimation using Hidden Markov Model weight supervectors | |
CN112233651A (en) | Dialect type determining method, dialect type determining device, dialect type determining equipment and storage medium | |
CN112632248A (en) | Question answering method, device, computer equipment and storage medium | |
CN109377984B (en) | ArcFace-based voice recognition method and device | |
Mande et al. | EMOTION DETECTION USING AUDIO DATA SAMPLES. | |
Akbal et al. | Development of novel automated language classification model using pyramid pattern technique with speech signals | |
Birla | A robust unsupervised pattern discovery and clustering of speech signals | |
JP6996627B2 (en) | Information processing equipment, control methods, and programs | |
Elbarougy | Speech emotion recognition based on voiced emotion unit | |
CN111145787B (en) | Voice emotion feature fusion method and system based on main and auxiliary networks | |
Hanilçi et al. | Investigation of the effect of data duration and speaker gender on text-independent speaker recognition | |
CN115315746A (en) | Speaker recognition method, recognition device, recognition program, gender recognition model generation method, and speaker recognition model generation method | |
Michalevsky et al. | Speaker identification using diffusion maps |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
CF01 | Termination of patent right due to non-payment of annual fee | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20220426 |