CN115273907A - Speech emotion recognition method and device - Google Patents

Abstract

The application discloses a speech emotion recognition method and a speech emotion recognition device, which can be applied to the field of big data or the field of finance, and the method comprises the following steps: acquiring a voice file; preprocessing a voice file to obtain voice characteristic information corresponding to the voice file; starting a preset text processing tool, converting the voice file into text information, and generating a text vector corresponding to the text information; carrying out weighting processing on the voice characteristic information and the text vector to obtain weighted voice characteristic information and weighted text vector; fusing the weighted voice characteristic information and the weighted text vector to obtain fusion characteristics corresponding to the voice file; and inputting the fusion characteristics into a preset maximum pooling layer and a preset full connection layer for emotion analysis to obtain the emotion types corresponding to the voice files. By applying the method provided by the invention, the speech emotion can be recognized by combining text information besides the speech characteristics, and the recognition precision of the speech emotion is improved.

Description

Speech emotion recognition method and device

technical field

The invention relates to the technical field of speech recognition, in particular to a speech emotion recognition method and device.

Background technique

With the development of artificial intelligence, the status of affective computing is becoming more and more important. Affective computing tries to endow machines with the ability to observe, understand and generate various emotions, making machines more human-like with emotions. Speech, as an important transmission medium in human communication, contains a lot of emotional information. Speech emotion recognition can improve the ability of machines to understand human voice emotions, and thus be more widely used in human-computer dialogue, making human-computer interaction more natural harmonious.

The speech emotion recognition method in the prior art is to carry out emotion classification on the speech features through the neural network, but the method in the prior art only pays attention to the analysis of the aspect of the rhythm, so the method in the speech emotion recognition in the prior art still cannot express the speech well. emotion.

Contents of the invention

In view of this, the present invention provides a speech emotion recognition method, by which speech emotion can be recognized in combination with text information in addition to speech features, and the recognition accuracy of speech emotion can be improved.

The present invention also provides a speech emotion recognition device to ensure the realization and application of the above method in practice.

A speech emotion recognition method, comprising:

Get audio files;

Preprocessing the voice file to obtain voice feature information corresponding to the voice file;

Enable a preset text processing tool, convert the voice file into text information, and generate a text vector corresponding to the text information;

performing weighting processing on the speech feature information and the text vector to obtain weighted speech feature information and weighted text vector;

Fusing the weighted speech feature information with the weighted text vector to obtain fusion features corresponding to the speech file;

The fusion feature is input into the preset maximum pooling layer and fully connected layer for emotion analysis, and the emotion type corresponding to the voice file is obtained.

In the above method, optionally, the preprocessing of the voice file to obtain the voice feature information corresponding to the voice file includes:

Obtain the MFCC feature in the voice file;

Applying the preset BiLSTM to process the MFCC to obtain the voice feature information corresponding to the voice file.

In the above method, optionally, the enabling of a preset text processing tool to convert the voice file into text information includes:

Enable the text processing tool to convert the voice file into initial text information;

Perform data cleaning on the initial text information, remove invalid characters and stop words in the initial text information, and obtain text information corresponding to the voice file.

In the above method, optionally, the generating the text vector corresponding to the text information includes:

Applying a preset word segmentation tool to perform word segmentation processing on the text information to obtain a plurality of words corresponding to the text information;

Apply the pre-set natural language processing tool NLTK to carry out part-of-speech tagging for each of the words, and based on the part of speech of each of the words, convert each of the words into its corresponding 300-dimensional vector;

Input the 300-dimensional vector corresponding to each word into the BiLSTM, and obtain the text vector corresponding to the text information output by the BiLSTM.

In the above method, optionally, the merging of the weighted speech feature information and the weighted text vector to obtain the fusion feature corresponding to the speech file includes:

Acquiring frame speech features of each frame of speech in the speech feature information;

Applying a pre-set attention mechanism based on the word features of each of the words and each of the frame voice features, weighted to obtain the word voice features of the voice corresponding to each of the words;

Splicing the word speech feature of the speech corresponding to each of the words with the 300-dimensional vector corresponding to the word, to obtain the fusion feature corresponding to the speech file.

A speech emotion recognition device, comprising:

an acquisition unit, configured to acquire a voice file;

A first processing unit, configured to preprocess the voice file to obtain voice feature information corresponding to the voice file;

A conversion unit, configured to enable a preset text processing tool, convert the voice file into text information, and generate a text vector corresponding to the text information;

A second processing unit, configured to perform weighting processing on the speech feature information and the text vector, to obtain weighted speech feature information and a weighted text vector;

A feature fusion unit, configured to fuse the weighted speech feature information with the weighted text vector to obtain fusion features corresponding to the speech file;

An analysis unit, configured to input the fusion feature into a preset maximum pooling layer and a fully connected layer to perform emotion analysis, and obtain the emotion type corresponding to the voice file.

In the above device, optionally, the first processing unit includes:

The first obtaining subunit is used to obtain the MFCC feature in the voice file;

The first processing subunit is configured to apply a preset BiLSTM to process the MFCC to obtain voice feature information corresponding to the voice file.

The above device, optionally, the conversion unit includes:

The first conversion subunit is used to enable the text processing tool to convert the voice file into initial text information;

The data cleaning subunit is configured to perform data cleaning on the initial text information, remove invalid characters and stop words in the initial text information, and obtain text information corresponding to the voice file.

The above device, optionally, the conversion unit includes:

The second processing subunit is configured to apply a preset word segmentation tool to perform word segmentation processing on the text information, and obtain multiple words corresponding to the text information;

The second conversion subunit is used to apply the pre-set natural language processing tool NLTK to carry out part-of-speech tagging on each of the words, and based on the part-of-speech of each of the words, convert each of the words into its corresponding 300-dimensional vector;

The input subunit is configured to input the 300-dimensional vector corresponding to each word into the BiLSTM, and obtain the text vector corresponding to the text information output by the BiLSTM.

In the above device, optionally, the feature fusion unit includes:

The second acquisition subunit is used to acquire the frame speech features of each frame of speech in the speech feature information;

The weighting subunit is used to apply a preset attention mechanism based on the word features of each of the words and each of the frame voice features, weighted to obtain the word voice features of the voice corresponding to each of the words;

The splicing subunit is used to splice the word phonetic feature of the voice corresponding to each word with the 300-dimensional vector corresponding to the word, to obtain the fusion feature corresponding to the voice file.

A storage medium, the storage medium includes stored instructions, wherein when the instructions are run, the device where the storage medium is located is controlled to execute the above speech emotion recognition method.

An electronic device includes a memory, and one or more instructions, wherein the one or more instructions are stored in the memory, and are configured to be executed by one or more processors of the above speech emotion recognition method.

Compared with the prior art, the present invention includes the following advantages:

The present invention provides a speech emotion recognition method, comprising: obtaining a speech file; preprocessing the speech file to obtain speech feature information corresponding to the speech file; enabling a preset text processing tool to convert the speech file into text information, and generate a text vector corresponding to the text information; carry out weighting processing on the speech feature information and the text vector to obtain weighted speech feature information and a weighted text vector; The voice feature information is fused with the weighted text vector to obtain the fusion feature corresponding to the voice file; the fusion feature is input into a preset maximum pooling layer and a fully connected layer for sentiment analysis to obtain the voice file corresponding emotion type. By applying the method provided by the present invention, voice emotion recognition can be carried out in combination with text information in addition to voice features, and the recognition accuracy of voice emotion can be improved.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention, and those skilled in the art can also obtain other drawings according to the provided drawings on the premise of not paying creative efforts.

Fig. 1 is the method flowchart of a kind of speech emotion recognition method that the embodiment of the present invention provides;

Fig. 2 is another method flowchart of a speech emotion recognition method provided by an embodiment of the present invention;

Fig. 3 is another method flowchart of a speech emotion recognition method provided by an embodiment of the present invention

FIG. 4 is a device structural diagram of a speech emotion recognition device provided by an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

In this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any actual relationship or order, the terms "comprises," "comprises," or any other variation thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that includes a series of elements includes not only those elements, but also Including other elements not expressly listed, or also including elements inherent in such process, method, article or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The invention is applicable to numerous general purpose or special purpose computing device environments or configurations. For example: personal computer, server computer, handheld or portable device, tablet type device, multiprocessor device, distributed computing environment including any of the above devices or devices, etc.

The embodiment of the present invention provides a voice emotion recognition method, which can be applied to various system platforms, and its execution body can be a computer terminal or a processor of various mobile devices. The method flow chart of the method is shown in Figure 1 , specifically include:

S101: Obtain a voice file.

In the present invention, the voice file contains voice data.

S102: Perform preprocessing on the voice file to obtain voice feature information corresponding to the voice file.

Among them, the process of preprocessing the voice file is:

Obtain the MFCC feature in the voice file;

Applying the preset BiLSTM to process the MFCC to obtain the voice feature information corresponding to the voice file.

Among them, MFCC (Mel-scale Frequency Cepstral Coefficients, Mel cepstral coefficient) is the feature of speech at low latitude, first obtain the low-dimensional frame-based MFCC feature of speech, and then use BiLSTM to perform high-dimensional feature representation of speech based on frame , to obtain the voice feature information corresponding to the voice file.

S103: Activate a preset text processing tool, convert the voice file into text information, and generate a text vector corresponding to the text information.

It should be noted that the text processing tool may be automatic speech recognition (ASR) technology.

Further, when converting the voice file into text information, the following process can be performed:

Enable the text processing tool to convert the voice file into initial text information;

Perform data cleaning on the initial text information, remove invalid characters and stop words in the initial text information, and obtain text information corresponding to the voice file.

It should be noted that the voice data in the voice file may have some errors in text conversion when the voice is converted into text information due to factors such as slurred speech or polyphonic characters. By cleaning the data, correct the text in the initial text information content, and remove invalid characters and stop words to obtain the final text information.

S104: Perform weighting processing on the speech feature information and the text vector to obtain weighted speech feature information and a weighted text vector.

Specifically, the Attention mechanism can be used to dynamically learn the weight of each word text feature and the feature of each frame of speech. In the entire voice file, the amount of information contained in each frame of voice is different, and the voice of some frames contains key information content, so the present invention uses the weight of text features to multiply the features of each frame of voice, thereby determining the importance of each frame of voice , which is a weighting process, adding the weighted speech features of each frame to the text features of each word to obtain the speech alignment features of each word, concatenating the aligned features and text features to obtain fused features, and finally Input these features into BiLSTM for feature processing.

S105: Fusion the weighted speech feature information and the weighted text vector to obtain fusion features corresponding to the speech file.

Among them, the fusion of features can be done through BiLSTM.

S106: Input the fusion feature into the preset maximum pooling layer and fully connected layer to perform emotion analysis, and obtain the emotion type corresponding to the voice file.

It should be noted that the maximum pooling layer and the fully connected layer can be a processing module in BiLSTM.

In the method provided by the embodiment of the present invention, the voice file is obtained, and its corresponding voice feature information is obtained after preprocessing it, and converted into text information at the same time, the file information is converted into a text vector, and the voice feature information and the text vector are processed. After the weighting process, the features of the two are fused, and the maximum pooling layer and the fully connected layer can be considered for analysis, such as: context, text meaning, and speech rate, etc., to analyze the emotion type corresponding to the voice file.

Further, the emotion number corresponding to the emotion type is output, and the emotion of the user corresponding to the voice file can be known according to the emotion number.

By applying the method provided by the embodiment of the present invention, voice emotion recognition can be performed in combination with text information in addition to voice features, and the recognition accuracy of voice emotion can be improved.

In the method provided in the embodiment of the present invention, the process of generating the text vector corresponding to the text information is shown in Figure 2, which may specifically include:

S201: Applying a preset word segmentation tool to perform word segmentation processing on the text information to obtain multiple words corresponding to the text information.

S202: Using the preset natural language processing tool NLTK to perform part-of-speech tagging on each of the words, and based on the part-of-speech of each of the words, converting each of the words into its corresponding 300-dimensional vector.

Among them, the 300-dimensional vector corresponding to each word contains additional contextual meaning between words.

S203: Input the 300-dimensional vector corresponding to each word into the BiLSTM, and obtain a text vector corresponding to the text information output by the BiLSTM.

It should be noted that BiLSTM is a bidirectional LSTM, which is synthesized by combining two separate LSTMs.

In the present invention, automatic speech recognition (ASR) technology is used to extract text information from audio files with high precision. The present invention uses processed text information as another form to predict the emotional category of a given signal. To use textual information, speech transcripts were tokenized and encoded into tokenized sequences using the Natural Language Toolkit (NLTK). Each token is then passed through a word-embedding layer that converts word indices into corresponding 300-dimensional vectors that contain additional contextual meaning between words. The sequence of embedded tokens is fed into the text RNN, and finally, the sentiment class is predicted from the last hidden state of the text RNN using the SoftMax function.

In the method provided in the embodiment of the present invention, the process of merging the weighted speech feature information with the weighted text vector to obtain the fusion feature corresponding to the speech file is shown in Figure 3, which can be specifically include:

S301: Obtain frame speech features of each frame of speech in the speech feature information;

S302: Applying a preset attention mechanism based on the word features of each of the words and the speech features of each of the frames, weighting the word speech features of the speech corresponding to each of the words;

S303: Concatenate the word speech features of the speech corresponding to each word with the 300-dimensional vector corresponding to the word, to obtain the fusion feature corresponding to the speech file.

In the present invention, the Attention mechanism is used to dynamically learn the weight of the text features of each word and the features of each frame of speech, and then weighted and summed to obtain the speech alignment features of each word, and then we splice the aligned features and text features And use BiLSTM for feature fusion, and finally we use the maximum pooling layer and the fully connected layer for sentiment classification.

The specific implementation process of each of the above embodiments and its derivation methods are within the protection scope of the present invention.

Corresponding to the method described in Figure 1, the embodiment of the present invention also provides a voice emotion recognition device for the specific implementation of the method in Figure 1, the voice emotion recognition device provided in the embodiment of the present invention can be applied to a computer terminal or Among the various mobile devices, their structural diagrams are shown in Figure 4, specifically including:

An acquisition unit 401, configured to acquire a voice file;

The first processing unit 402 is configured to preprocess the voice file to obtain voice feature information corresponding to the voice file;

The conversion unit 403 is configured to enable a preset text processing tool, convert the voice file into text information, and generate a text vector corresponding to the text information;

The second processing unit 404 is configured to perform weighting processing on the speech feature information and the text vector to obtain weighted speech feature information and a weighted text vector;

A feature fusion unit 405, configured to fuse the weighted speech feature information with the weighted text vector to obtain fusion features corresponding to the speech file;

The analysis unit 406 is configured to input the fusion feature into the preset maximum pooling layer and fully connected layer for sentiment analysis, and obtain the emotion type corresponding to the voice file.

In the device provided by the embodiment of the present invention, the voice file is obtained, and its corresponding voice feature information is obtained after preprocessing, and converted into text information at the same time, the file information is converted into a text vector, and the voice feature information and the text vector are processed. After the weighting process, the features of the two are fused, and the maximum pooling layer and the fully connected layer can be considered for analysis, such as: context, text meaning, and speech rate, etc., to analyze the emotion type corresponding to the voice file.

By applying the device provided by the embodiment of the present invention, in addition to speech features, text information can be combined to recognize speech emotion, and the recognition accuracy of speech emotion can be improved.

In the device provided in the embodiment of the present invention, the first processing unit 402 includes:

The first obtaining subunit is used to obtain the MFCC feature in the voice file;

The first processing subunit is configured to apply the BiLSTM to process the MFCC to obtain voice feature information corresponding to the voice file.

In the device provided in the embodiment of the present invention, the conversion unit 403 includes:

The first conversion subunit is used to enable the text processing tool to convert the voice file into initial text information;

The data cleaning subunit is configured to perform data cleaning on the initial text information, remove invalid characters and stop words in the initial text information, and obtain text information corresponding to the voice file.

In the device provided by the embodiment of the present invention, it is characterized in that the conversion unit 403 includes:

The second processing subunit is configured to apply a preset word segmentation tool to perform word segmentation processing on the text information, and obtain multiple words corresponding to the text information;

The second conversion subunit is used to apply the pre-set natural language processing tool NLTK to carry out part-of-speech tagging on each of the words, and based on the part-of-speech of each of the words, convert each of the words into its corresponding 300-dimensional vector;

The input subunit is configured to input the 300-dimensional vector corresponding to each word into the BiLSTM, and obtain the text vector corresponding to the text information output by the BiLSTM.

In the device provided in the embodiment of the present invention, the feature fusion unit 405 includes:

The second acquisition subunit is used to acquire the frame speech features of each frame of speech in the speech feature information;

The weighting subunit is used to apply a preset attention mechanism based on the word features of each of the words and each of the frame voice features, weighted to obtain the word voice features of the voice corresponding to each of the words;

The splicing subunit is used to splice the word phonetic feature of the voice corresponding to each word with the 300-dimensional vector corresponding to the word, to obtain the fusion feature corresponding to the voice file.

For the specific working process of each unit and subunit in the speech emotion recognition device disclosed in the above embodiments of the present invention, please refer to the corresponding content in the speech emotion recognition method disclosed in the above embodiments of the present invention, which will not be repeated here.

It should be noted that the voice emotion recognition method and device provided by the present invention can be used in the cloud computing field or the financial field. The above is only an example, and does not limit the application field of the speech emotion recognition method and device provided by the present invention.

The voice emotion recognition method and device provided by the present invention can be used in the financial field or other fields, for example, can be used in voice business application scenarios in the financial field. Other domains are arbitrary domains other than the financial domain, for example, the cloud computing domain. The above is only an example, and does not limit the application field of the speech emotion recognition method and device provided by the present invention.

An embodiment of the present invention also provides a storage medium, the storage medium includes stored instructions, wherein when the instructions are run, the device where the storage medium is located is controlled to execute the above voice emotion recognition method.

The embodiment of the present invention also provides an electronic device, the structural diagram of which is shown in FIG. Configured to execute the one or more instructions 502 by one or more processors 503 to perform the following operations:

Get audio files;

Preprocessing the voice file to obtain voice feature information corresponding to the voice file;

Enable a preset text processing tool, convert the voice file into text information, and generate a text vector corresponding to the text information;

performing weighting processing on the speech feature information and the text vector to obtain weighted speech feature information and weighted text vector;

Fusing the weighted speech feature information with the weighted text vector to obtain fusion features corresponding to the speech file;

The fusion feature is input into the preset maximum pooling layer and fully connected layer for emotion analysis, and the emotion type corresponding to the voice file is obtained.

Each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment. The systems and system embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is It can be located in one place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without creative effort.

Professionals can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software or a combination of the two.

In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of functions in the above description. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A speech emotion recognition method, characterized in that, comprising: Get audio files; Preprocessing the voice file to obtain voice feature information corresponding to the voice file; Enable a preset text processing tool, convert the voice file into text information, and generate a text vector corresponding to the text information; performing weighting processing on the speech feature information and the text vector to obtain weighted speech feature information and weighted text vector; Fusing the weighted speech feature information with the weighted text vector to obtain fusion features corresponding to the speech file; The fusion feature is input into the preset maximum pooling layer and fully connected layer for emotion analysis, and the emotion type corresponding to the voice file is obtained. 2. The method according to claim 1, wherein the preprocessing the voice file to obtain the voice feature information corresponding to the voice file comprises: Obtain the MFCC feature in the voice file; Apply the preset BiLSTM to process the MFCC to obtain the voice feature information corresponding to the voice file. 3. The method according to claim 1, wherein said enabling a preset text processing tool to convert said voice file into text information comprises: Enable the text processing tool to convert the voice file into initial text information; Perform data cleaning on the initial text information, remove invalid characters and stop words in the initial text information, and obtain text information corresponding to the voice file. 4. The method according to claim 2, wherein said generating a text vector corresponding to said text information comprises: Applying a preset word segmentation tool to perform word segmentation processing on the text information to obtain a plurality of words corresponding to the text information; Apply the pre-set natural language processing tool NLTK to carry out part-of-speech tagging for each of the words, and based on the part of speech of each of the words, convert each of the words into its corresponding 300-dimensional vector; Input the 300-dimensional vector corresponding to each word into the BiLSTM, and obtain the text vector corresponding to the text information output by the BiLSTM. 5. The method according to claim 4, wherein said merging said weighted speech feature information and said weighted text vector to obtain a fusion feature corresponding to said speech file comprises: Acquiring frame speech features of each frame of speech in the speech feature information; Applying a pre-set attention mechanism based on the word features of each of the words and each of the frame voice features, weighted to obtain the word voice features of the voice corresponding to each of the words; Splicing the word speech feature of the speech corresponding to each of the words with the 300-dimensional vector corresponding to the word, to obtain the fusion feature corresponding to the speech file. 6. A speech emotion recognition device, characterized in that, comprising: an acquisition unit, configured to acquire a voice file; A first processing unit, configured to preprocess the voice file to obtain voice feature information corresponding to the voice file; A conversion unit, configured to enable a preset text processing tool, convert the voice file into text information, and generate a text vector corresponding to the text information; A second processing unit, configured to perform weighting processing on the speech feature information and the text vector, to obtain weighted speech feature information and a weighted text vector; A feature fusion unit, configured to fuse the weighted speech feature information with the weighted text vector to obtain fusion features corresponding to the speech file; An analysis unit, configured to input the fusion feature into a preset maximum pooling layer and a fully connected layer to perform emotion analysis, and obtain the emotion type corresponding to the voice file. 7. The device according to claim 6, wherein the first processing unit comprises: The first obtaining subunit is used to obtain the MFCC feature in the voice file; The first processing subunit is configured to apply a preset BiLSTM to process the MFCC to obtain voice feature information corresponding to the voice file. 8. The device according to claim 6, wherein the converting unit comprises: The first conversion subunit is used to enable the text processing tool to convert the voice file into initial text information; The data cleaning subunit is configured to perform data cleaning on the initial text information, remove invalid characters and stop words in the initial text information, and obtain text information corresponding to the voice file. 9. The device according to claim 7, wherein the converting unit comprises: The second processing subunit is configured to apply a preset word segmentation tool to perform word segmentation processing on the text information, and obtain multiple words corresponding to the text information; The second conversion subunit is used to apply the pre-set natural language processing tool NLTK to carry out part-of-speech tagging on each of the words, and based on the part-of-speech of each of the words, convert each of the words into its corresponding 300-dimensional vector; The input subunit is configured to input the 300-dimensional vector corresponding to each word into the BiLSTM, and obtain the text vector corresponding to the text information output by the BiLSTM. 10. The device according to claim 9, wherein the feature fusion unit comprises: The second acquisition subunit is used to acquire the frame speech features of each frame of speech in the speech feature information; The weighting subunit is used to apply a preset attention mechanism based on the word features of each of the words and each of the frame voice features, weighted to obtain the word voice features of the voice corresponding to each of the words; The splicing subunit is used to splice the word phonetic feature of the voice corresponding to each word with the 300-dimensional vector corresponding to the word, to obtain the fusion feature corresponding to the voice file.

Images