CN117174082A - Training and execution method, device, equipment and storage medium of voice wake-up model - Google Patents

Abstract

This application provides a training method, execution method, device, equipment and storage medium for a voice wake-up model. The voice awakening model includes a neural network structure, an awakening word classification branch and a starting and ending point judgment branch. The training method includes: obtaining audio samples and performing acoustic feature processing on the audio samples to obtain acoustic feature frames; inputting the acoustic feature frames into the neural network structure to generate an output vector, wherein the output vector has the first attribute and/or the second attribute. Attributes, the first attribute is whether it contains a wake-up word, and the second attribute is the start point position and end point position of the wake-up word in the output vector; and based on the output vector, the wake-up word classification branch and the start and end point judgment branch are trained respectively. This application can improve the detection accuracy of wake-up words.

Description

Training and execution method, device, equipment and storage medium of voice wake-up model

Technical field

The present application relates to the field of computer technology, and in particular to training methods, execution methods, devices, equipment and storage media of voice arousal models.

Background technique

With the development of computer technology, more and more smart devices have emerged that support voice interaction functions. For example, a large number of devices such as smart speakers, smart TVs, and cars are equipped with voice interaction functions. They can directly control and operate devices using voice, providing users with A great convenience. In the process of voice interaction, wake word detection is generally used as the beginning of a round of interaction. Once the wake word is triggered, the device will actively collect user voice commands. In order to reduce the sudden triggering of the device caused by misrecognition of the wake word, many solutions will add a second level wake word verification in the cloud based on the first level wake word verification on the local device, which requires accurate interception of the wake word.

In traditional technology, common methods of intercepting wake words include:

First, at the wake word trigger position, a fixed-length audio is intercepted forward. However, considering that different users have different pronunciation speeds, the fixed-length interception is generally set according to the length of a slow speaking speed, so that for a fast speaking wake-up call Word triggering may intercept some irrelevant audio, thus affecting the accuracy of secondary wake-up verification.

Second, use Voice Activate Detection (VAD) to intercept the starting point of the wake word, but this method can only be used in a quieter environment, but in a noisy environment, the performance of VAD declines rapidly and cannot Effectively obtains the starting position of the wake word.

Therefore, with the rapid development of voice interaction functions, there is an urgent need for an accurate and efficient voice wake-up technology.

Contents of the invention

This application aims to solve at least one of the technical problems existing in the prior art or related technologies. To this end, this application provides a training method, execution method, device, equipment and storage medium for a voice wake-up model.

According to the first aspect of the present application, a training method for a voice awakening model is provided, wherein the voice awakening model includes a neural network structure, an awakening word classification branch, and a start and end point judgment branch. The training method includes:

Obtain audio samples and perform acoustic feature processing on the audio samples to obtain acoustic feature frames;

Input the acoustic feature frame to the neural network structure to generate an output vector, wherein the output vector has a first attribute and/or a second attribute, the first attribute is whether the wake-up word is included, and the second attribute is the position of the wake-up word in the output vector. the starting point position and the ending point position; and

According to the output vector, the wake word classification branch and the start and end point judgment branch are trained respectively, including:

The output vectors are input to the wake word classification branch and the start and end point judgment branch respectively, and the wake word classification branch outputs the probability that the audio sample contains the wake word, and the start and end point judgment branch outputs the starting point position and end of the wake word in the audio sample. point position; and

The output results of the wake word classification branch and the output results of the start and end point judgment branches are used to adjust the parameters of the wake word classification branch and the parameters of the start and end point judgment branches respectively to obtain the updated wake word classification branch and the start and end point judgment branches.

As an embodiment of the present application, the wake-up word classification branch at least includes a first fully connected layer structure connected to a neural network structure, and the output vector of the neural network structure includes first label data for the wake-up word classification branch. The first label data Relating to the first attribute, the first label data is used to characterize whether the output vector contains the wake-up word and is input to the first fully connected layer structure as the first supervised learning target.

As an embodiment of the present application, using the output result of the wake-up word classification branch to adjust the parameters of the wake-up word classification branch includes: determining the first loss of the wake-up word classification branch according to the first attribute of the output vector and the output result of the wake-up word classification branch. function; and, adjusting the parameters of the wake word classification branch based on the first loss function.

As an embodiment of the present application, adjusting the parameters of the wake-up word classification branch based on the first loss function includes: iteratively updating the parameters of the wake-up word classification branch according to the calculated value of the first loss function until convergence, so as to obtain the updated wake-up word classification. branch.

As an embodiment of the present application, the first loss function of the wake-up word classification branch is cross entropy.

As an embodiment of this application, the calculation formula of the first loss function is:

in, is the label of the wake-up word; when the output vector contains the wake-up word, /> When the output vector does not contain the wake word, /> y is the wake word probability output by the wake word classification branch, and its range is [0, 1].

As an embodiment of the present application, the start and end point judgment branch at least includes a second fully connected layer structure connected to the neural network structure, and the output vector of the neural network structure includes second label data for the start and end point judgment branch. The second label data Relating to the second attribute, the second label data is used to characterize the start point position and the end point position of the wake-up word in the output vector and is input to the second fully connected layer structure as the second supervised learning target.

As an embodiment of the present application, using the output result of the start and end point judgment branch to adjust the parameters of the start and end point judgment branch includes: determining the second loss of the start and end point judgment branch according to the second attribute of the output vector and the output result of the start and end point judgment branch. function; and, adjusting the parameters of the start and end point judgment branch based on the second loss function.

As an embodiment of the present application, adjusting the parameters of the start and end point judgment branches based on the second loss function includes: iteratively updating the parameters of the start and end point judgment branches according to the calculated value of the second loss function until convergence, so as to obtain the updated start and end point judgments. branch.

As an embodiment of the present application, the second loss function of the start and end point determination branch is the mean square error between the output starting point position and end point position and the true starting point position and end point position.

As an embodiment of this application, the calculation formula of the second loss function is:

L ₂ =(s1-k1) ² +(s2-k2) ²

Among them, L ₂ is the mean square error value; s1 is the real starting point position of the wake-up word in the output vector; s2 is the real end point position of the wake-up word in the output vector; k1 is the starting point position of the start and end point judgment branch output; k2 is the end point position of the start and end point judgment branch output.

As an embodiment of the present application, the training method further includes: using the output result of the wake word classification branch and/or the output result of the start and end point determination branch to adjust parameters of the neural network structure to obtain an updated neural network structure.

As an embodiment of the present application, performing feature processing on audio samples includes: performing endpoint detection on wake-up words in audio samples, starting from the starting point of the wake-up words, intercepting data frames according to a fixed-length time window until the end of the wake-up words points to obtain a plurality of data frames of the same time length; the plurality of data frames are in the form of one-dimensional arrays; and the audio spectrum features are extracted from the one-dimensional arrays and a two-dimensional feature array is output.

As an embodiment of the present application, in response to the intercepted data frame including a plurality of wake words, the data frame containing the first wake word is retained.

As an embodiment of the present application, inputting the acoustic feature frame into the neural network structure to generate an output vector includes: inputting the two-dimensional feature array into the neural network structure and generating a one-dimensional vector through a convolution algorithm.

According to the second aspect of the present application, a method for executing a voice arousal model is also provided, including:

Get the voice data to be detected;

Input the voice data to be detected into any of the above-trained voice arousal models;

Obtain the output results of the wake word classification branch and the output results of the start and end point judgment branches; and

The wake-up result of the voice wake-up model is output based on the output result of the wake-up word classification branch and/or the output result of the start and end point judgment branch.

As an embodiment of the present application, the execution method further includes: sending the output result of the wake word classification branch and/or the output result of the start and end point determination branch to the speech recognition cloud platform for secondary recognition of the wake word.

According to the third aspect of the present application, a training device for a voice wake-up model is also provided. The voice wake-up model includes a neural network structure, a wake-up word classification branch, and a starting and ending point judgment branch. The training device includes:

An acquisition module is used to acquire audio samples and perform acoustic feature processing on the audio samples to obtain acoustic feature frames;

A generation module for inputting the acoustic feature frame into the neural network structure to generate an output vector, wherein the output vector has a first attribute and/or a second attribute, the first attribute is whether it contains a wake-up word, and the second attribute is the wake-up word. The starting point position and the ending point position in the output vector; and

The training module trains the wake word classification branch and the start and end point judgment branch respectively based on the output vector, including:

The output vectors are input to the wake word classification branch and the start and end point judgment branch respectively, and the wake word classification branch outputs the probability that the output vector contains the wake word, and the start and end point judgment branch outputs the starting point position and end of the wake word in the output vector. point position; and

The output results of the wake word classification branch and the output results of the start and end point judgment branches are used to adjust the parameters of the wake word classification branch and the parameters of the start and end point judgment branches respectively to obtain an updated voice arousal model.

According to a fourth aspect of the present application, a voice wake-up device is also provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, it implements any of the above training methods and/or any of the above execution methods.

According to the fifth aspect of the present application, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, any of the above training methods and/or any of the above execution methods are implemented.

According to a sixth aspect of the present application, a computer program is also provided, which when executed by a processor implements any of the above training methods and/or any of the above execution methods.

The training method of the speech wake-up model in the embodiment of the present application adopts a multi-task training method. The wake-up word classification branch and the start and end point judgment branch are set in parallel at the end of the wake-up word detection network, and at the same time, the wake-up word classification task and the start and end of the wake-up word are The point regression task is trained to output whether the audio sample contains the wake-up word, and also output the starting point position and end point position of the wake-up word in the audio sample. Therefore, the speech wake-up model trained in the embodiment of the present application can quickly and accurately obtain the starting point and end point of the wake-up word, which helps to intercept the wake-up word in the audio frame more accurately. In addition, since the classification task of wake words and the regression task of start and end points of wake words are trained at the same time, it can help enhance the effect of the neural network structure. Compared with a single training task, the accuracy of wake word detection can be further improved.

In addition, the voice wake-up execution method in the embodiment of the present application is based on the above-trained voice wake-up model. It can output whether the wake-up word is included in the audio sample while also outputting the starting point position and end point of the wake-up word in the audio sample. position, significantly improving the accuracy of wake word detection and greatly improving the performance of voice wake-up. In addition, this application can intercept the wake-up word in the audio frame more accurately in scenarios that require secondary verification on the cloud. Therefore, it is particularly advantageous in scenarios that require secondary verification on the cloud.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to describe the embodiments. The drawings in the following description are only some embodiments of the present application. For ordinary people in the art Technical personnel can also obtain other drawings based on these drawings without exerting creative labor.

Figure 1 is a schematic structural diagram of a voice arousal model provided by an embodiment of the present application.

FIG. 2 is a schematic flowchart of a training method for a voice arousal model provided by an embodiment of the present application.

FIG. 3 is a schematic flowchart of the training method of the wake word classification branch in the voice wake-up model provided by the embodiment of the present application.

FIG. 4 is a schematic flowchart of the training method of the starting and ending point judgment branch of the voice arousal model provided by the embodiment of the present application.

FIG. 5 is another schematic flowchart of the training method of the voice arousal model provided by the embodiment of the present application.

FIG. 6 is a schematic flowchart of the execution method of the voice wake-up model provided by the embodiment of the present application.

FIG. 7 is a schematic structural diagram of a voice arousal model training device provided by an embodiment of the present application.

Figure 8 is a schematic structural diagram of a voice wake-up device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

The embodiments of the present application provide a training method and execution method of a voice wake-up model, which are applied to the fields of machine learning and speech technology in the field of artificial intelligence. Specifically, the embodiments of the present application can be applied to smart speaker voice interaction and smart vehicle voice interaction. , smart TV voice interaction and other scenarios that require voice interaction to improve the detection accuracy of wake words.

In order to facilitate understanding of the technical solutions provided by this application, exemplary application scenarios of the embodiments of this application are described below.

The training method of the voice wake-up model provided by the embodiment of the present application can be executed by the training device of the voice wake-up model provided by the embodiment of the present application. The training device of the voice wake-up model provided by the embodiment of the present application can be a terminal device or a server. Illustratively, the voice wake-up model training method provided by the embodiment of the present application can be applied to a terminal device. For example, it can be implemented through a processor, an application program or a web page in the terminal device. There is data communication between the terminal device and the server. This application The embodiment does not limit this. The embodiments of this application do not limit the specific type of the terminal device. For example, the terminal device may be a smartphone, a personal computer, a tablet computer, a wearable device, a vehicle-mounted terminal, etc. The embodiments of the present application do not limit the type and number of servers. For example, the server can be a single independent server or a server cluster. The embodiments of the present application only take this as an example and are not limited thereto.

After the voice wake-up model is trained, the voice wake-up model can be applied to a terminal device that requires a voice recognition function to perform the voice wake-up function. For example, the voice wake-up model can be applied to terminal devices such as smart speakers, smart home devices, smartphones, vehicle-mounted terminals, and wearable devices. The embodiments of this application do not limit this.

Figure 1 is a schematic structural diagram of a voice arousal model provided by an embodiment of the present application. As shown in Figure 1, the voice wake-up model provided by the embodiment of the present application includes an acoustic feature processing unit, a neural network structure, a wake-up word classification branch, and a start and end point determination branch. Among them, the acoustic feature processing unit is used to extract acoustic features from the input audio signal, and input the extracted acoustic features into the neural network structure. The output of the neural network structure serves as the input of the wake word classification branch and the start and end point judgment branch, and is used to simultaneously train the wake word classification branch to identify wake words and non-wake words, and calculate the start and end points of the wake word of the start and end point judgment branch. Point return mission. Advantageously, the wake-up word classification branch and the start and end point judgment branch share the output of the same neural network structure, which helps to save computing costs.

FIG. 2 is a schematic flowchart of a training method for a voice arousal model provided by an embodiment of the present application.

As shown in Figure 2, the training method of the voice arousal model provided by the embodiment of the present application includes the following steps:

Step S201: Obtain audio samples and perform acoustic feature processing on the audio samples to obtain acoustic feature frames.

In this step, the audio samples include wake word samples and non-wake word samples. In some examples, the wake word sample may be a manually recorded audio containing the wake word. In addition, the diversity and complexity of the audio samples can be increased by adding noise later, changing the position of the wake-up word in the audio, etc., thereby enhancing the training effect. Non-wake word samples can be audio files that do not contain wake words. They can be human voices in daily scenes, such as audio from movies and TV series, or they can be automatically collected and generated by a computer.

Before training the voice wake-up model, the type of each audio sample can be marked through machine recognition or manual recognition, that is, whether the audio sample contains the wake-up word.

At the same time, since the starting and ending point training branches are set up in the training method of this embodiment, before training the voice awakening model, the starting and ending points of the wake-up words in the wake-up word samples can also be accurately marked through machine recognition or manual recognition. Among them, the start and end points of the wake-up word are the starting time point (also called the starting point) and the end time point (also called the end point) of the wake-up word in the audio sample. It can be understood that in some examples, non-wake word samples do not need to be marked with starting points and ending points.

In this step, the acoustic feature processing of the audio samples includes: for the input audio samples, starting from the starting point of the wake-up word, intercepting data frames according to a fixed-length time window until the end point of the wake-up word, so as to extract the data from the audio sample. Obtain multiple data frames of the same time length. The plurality of data frames may be in the form of one-dimensional array data. In some examples, the start and end points of the wake word can be identified from audio samples based on Voice Endpoint Detection (VAD) technology. In some examples, in response to the intercepted data frame including a plurality of the wake-up words, the data frame containing the first wake-up word is retained, thereby avoiding repeated calculations to save computing resources and improve recognition speed.

Further, acoustic features are extracted from the data frame of the above one-dimensional array, and a two-dimensional feature array is output. In some examples, Mel-Frequency Cepstral Coefficients (MFCC) can be used to perform acoustic feature extraction and feature enhancement on the data frame of the one-dimensional array to output a two-dimensional feature array. Specifically, in the MFCC feature extraction method, the audio data is first subjected to Fast Fourier Transform (FFT), and then the spectrum of different frequency bands is filtered and compressed according to the human auditory characteristics to obtain the MFCC features. In other examples, static filter network (Filter Bank, FBANK), normalized energy coefficients (Power-Normalized Coefficients, PNCC), etc. can also be used to extract audio spectrum features, and this application does not impose specific restrictions on this.

For example, for the input audio sample, starting from the starting point of the audio information containing the wake-up word identified in the audio sample, data frames are intercepted through the time window every a second until the end of the audio information containing the wake-up word. Based on this, m data frames of the same length can be intercepted from the audio sample and used as training samples for the input neural network in the form of one-dimensional array data. Furthermore, MFCC coefficients can be used to extract and enhance features of the input training samples, and output n×m 32-bit two-dimensional feature arrays as acoustic feature frames for subsequent use.

Step S202: Input the acoustic feature frame into the neural network structure to generate an output vector, where the output vector has a first attribute and/or a second attribute. The first attribute is whether it contains a wake-up word, and the second attribute is the wake-up word. The starting point position and the ending point position in the output vector.

In this step, the neural network structure may be the backbone structure of the wake word recognition network. The Backbone structure can be constructed using a convolutional neural network. Generally speaking, it can be composed of convolutional layers and pooling layers. The Backbone structure can be a convolutional neural network (CNN), a recurrent neural network (RNN), an attention network, and other types of recognition network structures. This application does not impose specific restrictions on this.

When the acoustic feature frame is input to the Backbone structure, the Backbone structure can be compressed through a convolution algorithm to generate a one-dimensional vector and then output. This one-dimensional vector is used as the common input of the subsequent wake word classification branch and the start and end point judgment branch.

It can be understood that, as mentioned in the aforementioned step S201, before training the voice wake-up model, the audio samples include wake-up word samples and non-wake-up word samples, that is, whether they contain wake-up words have been marked. Moreover, for the wake word sample, the start point and end point of the wake word have been marked. Therefore, in this step, correspondingly, the output vector of the neural network structure has a first attribute and/or a second attribute, where the first attribute is whether it contains the wake-up word, and the second attribute is the position of the wake-up word in the output vector. The starting point position and the ending point position.

Step S203: According to the output vector, separately train the wake-up word classification branch and the start and end point determination branch.

In this step, the wake word classification branch is used to train the classification task of judging whether it is a wake word. During the training process, the wake word classification branch uses all audio samples for training. Specifically, the input of the wake word classification branch is one-dimensional vector data processed by the neural network structure, including wake word and non-wake word samples. , the probability that the sample is a wake-up word and the probability that the sample is not a wake-up word are output through training. It can be understood that the sum of the two probabilities is 1.

The start and end point judgment branch is used to expand the regression task of calculating the start point and end point of the wake word, and output the calculated start and end points of the wake word. For example, during the training process, the start and end point judgment branch only inputs samples containing wake-up words for training. Specifically, the start and end point judgment branch inputs one-dimensional vector data obtained after processing by the neural network structure, and outputs the samples through training. The position of the start point and the end point of the wake word.

Continuing to refer to Figure 2, step S203 includes the following steps:

Step S2031: Input the output vector of the neural network structure to the wake word classification branch and the start and end point judgment branch respectively, and the wake word classification branch outputs the probability that the audio sample contains the wake word, and the start and end point judgment branch outputs the frequency of the wake word in the audio sample. The starting point position and the ending point position in ; and

Step S2032: Use the output results of the wake word classification branch and the output results of the start and end point judgment branches to respectively adjust the parameters of the wake word classification branch and the parameters of the start and end point judgment branches to obtain updated wake word classification branches and start and end point judgment branches.

In steps S2031 and S2032, taking the wake-up word classification branch as an example, in some examples, it at least includes a first fully connected layer structure connected to a neural network structure, and the output vector of the neural network structure includes a wake-up word classification branch. The first label data, the first label data relates to the first attribute, the first label data is used to characterize whether the output vector contains the wake-up word and is input to the first fully connected layer structure as the first supervised learning target.

In some examples, see FIG. 3 , which is a schematic flowchart of the training method of the wake-up word classification branch of the speech wake-up model provided by the embodiment of the present application. In step S2032, the output result of the wake-up word classification branch is used to adjust the wake-up word classification. Branch parameters include:

Step S301: Determine the first loss function of the wake-up word classification branch according to the first attribute of the output vector and the output result of the wake-up word classification branch.

In this step, the first loss function of the wake-up word classification branch is cross entropy.

Step S302: Adjust the parameters of the wake-up word classification branch based on the first loss function.

In this step, the parameters of the wake-up word classification branch are iteratively updated according to the calculated value of the first loss function until convergence, so as to obtain the updated wake-up word classification branch.

In one example, the first loss function is calculated as:

in, is the label of the wake-up word; when the output vector contains the wake-up word, /> When the output vector does not contain the wake word, /> y is the wake word probability output by the wake word classification branch, and its range is [0,1].

Based on this, the calculated output wake word probability can be compared with the information of whether the annotated audio sample in the audio sample contains the wake word. On the one hand, it can be used to evaluate the training effect, and on the other hand, it can be used to adjust parameters to optimize wake word classification. branch, and use the wake word classification branch after parameter optimization as the updated wake word classification branch. In this embodiment, the parameters can be adjusted so that the wake-up word probability output by the wake-up word classification branch is close to or equal to the label of the marked wake-up word.

In steps S2031 and S2032, taking the start and end point judgment branch as an example, it at least includes a second fully connected layer structure connected to the neural network structure, and the output vector of the neural network structure includes second label data for the start and end point judgment branch. , the second label data relates to the second attribute, and the second label data is used to characterize the start point position and end point position of the wake-up word in the output vector and is input to the second fully connected layer structure as the second supervised learning target.

In some examples, see FIG. 4 , which is a schematic flowchart of the training method of the start and end point judgment branch of the voice arousal model provided by the embodiment of the present application. In step S2032, the output result of the start and end point judgment branch is used to adjust the start and end point judgment. Branch parameters include:

Step S401: Determine the second loss function of the start and end point judgment branch according to the second attribute of the output vector and the output result of the start and end point judgment branch.

In this step, the second loss function of the start and end point judgment branch is the mean square error between the output starting point position and end point position and the real starting point position and end point position.

Step S402: Adjust the parameters of the start and end point determination branch based on the second loss function.

In this step, the parameters of the start and end point judgment branches are iteratively updated according to the calculated value of the second loss function until convergence, so as to obtain the updated start and end point judgment branches.

In one example, the second loss function is calculated as:

L ₂ =(s1-k1) ² +(s2-k2) ²

Among them, L ₂ is the mean square error value;

s1 is the real starting point position of the wake-up word in the output vector;

s2 is the real end point position of the wake-up word in the output vector;

k1 is the starting point position of the starting and ending point judgment branch output;

k2 is the end point position of the start and end point judgment branch output.

Based on this, the calculated start and end points of the wake word can be compared with the marked start and end point information in the audio sample. On the one hand, it can be used to evaluate the training effect. On the other hand, it can be used to adjust parameters to optimize the start and end point judgment branch, and The starting and ending point judgment branches after parameter optimization are used as the updated starting and ending point judgment branches. In this embodiment, parameters can be adjusted so that the start point position k1 and the end point position k2 of the start and end point judgment branch output are close to or equal to the real start point position s1 and the real end point position s2 of the wake word in the output vector respectively.

Referring again to Figure 1, in some examples, the output results of the wake word classification branch and/or the output results of the start and end point judgment branches can be used to adjust the parameters of the neural network structure to obtain an updated neural network structure. For example, the neural network structure can use an error backpropagation algorithm to modify the parameters in the initial neural network structure during the training process, so that the reconstruction error loss of the neural network structure becomes smaller and smaller until the error loss converges.

In order to enable those skilled in the art to better understand the present application, a more detailed embodiment is enumerated below. As shown in Figure 5, the embodiment of the present application provides a training method for a voice arousal model, which includes the following steps:

Step S501: Generate wake word samples and non-wake word samples.

In this step, the wake-up word of the voice wake-up model is set to "Hi Lexin", and other audio samples are non-wake-up words. The wake-up word can be sampled in a quiet, low-noise environment by manually recording the voice "Hi Lexin". In order to enhance the training effect, after collecting the wake word samples, the complexity of the wake word samples is increased and the training effect is improved by adding noise and adjusting the position of the wake word in the sample audio. Non-wake word samples are obtained by intercepting TV series, movies and other audios that contain human voices but do not contain wake words.

Step S502: Mark the starting and ending points of the wake-up word.

In this step, the start and end points of the wake word in the audio sample need to be given for training purposes. In this regard, the time starting point and time end point of the wake word in the audio sample can be marked through machine annotation or manual annotation. There is no need to label non-wake word samples.

Step S503: Intercept the data frame.

In this step, through a 1s long window, starting from the starting point where the wake word in the audio sample is recognized, moving at intervals of 32ms, intercepting several data frames, and optionally uploading them to the cloud for secondary wake word recognition Identify.

In one example, in order to avoid repeated calculations to save computing resources and improve recognition speed, among several intercepted data frames containing wake words, only the first data frame can be uploaded to the cloud for secondary recognition of wake words. .

Step S504: Feature extraction and feature enhancement of the data frame.

In this step, the one-dimensional array in the speech sample is used for feature extraction and feature enhancement through MFCC, and 50*32 32-bit feature two-dimensional arrays are output.

Step S505: Output a one-dimensional vector through convolution calculation

In this step, 50*32 32-bit feature two-dimensional array samples are input into the neural network structure, and a one-dimensional vector is output after convolution calculation.

Step S506: Train the wake word classification branch and the start and end point judgment branch

In this step, the one-dimensional vector output by the neural network structure is input into the wake word classification branch and the start and end point judgment branch respectively, and simultaneously runs the classification task of wake words and non-wake words, and the regression task of calculating the start point and end point of the wake word .

Both the wake word classification branch and the start and end point judgment branch adopt a fully connected layer structure, also known as a linear layer structure, which is used for the output of the last layer of the neural network for subsequent operations. Among them, the loss function of the wake-up word classification branch is cross entropy. The loss function of the start and end point judgment branch is the mean square error between the output starting point position and end point position and the real starting point position and end point position. Correspondingly, for the specific data types and parameter adjustment methods of the wake word classification branch and the start and end point determination branch, please refer to the detailed description of the first loss function and the second loss function above, and will not be described again here. Considering that the wake word classification branch and the start and end point judgment branch use different parameters in subsequent operations, the two tasks need to be trained separately.

The wake word classification branch is used to train wake word classification tasks. The input of the wake word classification branch includes one-dimensional vector data samples of wake words and non-wake words. After training, two sets of data are output, which are the probability that the sample is a wake word and the probability that the sample is not a wake word. The sum of the two probabilities is 1. The output results of the wake word classification branch are compared with the audio information to evaluate the training effect of the model and used to adjust parameters to optimize the wake word classification branch.

The start and end point judgment branch is used to train the regression task of the start point and end point of the wake word. The input of the start and end point judgment branch only contains one-dimensional vector data samples of the wake-up word. After training, two sets of data are output, which are the starting point position of the wake-up word and the end point position of the wake-up word. The output results of the wake word classification branch are compared with the audio information to evaluate the training effect of the model, and are used to adjust parameters to optimize the start and end point determination branch.

The embodiments of this application have been tested extensively and proven effective. For example, on an ESP32-S3 with very limited resources, it used 5% of the computing resources of the CPU to achieve a recognition rate of more than 95%, and passed Amazon's Alexa performance test.

In summary, the training method of the speech wake-up model in the embodiment of the present application adopts a multi-task training method, and sets up the wake-up word classification branch and the start and end point judgment branch in parallel at the end of the wake-up word detection network, and simultaneously performs the wake-up word classification task and The regression task of the start and end points of the wake word is trained to output not only whether the audio sample contains the wake word, but also the starting point position and end point position of the wake word in the audio sample. Therefore, the speech wake-up model trained in the embodiment of the present application can quickly and accurately obtain the starting point and end point of the wake-up word, which helps to intercept the wake-up word in the audio frame more accurately. In addition, since the classification task of wake words and the regression task of start and end points of wake words are trained at the same time, it can help enhance the effect of the neural network structure. Compared with a single training task, the accuracy of wake word detection can be further improved. In addition, because the embodiment of the present application can intercept the wake word more accurately, it can particularly show advantages in scenarios that require secondary verification in the cloud.

Based on the above embodiments, the trained voice wake-up model can be preset in the voice wake-up device, so that the voice wake-up device has the voice wake-up function. Correspondingly, the embodiment of the present application provides a method for executing the voice arousal model. As shown in Figure 6, the execution method includes the following steps:

Step S601: Obtain the voice data to be detected.

Step S602: Input the voice data to be detected into the aforementioned trained voice arousal model.

Step S603: Obtain the output result of the wake word classification branch and the output result of the start and end point determination branch.

Step S604: Output the wake-up result of the voice wake-up model based on the output result of the wake-up word classification branch and/or the output result of the start and end point determination branch.

In some embodiments, the execution method of the above-mentioned voice wake-up model also includes: sending the output result of the wake-up word classification branch and/or the output result of the start and end point determination branch to the speech recognition cloud platform for secondary recognition of the wake-up word.

The voice wake-up execution method in the embodiment of the present application is based on the above-trained voice wake-up model. It can output whether the wake-up word is included in the audio sample while also outputting the starting point position and the end point position of the wake-up word in the audio sample. It can significantly improve the accuracy of wake word detection and greatly improve the performance of voice wake-up. In addition, this application can intercept the wake-up word in the audio frame more accurately in scenarios that require secondary verification on the cloud. Therefore, it is particularly advantageous in scenarios that require secondary verification on the cloud.

Referring to Figure 7, an embodiment of the present application also provides a training device for a voice arousal model, which includes an acquisition module, a generation module and a training module. The acquisition module is used to acquire audio samples and perform acoustic feature processing on the audio samples to obtain acoustic feature frames. The generation module is used to input the acoustic feature frame into the neural network structure to generate an output vector, wherein the output vector has a first attribute and/or a second attribute. The first attribute is whether the wake-up word is included, and the second attribute is whether the wake-up word is included. Outputs the starting point position and the ending point position in the vector. The training module is used to train the wake word classification branch and the start and end point judgment branch respectively according to the output vector, including: inputting the output vector to the wake word classification branch and the start and end point judgment branch respectively, and the output vector of the wake word classification branch contains The probability of the wake-up word, and the starting point position and the end point position of the wake-up word in the output vector output by the start and end point judgment branch; and, using the output result of the wake word classification branch and the output result of the start and end point judgment branch to adjust the wake word classification respectively The parameters of the branch and the starting and ending points are used to determine the parameters of the branch to obtain the updated voice arousal model.

In this embodiment, the specific limitations, implementation principles, and beneficial effects of the training device for the voice arousal model can be found in the description of the training method for the voice arousal model above, and will not be described again here. Each of the above modules can be implemented in whole or in part through software, hardware and combinations thereof. Each of the above modules may be embedded in or independent of the processor of the computer device in the form of hardware, or may be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

Referring to Figure 8, an embodiment of the present application also provides a voice wake-up device, which includes a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, it can implement the training steps and steps of the voice wake-up model in the above embodiments. / Or the execution steps of the voice wake-up model.

Embodiments of the present application also provide a computer-readable storage medium on which a computer program is stored. When executed by a processor, the computer program can implement the training steps of the voice arousal model and/or the execution of the voice arousal model in the above embodiments. step.

An embodiment of the present application also provides a computer program, which when executed by a processor can implement the training steps of the voice arousal model and/or the execution steps of the voice arousal model in the above embodiments.

In the above embodiments, the implementation principles and beneficial effects of the voice wake-up device, computer-readable storage medium and computer program of the voice wake-up model can be found in the above description of the training method and execution method of the voice wake-up model, and will not be described again here.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage. In the media, when executed, the computer program may include the processes of the above method embodiments. Any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetic variable memory (Magneto resistive Random Access Memory (MRAM), ferroelectric memory (Ferroelectric Random Access Memory (FRAM)), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory. By way of illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM). The databases involved in the various embodiments provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include blockchain-based distributed databases, etc., but are not limited thereto. The processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to this.

While various embodiments of various aspects of the present application have been described for purposes of this disclosure, they should not be construed as limiting the teachings of the disclosure to these embodiments. Features disclosed in one specific embodiment are not limited to this embodiment, but may be combined with features disclosed in different embodiments. For example, one or more features and/or operations of the method according to the present application described in one embodiment can also be applied in another embodiment individually, in combination or as a whole. Those skilled in the art will understand that there are more possible optional implementations and modifications, and various changes and modifications can be made to the above system without departing from the scope defined by the claims of the present application.

Claims (20)

1. The method for training the voice wake-up model is characterized in that the voice wake-up model comprises a neural network structure, wake-up word classification branches and start and stop point judgment branches, and comprises the following steps:

acquiring an audio sample, and performing acoustic feature processing on the audio sample to obtain an acoustic feature frame;

inputting the acoustic feature frame into a neural network structure to generate an output vector, wherein the output vector has a first attribute and/or a second attribute, the first attribute is whether a wake-up word is contained or not, and the second attribute is a starting point position and an ending point position of the wake-up word in the output vector; and

Training the wake-up word classification branch and the start-stop point judgment branch according to the output vector, wherein the training comprises the following steps:

the output vector is respectively input into the wake-up word classification branch and the start-stop point judgment branch, the probability that the audio sample contains the wake-up word is output by the wake-up word classification branch, and the start point position and the end point position of the wake-up word in the audio sample are output by the start-stop point judgment branch; and

and respectively adjusting parameters of the wake-up word classification branch and parameters of the start-stop point judgment branch by using the output result of the wake-up word classification branch and the output result of the start-stop point judgment branch so as to obtain the updated wake-up word classification branch and the updated start-stop point judgment branch.

2. The method of claim 1, wherein the wake word classification branch comprises at least a first fully-connected layer structure connected to the neural network structure, an output vector of the neural network structure comprising first tag data for the wake word classification branch, the first tag data relating to a first attribute, the first tag data being used to characterize whether the wake word is contained in the output vector and input to the first fully-connected layer structure as a first supervised learning objective.

3. The method of claim 1, wherein adjusting parameters of the wake word classification branch using output results of the wake word classification branch comprises:

determining a first loss function of the wake-up word classification branch according to the first attribute of the output vector and the output result of the wake-up word classification branch;

and adjusting parameters of the wake-up word classification branches based on the first loss function.

4. The method of claim 3, wherein the adjusting parameters of the wake word classification branch based on the first loss function comprises:

and iteratively updating parameters of the wake-up word classification branches according to the calculated value of the first loss function until convergence to obtain the updated wake-up word classification branches.

5. A method according to claim 3, wherein the first loss function of the wake word classification branch is cross entropy.

6. The method of claim 5, wherein the first loss function is calculated as:

wherein,labels that are wake words; when the wake word is included in the output vector, and (2)>When the wake word is not included in the output vector, and (2)>

And y is the wake-up word probability output by the wake-up word classification branch, and the range of y is [0,1].

7. The method of claim 1, wherein the start-stop decision branch comprises at least a second fully-connected layer structure connected to a neural network structure, an output vector of the neural network structure comprising second tag data for the start-stop decision branch, the second tag data relating to a second attribute, the second tag data being used to characterize a start point position and an end point position of the wake-up word in the output vector and being input to the second fully-connected layer structure as a second supervised learning objective.

8. The method of claim 1, wherein adjusting parameters of the start-stop point determination branch using output results of the start-stop point determination branch comprises:

determining a second loss function of the start-stop point judging branch according to a second attribute of the output vector and an output result of the start-stop point judging branch;

and adjusting parameters of the start and stop point judgment branch based on the second loss function.

9. The method of claim 8, wherein the adjusting the parameters of the start-stop decision branch based on the second loss function comprises:

and iteratively updating the parameters of the start-stop point judgment branch according to the calculated value of the second loss function until convergence to obtain the updated start-stop point judgment branch.

10. The method of claim 8, wherein the second loss function of the start-stop point judgment branch is a mean square error between the output start point position and end point position and the actual start point position and end point position.

11. The method of claim 10, wherein the second loss function is calculated as:

L ₂ ＝(s1-k1) ² +(s2-k2) ²

wherein L is ₂ Is a mean square error value;

s1 is the true starting point position of the wake-up word in the output vector;

s2 is the true end point position of the wake-up word in the output vector;

k1 is the starting point position of the branch output judged by the starting point and the ending point;

and k2 is the end point position of the branch output judged by the start point and the stop point.

12. The method of claim 1, further comprising: and adjusting parameters of the neural network structure by utilizing the output result of the wake-up word classification branch and/or the output result of the start-stop point judgment branch so as to obtain an updated neural network structure.

13. The method of claim 1, wherein the characterizing the audio sample comprises:

detecting the end point of the wake-up word in the audio sample, starting from the starting point of the wake-up word, intercepting the data frames according to a time window with a fixed length until the end point of the wake-up word to obtain a plurality of data frames with the same time length; the plurality of data frames are in the form of one-dimensional array data; and

And extracting the audio frequency spectrum characteristics of the one-dimensional array, and outputting a two-dimensional characteristic array.

14. The method of claim 13, wherein, in response to the intercepted data frame including a plurality of the wake words, a data frame including a first wake word is retained.

15. The method of claim 13, wherein the inputting the acoustic feature frame into a neural network structure to generate an output vector comprises:

inputting the two-dimensional feature array into the neural network structure, and generating a one-dimensional vector through a convolution algorithm.

16. A method of executing a voice wake model, comprising:

acquiring voice data to be detected;

inputting the voice data to be detected into a voice awakening model trained by any one of claims 1-15;

obtaining the output result of the wake-up word classification branch and the output result of the start-stop point judgment branch; and

and outputting the wake-up result of the voice wake-up model based on the output result of the wake-up word classification branch and/or the output result of the start-stop point judgment branch.

17. The method of claim 16, further comprising:

and sending the output result of the wake-up word classification branch and/or the output result of the start-stop point judgment branch to a voice recognition cloud platform to perform secondary recognition of the wake-up word.

18. A training device for a voice wake model, the voice wake model comprising a neural network structure, wake word classification branches and start and stop point judgment branches, the device comprising:

the acquisition module is used for acquiring an audio sample and carrying out acoustic feature processing on the audio sample to obtain an acoustic feature frame;

the generation module is used for inputting the acoustic feature frame into a neural network structure to generate an output vector, wherein the output vector has a first attribute and/or a second attribute, the first attribute is whether a wake-up word is contained or not, and the second attribute is a starting point position and an ending point position of the wake-up word in the output vector; and

the training module is used for training the wake-up word classification branch and the start-stop point judgment branch according to the output vector, and comprises the following steps:

the output vector is respectively input into the wake-up word classification branch and the start-stop point judgment branch, the probability that the output vector contains the wake-up word is output by the wake-up word classification branch, and the start point position and the end point position of the wake-up word in the output vector are output by the start-stop point judgment branch; and

And respectively adjusting parameters of the wake-up word classification branch and parameters of the start-stop point judgment branch by using the output result of the wake-up word classification branch and the output result of the start-stop point judgment branch so as to obtain an updated voice wake-up model.

19. A voice wakeup apparatus comprising a memory and a processor, the memory having stored therein a computer program, characterized in that the processor when executing the computer program implements the training method of any one of claims 1 to 15 and/or the execution method of claim 16 or 17.

20. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the training method of any one of claims 1 to 15 and/or the execution method of claim 16 or 17.