CN114155883B - Progressive type based speech deep neural network training method and device - Google Patents

Abstract

The invention discloses an advanced speech deep neural network training method, device, storage medium and electronic device. Wherein, the advanced voice deep neural network training method includes: obtaining a mixed voice sample and a target sample voice, wherein the mixed voice sample includes target voice and noise voice; inputting the mixed voice sample into a preset voice deep neural network model, The predicted target speech is obtained, wherein the preset speech neural network model includes an advanced extractor, an encoder and a reconstructor, and the preset speech deep neural network model is determined as the target speech deep neural network model, based on the training in this program. The speech deep neural network of advanced extractor, encoder and reconstructor solves the technical problem that the target speech cannot be effectively separated from the mixed speech in the prior art.

Description

Advanced speech deep neural network training method and device

technical field

The present invention relates to the related field of speech signal processing, in particular, to an advanced speech deep neural network training method, device, storage medium and electronic device.

Background technique

Smart devices such as smart speakers, hearing aids, smart headphones, etc. have become an indispensable part of people's daily life. The rapid development of these devices has benefited from the continuous improvement of voice interaction technology in recent years. During voice interaction, the speaker often speaks passwords in complex scenes. Therefore, the speaker's voice is usually disturbed by noise, reverberation, or other speakers. If these background noises or overlapping voices cannot be removed in time, it will seriously affect the back-end speech recognition, semantic recognition or wake-up applications. Therefore, it is necessary to take speech extraction and separation technology as the research focus of speech signal processing. Single-channel speech separation technology is the most widely researched and applied technology in speech separation algorithms. Compared with multi-channel speech separation tasks, its advantages are lower hardware requirements and costs, and less calculation, but the disadvantage is that algorithm design is more difficult. High, because single-channel speech separation mainly uses the signal collected by a single microphone, and is modeled by the difference in time-frequency domain acoustic and statistical properties between the target speech and the interference signal.

In recent years, the rapid development of neural network and deep learning technology has made speech separation technology widely studied in this field. The basic idea of the speech separation method based on deep learning is: establish a speech separation model, extract the characteristic parameters from the mixed speech, and then find the mapping relationship between the characteristic parameters and the characteristic parameters of the target speech signal through network training, and then mix any input The signal can output the signal of the target voice through the trained model, so as to achieve the purpose of voice separation. A lot of research work has been carried out on the end-to-end time domain and frequency domain algorithms. The algorithms in the frequency domain include Deep Clustering, DANet, uPIT, Deep CASA and other algorithms, and the algorithms in the time domain include Conv-TasNet, BLSTM-TasNet, FurcaNeXt, wavesplit, etc. Most of these algorithms are designed on the basis of pure speech separation. Although the separation effect is good, when these algorithms are applied in complex scenarios, the separation accuracy is greatly reduced. However, real life scenes are often accompanied by factors such as background noise, reverberation, and other speaker voices. If it is inevitable to study the separation of speech, it is inevitable to study the mixed speech that contains more interference factors. What method can be adopted to make the algorithm more efficient? Accurate and more efficient.

For the above problems, no effective solution has been proposed yet.

Contents of the invention

The embodiment of the present invention provides an advanced speech deep neural network training method, device, storage medium and electronic device to at least solve the technical problem in the prior art that the target speech cannot be effectively separated from the mixed speech .

According to an aspect of an embodiment of the present invention, there is provided an advanced speech-based deep neural network training method, comprising: acquiring a mixed speech sample and a target sample speech, wherein the mixed speech sample includes target speech and noise speech; Inputting the mixed speech sample into a preset speech deep neural network model to obtain a predicted target speech, wherein the preset speech deep neural network model includes an advanced extractor, a reconstructor and an encoder, and the encoder is used for performing feature extraction on the mixed speech to obtain a first feature, the advanced extractor is used to calculate a high-dimensional mapping relationship feature based on the first feature, and the reconstructor is used to obtain a high-dimensional mapping relationship feature according to the high-dimensional mapping relationship feature, to obtain the predicted target voice in the mixed voice sample; the loss function determined according to the target sample voice and the predicted target voice satisfies a preset condition, and the preset voice deep neural network model is determined to be the target voice depth neural network model.

Optionally, the encoder is used to perform feature extraction on the mixed speech to obtain the first feature, comprising: inputting the mixed speech sample into the preset speech deep neural network model, and the encoder includes The two-layer convolutional network, ReLU activation function and batch normalization are used to obtain the first feature.

Optionally, the advanced extractor is used to calculate and obtain high-dimensional mapping relationship features according to the first feature, including: the advanced extractor includes a plurality of advanced units, and each advanced unit includes : In the case of delayed neural network, ReLU activation function, batch normalization processing, time-delayed neural network, pooling layer, batch normalization processing, and graph convolution layer; each element in the first feature The corresponding advanced units are respectively input to obtain the high-dimensional mapping relationship features.

Optionally, inputting each element in the first feature into a corresponding advanced unit to obtain the high-dimensional mapping relationship feature includes: expressing the first feature as H={h0,... , hi,..., hM-1}, wherein, i=0 to M-1, the advanced unit includes M, that is, in the case of J={j0,...,ji,...,jM-1}; h0 Input to the first advanced unit to obtain the corresponding output p0; the result of adding h1 and p0 is input to the second advanced unit for calculation, and the output p1 corresponding to the position of h1 is obtained; after adding h2 and p1, it is input to the third advanced unit The unit obtains the output p2 corresponding to the h2 position; each position is calculated by analogy, until the final hM-1 is added to pM-2 to obtain the corresponding output pM-1, and the high-dimensional mapping relationship feature P={p0,..., pM-1}.

Optionally, the reconstructor is used to obtain the predicted target speech in the mixed speech samples according to the high-dimensional mapping relationship features, including: inputting the mapping relationship P into the reconstructor, and performing two-layer After convolutional network layer, ReLU activation function and batch normalization processing, the predicted target speech in the mixed speech samples is obtained.

Optionally, the loss function determined according to the target sample speech and the predicted target speech satisfies a preset condition, and determining the preset speech deep neural network model as the target speech deep neural network model includes: calculating the The proportional invariant signal-to-noise ratio of the target sample speech and the predicted target language, the loss function is determined according to the proportional invariant signal-to-noise ratio; according to the loss value of the loss function, the gradient descent method is used to adjust the The weight and offset of each parameter of the preset voice deep neural network model; the loss function determined according to the target sample voice and the predicted target voice meets the preset condition, and the preset voice deep neural network model is determined to be Target Speech Deep Neural Network Model.

According to an aspect of an embodiment of the present invention, there is provided an advanced-based voice deep neural network training device, including: an acquisition unit for acquiring mixed voice samples and target sample voices, wherein the mixed voice samples include target Speech and noise speech; a prediction unit, used to input the mixed speech sample into a preset speech deep neural network model to obtain a predicted target speech, wherein the preset speech deep neural network model includes an advanced extractor, reconstruction An encoder and an encoder, the encoder is used to perform feature extraction on the mixed speech to obtain a first feature, the advanced extractor is used to calculate a high-dimensional mapping relationship feature based on the first feature, and the heavy The constructor is used to obtain the predicted target speech in the mixed speech sample according to the high-dimensional mapping relationship feature; the determining unit is used to determine the loss function according to the target sample speech and the predicted target speech to meet a preset condition , determining the preset speech deep neural network model as the target speech deep neural network model.

Optionally, the prediction unit includes: an encoding module, configured to input the mixed speech samples into the preset speech deep neural network model, and activate the two-layer convolutional network and ReLU through the encoder. function and batch normalization to obtain the first feature.

Optionally, the prediction unit is also used to perform the following operations: the advanced extractor includes a plurality of advanced units, each advanced unit includes: a delayed neural network, a ReLU activation function, batch normalization In the case of processing, time-delay neural network, pooling layer, batch normalization processing, and graph convolution layer; input each element in the first feature into the corresponding advanced unit to obtain the high-dimensional map relationship features.

Optionally, the prediction unit is further configured to perform the following operation: the first feature is expressed as H={h0,...,hi,...,hM-1}, wherein, i=0 to M-1, the The above-mentioned advanced unit includes M, that is, in the case of J={j0,...,ji,...,jM-1}; h0 is input to the first advanced unit, and the corresponding output p0 is obtained; h1 and p0 are added The result is input to the second advanced unit for calculation, and the output p1 corresponding to the h1 position is obtained; h2 is added to p1 and then input to the third advanced unit to obtain the output p2 corresponding to the h2 position; each position is calculated in the same way until the final hM -1 is added to pM-2 to obtain the corresponding output pM-1, and the high-dimensional mapping relationship feature P={p0,...,pM-1} is obtained.

Optionally, the prediction unit is further configured to perform the following operations: input the mapping relationship P to the reconstructor, and obtain the The predicted target speech in the mixed speech samples.

Optionally, the determining unit includes: a calculation module, configured to calculate the proportional invariant SNR of the target sample speech and the predicted target language, and determine the proportional invariant signal to noise ratio according to the proportional invariant signal to noise ratio. Loss function; adjustment module, used to adjust the weight and bias of each parameter of the preset voice deep neural network model by gradient descent method according to the loss value of the loss function; determination module, used to adjust the weight and bias of each parameter according to the target sample The speech and the loss function determined by the predicted target speech satisfy a preset condition, and the preset speech deep neural network model is determined as the target speech deep neural network model.

According to the first aspect of the embodiments of the present application, there is provided a computer-readable storage medium, which is characterized in that a computer program is stored in the storage medium, wherein the computer program is set to execute the above-mentioned Advanced speech deep neural network training method.

According to the first aspect of the embodiments of the present application, there is provided an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to Execute the above-mentioned progressive-based voice deep neural network training method.

In an embodiment of the present invention, a mixed speech sample and target sample speech are obtained, wherein the mixed speech sample includes target speech and noise speech; the mixed speech sample is input into a preset speech deep neural network model to obtain a predicted target speech, wherein the preset The speech deep neural network model includes an advanced extractor, a reconstructor, and an encoder. The encoder is used to extract features from the mixed speech to obtain the first feature. The advanced extractor is used to calculate a high-dimensional map based on the first feature. Relational features, the reconstructor is used to obtain the predicted target voice in the mixed voice sample according to the high-dimensional mapping relationship features; the loss function determined according to the target sample voice and the predicted target voice meets the preset conditions, and determines the preset voice deep neural network model For the target speech deep neural network model, based on the training of the speech deep neural network model including advanced extractor, encoder and reconstructor in this scheme, it solves the problem of not being able to effectively separate the target speech from the mixed speech in the prior art Voice technical issues.

Description of drawings

The accompanying drawings described here are used to provide a further understanding of the present invention and constitute a part of the application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations to the present invention. In the attached picture:

Fig. 1 is a kind of optional hardware structure block diagram of the mobile terminal based on the speech deep neural network training method of advanced type according to the embodiment of the present invention;

Fig. 2 is a flow chart of an optional advanced-based speech deep neural network training method according to an embodiment of the present invention;

Fig. 3 is an overall structural diagram of an optional advanced speech extraction network according to an embodiment of the present invention;

Fig. 4 is a structure diagram of an optional encoder according to an embodiment of the present invention;

FIG. 5 is a structural diagram of an optional advanced unit according to an embodiment of the present invention;

FIG. 6 is a structural diagram of an optional advanced extractor according to an embodiment of the present invention;

Fig. 7 is a structure diagram of an optional reconstructor according to an embodiment of the present invention;

Fig. 8 is a diagram of an optional progressive-based speech deep neural network training device according to an embodiment of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is an embodiment of a part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first" and "second" in the description and claims of the present invention and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

In order to better understand this application, some names are described as follows:

The embodiment of the advanced speech-based deep neural network training method provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, FIG. 1 is a hardware structural block diagram of a mobile terminal based on an advanced speech deep neural network training method according to an embodiment of the present invention. As shown in FIG. 1, the mobile terminal 10 may include one or more (only one is shown in FIG. 1) processors 102 (the processors 102 may include but not limited to processing devices such as microprocessor MCU or programmable logic device FPGA, etc. ) and a memory 104 for storing data. Optionally, the above-mentioned mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. Those skilled in the art can understand that the structure shown in FIG. 1 is only for illustration, and it does not limit the structure of the above mobile terminal. For example, the mobile terminal 10 may also include more or fewer components than those shown in FIG. 1 , or have a different configuration than that shown in FIG. 1 .

The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the advanced voice deep neural network training method in the embodiment of the present invention, and the processor 102 stores the memory in the memory 104 by running The computer program in the computer, so as to execute various functional applications and data processing, that is, to realize the above-mentioned method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory that is remotely located relative to the processor 102, and these remote memories may be connected to the mobile terminal 10 through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission device 106 is used to receive or transmit data via a network. The specific example of the above-mentioned network may include a wireless network provided by the communication provider of the mobile terminal 10 . In one example, the transmission device 106 includes a network interface controller (NIC for short), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (Radio Frequency, RF for short) module, which is used to communicate with the Internet in a wireless manner.

In this embodiment, an advanced-based voice deep neural network training method is also provided. FIG. 2 is a flowchart of an advanced-based voice deep neural network training method according to an embodiment of the present invention, as shown in FIG. 2 Shown, this advanced voice deep neural network training method process based on the advanced formula includes the following steps:

Step S202, acquiring a mixed speech sample and a target sample speech, wherein the mixed speech sample includes target speech and noise speech.

Step S204, input the mixed speech sample into the preset speech deep neural network model to obtain the predicted target speech, wherein the preset speech deep neural network model includes an advanced extractor, a reconstructor and an encoder, and the encoder is used to analyze the mixed speech Perform feature extraction to obtain the first feature, the advanced extractor is used to calculate the high-dimensional mapping relationship feature based on the first feature, and the reconstructor is used to obtain the predicted target voice in the mixed voice sample according to the high-dimensional mapping relationship feature.

Step S206, the loss function determined according to the target sample speech and the predicted target speech satisfies the preset condition, and the preset speech deep neural network model is determined as the target speech deep neural network model.

In this embodiment, the purpose of the present invention is to propose an advanced single-channel speech separation algorithm for target speaker extraction for the problem of target speaker extraction in background noise, reverberation and other speaker interference backgrounds. Compared with other similar types, this algorithm can enhance the extraction features of the target speech in an advanced manner, thereby greatly improving the accuracy of speech separation in noise and reverberation scenes, reducing the distortion rate after speech extraction, and improving the accuracy of speech. intelligibility.

The above-mentioned noise may include but not limited to a dialogue between the target voice information of the target user and other objects, and may also include other sounds in the environment.

Through the embodiments provided by this application, a mixed voice sample and a target sample voice are obtained, wherein the mixed voice sample includes target voice and noise voice; the mixed voice sample is input into a preset voice deep neural network model to obtain a predicted target voice, wherein the preset Suppose the voice deep neural network model includes an advanced extractor, a reconstructor and an encoder. The encoder is used to extract features from the mixed speech to obtain the first feature. The advanced extractor is used to calculate the high-dimensional The mapping relationship feature, the reconstructor is used to obtain the predicted target voice in the mixed voice sample according to the high-dimensional mapping relationship feature; the loss function determined according to the target sample voice and the predicted target voice meets the preset conditions, and determines the preset voice deep neural network The model is the target speech deep neural network model. Based on the training of the speech deep neural network model including advanced extractor, encoder and reconstructor in this scheme, it solves the problem that the existing technology cannot effectively separate the speech from the mixed speech. Technical issues with the target voice.

Optionally, the encoder is used to perform feature extraction on the mixed speech to obtain the first feature, which may include: input the mixed speech sample into the preset speech deep neural network model, and activate the two-layer convolutional network and ReLU through the encoder. function and batch normalization to get the first feature.

Optionally, the advanced extractor is used to calculate the high-dimensional mapping relationship features according to the first feature, which may include: the advanced extractor includes a plurality of advanced units, and each advanced unit includes: a delay neural network , ReLU activation function, batch normalization processing, time-delay neural network, pooling layer, batch normalization processing, and graph convolution layer; each element in the first feature is input into the corresponding advanced unit , to obtain high-dimensional mapping relationship features.

Optionally, each element in the first feature is input into the corresponding advanced unit to obtain the high-dimensional mapping relationship feature, which may include: the first feature is expressed as H={h0,...,hi,...,hM- 1}, wherein, i=0 to M-1, the advanced unit includes M, that is, in the case of J={j0,...,ji,...,jM-1}; h0 is input to the first advanced unit, The corresponding output p0 is obtained; the result of adding h1 and p0 is input to the second advanced unit for calculation, and the output p1 corresponding to the position of h1 is obtained; after the addition of h2 and p1, it is input to the third advanced unit to obtain the output p2 corresponding to the position of h2; The calculation of each position is deduced by analogy until the final hM-1 is added to pM-2 to obtain the corresponding output pM-1, and the high-dimensional mapping relationship feature P={p0,...,pM-1} is obtained.

Optionally, the reconstructor is used to obtain the predicted target speech in the mixed speech sample according to the high-dimensional mapping relationship features, which may include: input the mapping relationship P to the reconstructor, and pass through two layers of convolutional network layers, ReLU activation function and After batch normalization processing, the predicted target speech in the mixed speech samples is obtained.

Optionally, the loss function determined according to the target sample speech and the predicted target speech satisfies the preset conditions, and the preset speech deep neural network model is determined as the target speech deep neural network model, which may include: calculating the target sample speech and the predicted target language Equal proportion constant signal to noise ratio, determine the loss function according to the equal proportion constant signal to noise ratio; according to the loss value of the loss function, adjust the weight and bias of each parameter of the preset voice deep neural network model through the gradient descent method; according to the target The loss function determined by the sample speech and the predicted target speech satisfies the preset condition, and the preset speech deep neural network model is determined as the target speech deep neural network model.

As an optional embodiment, the present application also provides an advanced speech extraction algorithm. It has the following contents.

As shown in Figure 3, the overall structure diagram of the advanced speech extraction network. In this embodiment, the advanced speech extraction algorithm includes an advanced extractor, an encoder and a reconstructor. As shown in Figure 4, the encoder structure diagram, the encoder is mainly composed of two convolutional layers (CNN) and one pooling layer (Pooling). As shown in Figure 5, the advanced unit structure diagram, each advanced unit can include: delayed neural network, ReLU activation function, batch normalization processing, delayed neural network, pooling layer, batch normalization processing , graph convolution layer. Among them, as shown in Figure 6, the advanced extractor structure diagram, the advanced extractor mainly consists of two layers of time-delayed neural network layers (TDNN), one layer of pooling layer and one layer of graph convolutional network layer (GCN) constitute. As shown in Figure 7, the structure diagram of the reconstructor, the reconstructor is mainly composed of two layers of deconvolutional network layers (DCNN). It mainly includes the following contents:

The first part: Preprocessing the mixed speech samples required for training and testing;

The second part: use the loss function to train the established advanced extraction deep neural network to obtain the advanced extraction deep neural network model;

The third part: Preprocess the speech samples to be tested, and perform speech separation through the advanced extraction deep neural network model after training, and obtain the separation results.

Each part will be described in detail below.

Among them, the first part specifically includes:

Step 1, re-sampling the time-domain signal of speech signal samples and noise samples at 8kHz, randomly mixing the speech of different speakers with a signal-to-noise ratio between 0 and 5dB, and combining them with randomly drawn noise samples at Mix under the signal-to-noise ratio of -6 to 3dB, and then calculate the reverberation of the space and the microphone under different conditions according to the room response function, and obtain the final mixed voice signal y;

Step 2, the entire database obtained in the above steps is divided into training set, verification set and test set. The mixed voice is used as the input of the deep neural network for advanced extraction, and a speaker's voice in the mixed voice is used as the training target of the network.

The second part specifically includes:

Step 1. Establish an advanced extraction deep neural network model, including an encoder, an advanced extractor, and a reconstructor. The encoder consists of two convolutional layers (CNN) and one pooling layer (Pooling), as shown in Figure 4. The advanced extractor consists of two layers of time-delayed neural network (TDNN) and one layer of pooling layer and one layer of graph convolutional network (GCN), as shown in Figure 6. The reconstructor consists of two deconvolutional network layers (DCNN), as shown in Fig. 7.

Step 2. Randomly initialize the parameters of the advanced extraction deep neural network, including initializing the weights and biases between network neuron nodes.

Step 3, the deep neural network performs forward propagation. In the forward propagation process, the activation function can be used to increase the nonlinear relationship between the networks, and finally a nonlinear mapping between the input and output results can be generated.

Step 4, perform supervised training on the deep neural network according to the parameters initialized in step 2 and the network training target in the first part. In this embodiment, the loss function is used to backpropagate and update the weights and biases through the gradient descent method. The loss function of the entire network is:

s为理想目标语音，

为估计的目标语音，<·,·>表示两个向量之间的点积，而||·||₂表示欧式距离。in,

s is the ideal target speech,

is the estimated target speech, <·,·> represents the dot product between two vectors, and ||·|| ₂ represents the Euclidean distance.

Step 5, update the parameters of the deep neural network through the gradient descent method.

a. Within a certain period of time, the parameters in the network are fixed, and the gradient of the loss function of the output layer is calculated;

b. Calculate the gradient corresponding to each layer when the number of network layers l=L-1, L-2, ..., 2;

c. Update the weights and biases of the entire network.

Step 6, the training is completed, and the deep neural network model is obtained according to the training results.

It should be noted that the encoder part: input the mixed audio y to the network input, and then perform preliminary feature extraction on the target speech through two-layer convolutional network, ReLU activation function and batch normalization (BN), and obtain H ={h0,...,hi,...,hM-1}, i=0 to M-1. M is the output length corresponding to the last network layer of this global extractor.

Advanced extractor part: This part is composed of multiple advanced units, and the specific calculation operation is shown in Figure 5. Each advanced unit is shown in Figure 5, including: delayed neural network, ReLU activation function, batch normalization processing, delayed neural network, pooling layer, batch normalization processing, and graph convolution layer. Input H to the input terminal of this module, where h0 directly enters the advanced unit to obtain the corresponding output p0, and the result of adding h1 and p0 enters the advanced unit for calculation, and obtains the output p1 corresponding to the position of h1, and adds h2 and p1 After entering the advanced unit, the output p2 corresponding to the h2 position is obtained, and the calculation of each subsequent position is deduced by analogy, until the final hM-1 and pM-2 are added to obtain the corresponding output pM-1. At this time, all the output results are taken as the high-dimensional extraction map P={p0, . . . , pM-1} corresponding to the target speech.

Reconstructor part: Input P to the input of this module, and get the estimated speech corresponding to each speaker after two layers of deconvolution network layer, ReLU activation function and batch normalization

The speech reconstruction operation in the third part is: input the speech sample to be tested in the first part into the advanced extraction and separation network model after training, and the speech separation result of the target speaker can be directly obtained through calculation.

Through the embodiments provided in this application, the single-channel speech separation algorithm for advanced extraction of the target speaker can solve the problems of difficult speech extraction of the target speaker and attenuation of the separation effect under the background of noise, reverberation and other speaker interference. Compared with other single-channel speech separation methods, it can effectively extract the useful information of the target speech in an advanced manner, improve the accuracy of the separated speech, reduce the distortion rate of the speech, and improve the intelligibility.

Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products are stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to enable a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

This embodiment also provides an advanced-based speech deep neural network training device, which is used to implement the above-mentioned embodiments and preferred implementation modes, and those that have already been described will not be repeated. As used below, the term "module" may be a combination of software and/or hardware that realizes a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.

Fig. 8 is a structural block diagram of an advanced speech deep neural network training device according to an embodiment of the present invention. As shown in Fig. 8, the advanced speech deep neural network training device includes:

The obtaining unit 81 is configured to obtain a mixed speech sample and a target sample speech, where the mixed speech sample includes target speech and noise speech.

The prediction unit 83 is used to input the mixed speech samples into the preset speech deep neural network model to obtain the predicted target speech, wherein the preset speech deep neural network model includes an advanced extractor, a reconstructor and an encoder, and the encoder is used for Feature extraction is performed on the mixed speech to obtain the first feature. The advanced extractor is used to calculate the high-dimensional mapping relationship feature based on the first feature. The reconstructor is used to obtain the prediction target in the mixed speech sample based on the high-dimensional mapping relationship feature. voice.

The determining unit 85 is configured to determine the preset voice deep neural network model as the target voice deep neural network model by determining the loss function determined according to the target sample voice and the predicted target voice to meet the preset conditions.

Through the embodiment provided by this application, the acquisition unit 81 acquires the mixed speech sample and the target sample speech, wherein the mixed speech sample includes target speech and noise speech; the prediction unit 83 inputs the mixed speech sample into the preset speech deep neural network model to obtain the prediction The target speech, wherein the preset speech deep neural network model includes an advanced extractor, a reconstructor and an encoder, the encoder is used to perform feature extraction on the mixed speech to obtain the first feature, and the advanced extractor is used to obtain the first feature according to the first features, the calculated high-dimensional mapping relationship features, the reconstructor is used to obtain the predicted target voice in the mixed voice sample according to the high-dimensional mapping relationship features; the loss function determined by the determination unit 85 according to the target sample voice and the predicted target voice satisfies the preset Conditions, determine the preset speech deep neural network model as the target speech deep neural network model, based on the training of the speech deep neural network model including advanced extractor, encoder and reconstructor in this program, solve the problem in the prior art, The technical problem of not being able to effectively separate the target speech from the mixed speech.

Optionally, the above prediction unit 83 may include: an encoding module, configured to input the mixed speech samples into the preset speech deep neural network model, through the two-layer convolutional network included in the encoder, the ReLU activation function and batch normalization processing to obtain the first feature.

Optionally, the above prediction unit 83 can also be used to perform the following operations: the advanced extractor includes multiple advanced units, and each advanced unit includes: a delayed neural network, a ReLU activation function, batch normalization processing In the case of , time-delay neural network, pooling layer, batch normalization processing, and graph convolution layer; each element in the first feature is input into the corresponding advanced unit to obtain high-dimensional mapping relationship features.

Optionally, the above prediction unit 83 can also be used to perform the following operations: the first feature is expressed as H={h0,...,hi,...,hM-1}, wherein, i=0 to M-1, advanced The units include M, that is, in the case of J={j0,...,ji,...,jM-1}; h0 is input to the first advanced unit, and the corresponding output p0 is obtained; the result of adding h1 and p0 is input to the first The second advanced unit calculates to obtain the output p1 corresponding to the h1 position; h2 and p1 are added and input to the third advanced unit to obtain the output p2 corresponding to the h2 position; each position is calculated in the same way until the final hM-1 and The pM-2 is added to obtain the corresponding output pM-1, and the high-dimensional mapping relation feature P={p0, . . . , pM-1} is obtained.

Optionally, the above-mentioned prediction unit 83 can also be used to perform the following operations: input the mapping relationship P to the reconstructor, and after two layers of convolutional network layers, ReLU activation function and batch normalization, obtain the mixed speech samples predicted target speech.

Optionally, the above-mentioned determination unit 85 may include: a calculation module for calculating the proportional invariant signal-to-noise ratio of the target sample speech and the predicted target language, and determining the loss function according to the proportional invariant signal-to-noise ratio; According to the loss value of the loss function, the weight and bias of each parameter of the preset voice deep neural network model are adjusted by the gradient descent method; the determination module is used to satisfy the preset loss function according to the target sample voice and the predicted target voice. condition, determine the preset voice deep neural network model as the target voice deep neural network model.

It should be noted that the above-mentioned modules can be realized by software or hardware. For the latter, it can be realized by the following methods, but not limited to this: the above-mentioned modules are all located in the same processor; or, the above-mentioned modules can be combined in any combination The forms of are located in different processors.

An embodiment of the present invention also provides a storage medium, in which a computer program is stored, wherein the computer program is set to execute the steps in any one of the above method embodiments when running.

Optionally, in this embodiment, the above-mentioned storage medium may be configured to store a computer program for performing the following steps:

S1. Acquire a mixed voice sample and a target sample voice, where the mixed voice sample includes target voice and noise voice;

S2, input the mixed speech sample into the preset speech deep neural network model to obtain the predicted target speech, wherein the preset speech deep neural network model includes an advanced extractor, a reconstructor and an encoder, and the encoder is used to perform mixed speech processing feature extraction to obtain the first feature, the advanced extractor is used to calculate the high-dimensional mapping relationship feature according to the first feature, and the reconstructor is used to obtain the predicted target voice in the mixed voice sample according to the high-dimensional mapping relationship feature;

S3. The loss function determined according to the target sample speech and the predicted target speech satisfies a preset condition, and the preset speech deep neural network model is determined as the target speech deep neural network model.

Optionally, in this embodiment, the above-mentioned storage medium may include but not limited to: U disk, read-only memory (Read-Only Memory, ROM for short), random access memory (Random Access Memory, RAM for short), Various media that can store computer programs, such as removable hard disks, magnetic disks, or optical disks.

An embodiment of the present invention also provides an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments.

Optionally, the above-mentioned electronic device may further include a transmission device and an input-output device, wherein the transmission device is connected to the above-mentioned processor, and the input-output device is connected to the above-mentioned processor.

Optionally, in this embodiment, the above-mentioned processor may be configured to execute the following steps through a computer program:

S1. Acquire a mixed voice sample and a target sample voice, where the mixed voice sample includes target voice and noise voice;

S2, input the mixed speech sample into the preset speech deep neural network model to obtain the predicted target speech, wherein the preset speech deep neural network model includes an advanced extractor, a reconstructor and an encoder, and the encoder is used to perform mixed speech processing Feature extraction to obtain the first feature, the advanced extractor is used to calculate the high-dimensional mapping relationship feature according to the first feature, and the reconstructor is used to obtain the predicted target voice in the mixed voice sample according to the high-dimensional mapping relationship feature;

S3. The loss function determined according to the target sample speech and the predicted target speech satisfies a preset condition, and the preset speech deep neural network model is determined as the target speech deep neural network model.

Optionally, for specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiments and optional implementation manners, and details are not repeated in this embodiment.

Obviously, those skilled in the art should understand that each module or each step of the above-mentioned present invention can be realized by a general-purpose computing device, and they can be concentrated on a single computing device, or distributed in a network formed by multiple computing devices Alternatively, they may be implemented in program code executable by a computing device so that they may be stored in a storage device to be executed by a computing device, and in some cases in an order different from that shown here The steps shown or described are carried out, or they are separately fabricated into individual integrated circuit modules, or multiple modules or steps among them are fabricated into a single integrated circuit module for implementation. As such, the present invention is not limited to any specific combination of hardware and software.

The above are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims (10)

1. A method for training a speech deep neural network based on an advanced form is characterized by comprising the following steps:

acquiring a mixed voice sample and target sample voice, wherein the mixed voice sample comprises the target voice and noise voice;

inputting the mixed voice sample into a preset voice deep neural network model to obtain a predicted target voice, wherein the preset voice deep neural network model comprises a step extractor, a reconstructor and a coder, the coder is used for carrying out feature extraction on the mixed voice to obtain a first feature, the step extractor is used for calculating to obtain a high-dimensional mapping relation feature according to the first feature, and the reconstructor is used for obtaining the predicted target voice in the mixed voice sample according to the high-dimensional mapping relation feature;

determining the preset speech deep neural network model as a target speech deep neural network model according to the loss function determined by the target sample speech and the predicted target speech and meeting a preset condition;

the encoder is configured to perform feature extraction on the mixed speech to obtain a first feature, and includes:

and inputting the mixed voice sample into the preset voice deep neural network model, and obtaining the first characteristic through a two-layer convolution network, a ReLU activation function and batch normalization processing which are included by the encoder.

2. The method of claim 1, wherein the step extractor is configured to compute a high-dimensional mapping relationship feature according to the first feature, and comprises:

the step extractor comprises a plurality of step units, each step unit comprises: a time-delay neural network, a ReLU activation function, batch normalization processing, a time-delay neural network, a pooling layer, batch normalization processing and a graph convolution layer;

and inputting each element in the first characteristic into a corresponding advanced unit respectively to obtain the high-dimensional mapping relation characteristic.

3. The method according to claim 2, wherein the inputting each element in the first feature into a corresponding order unit to obtain the high-dimensional mapping relationship feature comprises:

in case the first feature is denoted as H = { H0, …, hi, …, hM-1}, where i =0 to M-1, the advanced cell comprises M, i.e., J = { J0, …, ji, …, jM-1 };

h0 is input into the first step unit to obtain a corresponding output p0;

the result of the addition of h1 and p0 is input into a second advanced unit for calculation to obtain an output p1 corresponding to the position of h 1;

adding h2 and p1, and inputting the sum to a third step unit to obtain an output p2 corresponding to the position of h 2;

and performing calculation in each position by analogy until the final hM-1 and pM-2 are added to obtain a corresponding output pM-1, and obtaining a high-dimensional mapping relation characteristic P = { P0, …, pM-1}.

4. The method according to claim 3, wherein the reconstructor is configured to obtain the predicted target speech in the mixed speech sample according to the high-dimensional mapping relation feature, and comprises:

and inputting the mapping relation P into the reconstructor, and obtaining the predicted target voice in the mixed voice sample after two layers of convolutional network layers, reLU activation functions and batch normalization processing.

5. The method according to claim 1, wherein the determining the predetermined deep neural network model as the target deep neural network model according to the loss function determined by the target sample speech and the predicted target speech satisfying a predetermined condition comprises:

calculating an equal-proportion invariant signal-to-noise ratio of the target sample voice and the predicted target voice, and determining the loss function according to the equal-proportion invariant signal-to-noise ratio;

according to the loss value of the loss function, adjusting the weight and the bias of each parameter of the preset speech deep neural network model by a gradient descent method;

and determining the preset speech deep neural network model as a target speech deep neural network model according to the loss function determined by the target sample speech and the predicted target speech and meeting a preset condition.

6. A speech deep neural network training device based on an advance formula is characterized by comprising:

the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a mixed voice sample and target sample voice, and the mixed voice sample comprises the target voice and noise voice;

the prediction unit is used for inputting the mixed voice sample into a preset voice deep neural network model to obtain a predicted target voice, wherein the preset voice deep neural network model comprises a step extractor, a reconstructor and a coder, the coder is used for extracting the characteristics of the mixed voice to obtain a first characteristic, the step extractor is used for calculating to obtain a high-dimensional mapping relation characteristic according to the first characteristic, and the reconstructor is used for obtaining the predicted target voice in the mixed voice sample according to the high-dimensional mapping relation characteristic;

the determining unit is used for determining the preset speech deep neural network model as a target speech deep neural network model according to the loss function determined by the target sample speech and the predicted target speech and meeting a preset condition;

the prediction unit includes:

and the coding module is used for inputting the mixed voice sample into the preset voice deep neural network model, and obtaining the first characteristic through two layers of convolutional networks, a ReLU activation function and batch normalization processing which are included by the coder.

7. The apparatus of claim 6, wherein the prediction unit is further configured to:

the step extractor comprises a plurality of step units, each step unit comprises: a time-delay neural network, a ReLU activation function, batch normalization processing, a time-delay neural network, a pooling layer, batch normalization processing and a graph convolution layer;

and inputting each element in the first characteristic into a corresponding advanced unit respectively to obtain the high-dimensional mapping relation characteristic.

8. The apparatus of claim 7, wherein the prediction unit is further configured to:

in case the first feature is denoted as H = { H0, …, hi, …, hM-1}, where i =0 to M-1, the advanced cell comprises M, i.e., J = { J0, …, ji, …, jM-1 };

h0 is input into the first step unit to obtain a corresponding output p0;

the result of the addition of h1 and p0 is input into a second advanced unit for calculation to obtain an output p1 corresponding to the position of h 1;

adding h2 and p1, and inputting the sum to a third step unit to obtain an output p2 corresponding to the position of h 2;

and performing calculation in each position by analogy until the final hM-1 and pM-2 are added to obtain a corresponding output pM-1, and obtaining a high-dimensional mapping relation characteristic P = { P0, …, pM-1}.

9. The apparatus of claim 8, wherein the prediction unit is further configured to:

and inputting the mapping relation P into the reconstructor, and obtaining the predicted target voice in the mixed voice sample after two layers of convolutional network layers, reLU activation functions and batch normalization processing.

10. The apparatus of claim 6, wherein the determining unit comprises:

the calculation module is used for calculating the equal-proportion invariable signal-to-noise ratio of the target sample voice and the predicted target voice and determining the loss function according to the equal-proportion invariable signal-to-noise ratio;

the adjusting module is used for adjusting the weight and the bias of each parameter of the preset speech depth neural network model through a gradient descent method according to the loss value of the loss function;

and the determining module is used for determining the preset voice deep neural network model as a target voice deep neural network model according to the condition that the loss function determined by the target sample voice and the predicted target voice meets a preset condition.

Images