KR102275656B1 - Method and apparatus for robust speech enhancement training using adversarial training - Google Patents

Abstract

A robust voice enhancement training method using an adversarial training model is disclosed. The voice enhancement training method includes the steps of extracting a feature vector from a noisy voice, extracting a latent variable by using the extracted feature vector as an input of a first artificial neural network, and using the extracted latent variable as an input of a second artificial neural network. A first operation of outputting the estimated speech feature vector and a second operation of outputting the estimated noise feature vector by using the extracted latent variable as an input to the third artificial neural network are performed, and the first operation and the second operation are antagonistic to each other It includes the steps of being trained by learning, outputting a latent variable from which a noise component is removed by learning, and generating a reconstructed speech based on the output latent variable.

Description

Robust voice enhancement training method and device using adversarial training model {METHOD AND APPARATUS FOR ROBUST SPEECH ENHANCEMENT TRAINING USING ADVERSARIAL TRAINING}

The present invention relates to a robust voice enhancement training method using an adversarial training model and an apparatus therefor, and more particularly, a robust voice enhancement training method capable of improving voice quality by removing noise from a noisy voice. and devices.

Speech enhancement technology is a technology for estimating a clear voice from a noisy voice. It is an important technology that can be used in various voice-related applications, such as helping to improve the intelligibility of voice in the field of voice communication and using it as a pre-processing technology in voice recognition, etc. to be.

In early studies, statistical methods were used a lot to estimate noise in the non-voice section (the section with only noise) and to remove the noise based on the information. However, this technique tends to deteriorate in performance in an environment in which noise varies over time (non-stationary) or in a heavily mixed environment (low signal to ratio (SNR)).

Recently, due to the development of deep learning, various deep learning techniques are being applied in the field of speech enhancement technology. In deep learning-based speech enhancement, a model is trained by taking a noisy voice as an input and targeting a clean voice before noise, which is typical of regression model learning.

In other words, the existing deep learning voice enhancement technique takes a noisy voice as an input and focuses only on the design of a model that estimates a clean voice matching it, and how the input works in the hidden layer in the middle of the deep learning model. No research has been conducted on whether it is learned.

An object of the present invention is to provide a voice enhancement training method using an adversatial training model and an apparatus therefor to effectively perform voice enhancement.

A robust voice enhancement training method using an adversarial learning model according to an embodiment of the present invention includes extracting a feature vector from a noise mixed voice, and extracting a latent variable by using the extracted feature vector as an input to the first artificial neural network. a first operation of outputting an estimated speech feature vector by using the extracted latent variable as an input of a second artificial neural network, and outputting an estimated noise feature vector by using the extracted latent variable as an input of a third artificial neural network performing a second operation, the first operation and the second operation are trained by adversarial learning, outputting a latent variable from which a noise component is removed by the learning, and based on the output latent variable and generating a restored restored voice.

In this case, the adversarial learning is performed by inverting the sign of the gradient through the gradient reversal layer during back-propagation in the third artificial neural network, and the extracted The latent variable can be trained to remove the noise characteristic.

Also, in the first operation, an original sound may be estimated by decoding the extracted latent variable, and a magnitude spectrum of the estimated original sound may be output from the speech feature vector.

A voice enhancement training apparatus using an adversarial training model according to an embodiment of the present invention includes a feature vector extractor for extracting a feature vector from a noise-mixed voice, and input the extracted feature vector to the first artificial neural network. An encoder that extracts a latent variable as an input, a decoder that outputs a speech feature vector estimated by using the extracted latent variable as an input of a second artificial neural network, and a noise estimated by using the extracted latent variable as an input of a third artificial neural network. and a noise latent variable removal unit for outputting a feature vector and a voice restoration unit for generating a restored voice, wherein the decoder and the noise latent variable removal unit perform adversarial learning to output a latent variable from which a noise component is removed, and the voice The restoration unit may generate the restored voice based on the output latent variable.

In this case, the adversarial learning is performed by inverting the sign of the gradient through the gradient reversal layer during back-propagation in the third artificial neural network to obtain noise characteristics in the extracted latent variable. can learn to be removed.

Also, in the first operation, an original sound may be estimated by decoding the extracted latent variable, and a magnitude spectrum of the estimated original sound may be output from the speech feature vector.

According to an embodiment of the present invention, as a computer-readable recording medium storing a computer program, when the computer program is executed by a processor, extracting a feature vector from a noisy voice, the extracted feature vector extracting a latent variable using as an input of a first artificial neural network, a first operation of outputting an estimated speech feature vector using the extracted latent variable as an input of a second artificial neural network, and a third operation of the extracted latent variable A second operation of outputting an estimated noise feature vector as an input to the artificial neural network is performed, and the first operation and the second operation are trained by adversarial learning; a latent variable from which the noise component is removed by the learning. It may include an instruction for causing the processor to perform a voice enhancement training method comprising the steps of outputting and generating a reconstructed reconstructed speech based on the output latent variable.

According to an embodiment of the present invention, as a computer program stored in a computer-readable recording medium, when the computer program is executed by a processor, extracting a feature vector from a noisy voice, the extracted feature vector extracting a latent variable using as an input of a first artificial neural network, a first operation of outputting an estimated speech feature vector using the extracted latent variable as an input of a second artificial neural network, and a third operation of the extracted latent variable A second operation of outputting an estimated noise feature vector as an input to the artificial neural network is performed, and the first operation and the second operation are trained by adversarial learning; a latent variable from which the noise component is removed by the learning. It may include an instruction for causing the processor to perform a voice enhancement training method including outputting , and generating a reconstructed reconstructed voice based on the output latent variable.

By directly accessing the latent variables of the noisy speech in the middle hidden layer of the deep learning model, which has not been presented in the existing speech enhancement models, the noise features can be removed and only the features of the original sound can be left, so the speech enhancement performance can be significantly improved. .

1 is a flowchart for briefly explaining the process of a voice enhancement training method according to an embodiment of the present invention;
2 is a block diagram schematically illustrating the configuration of a voice enhancement training apparatus according to an embodiment of the present invention.

First, the terms used in the present specification and claims have been selected in consideration of functions in various embodiments of the present invention. However, these terms may vary depending on the intention, legal or technical interpretation of a person skilled in the art, and the emergence of new technology. Also, some terms may be arbitrarily selected by the applicant. These terms may be interpreted in the meaning defined in the present specification, and if there is no specific term definition, it may be interpreted based on the general content of the present specification and common technical common sense in the art.

Also, the same reference numerals or reference numerals in each drawing appended to this specification indicate parts or components that perform substantially the same functions. For convenience of description and understanding, the same reference numerals or reference numerals are used in different embodiments. That is, even though all the components having the same reference number are shown in the plurality of drawings, the plurality of drawings do not mean one embodiment.

In addition, in this specification and claims, terms including ordinal numbers such as 'first' and 'second' may be used to distinguish between elements. This ordinal number is used to distinguish the same or similar components from each other, and the meaning of the term should not be limitedly interpreted due to the use of the ordinal number. As an example, the components combined with such an ordinal number should not be construed as limiting the order of use or arrangement by the number. If necessary, each ordinal number may be used interchangeably.

In this specification, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as 'comprise' or 'comprise' are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification is present, and one or more other It should be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In addition, in an embodiment of the present invention, when it is said that a part is connected to another part, this includes not only a direct connection but also an indirect connection through another medium. In addition, the meaning that a certain component includes a certain component does not exclude other components unless otherwise stated, but may further include other components.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings.

1 is a flowchart for briefly explaining a process of a voice enhancement training method according to an embodiment of the present invention.

In the robust voice enhancement training method using the adversarial learning model, referring to FIG. 1 , first, a feature vector is extracted from a noisy voice (hereinafter, noise voice) (S110).

Specifically, a magnitude spectrum, which is a speech feature vector, may be obtained by dividing the input noise speech into frame signals in units of time, and converting each divided frame signal into a signal of a frequency domain.

For example, the noisy speech may be divided into frame signals in units of 16 to 32 ms.

For example, the acquired magnitude spectrum itself may be a feature vector, and may include a pitch-related feature vector, a tone-related feature vector, and the like derived from this.

The feature vector can be mainly calculated using short-term Fourier analysis (STFT), etc. The feature vector is a mel frequency cepstral coefficient (MFCC), a spectral rolloff, a spectral flux, and an autocorrelation coefficient. (Autocorrelation coefficient) and the like.

MFCC is a coefficient obtained by using DCT after dividing an audio signal converted into the cepstrum domain into sub-bands using a mel-frequency filter bank reflecting auditory characteristics. used

Spectral rolloff is calculated by calculating the value of the frequency at which 85% of energy is distributed from the low-band signal in the frequency domain. It is a feature that can identify the distribution of the frequency domain together with the spectral centroid. It is a vector.

The spectral flux expresses the degree of change in the time axis for each frequency unit, and is a parameter that measures the change in local frequency.

The autocorrelation coefficient expresses the spectral distribution of a signal in the time domain, and for example, a first coefficient to a twelfth coefficient may be used.

Thereafter, a latent feature is extracted by using the extracted feature vector as an input of the encoder artificial neural network (first artificial neural network) (S120). A latent variable is a hidden variable that constitutes a multi-layered artificial neural network of an adversarial learning model.

Thereafter, a first operation of outputting an estimated speech feature vector by using the extracted latent variable as an input of a decoder artificial neural network (second artificial neural network) for estimating speech is performed (S130). Here, by the first operation, the artificial neural network of the decoder receiving the latent variable outputs the estimated original sound speech feature vector.

At the same time, a second operation of outputting an estimated noise feature vector by using the extracted latent variable as an input of the noise estimation artificial neural network (the third artificial neural network) is performed (S140). Here, learning proceeds in the direction of removing noise components from latent variables using a gradient reversal layer (GRL).

In this case, the first operation and the second operation are trained by learning based on an antagonistic learning model (S150).

The adversarial learning model is one of the representative unsupervised learning models. It is an artificial neural network model with a structure in which a generative model that tries to generate data identical to the actual data as much as possible and a discriminant model that tries to discriminate imitation data learn antagonistically. it means.

In the adversarial learning model, the discriminant model is first trained and then the process of learning the generative model is repeated. A typical example is a generative adversarial network (GAN).

These generative adversarial neural networks (GANs) often take the direction that counterfeiters (corresponding to 'generative models') deceive counterfeiters (corresponding to 'discrimination models'), and counterfeiters (corresponding to 'discriminatory models') to pneumatize and counterfeit banknotes forged by counterfeiters. It is compared to taking the direction of distinction.

By a generative adversarial neural network (GAN), the voice enhancement training model may be continuously updated to be more accurate and its performance may be improved.

In the present invention, the adversarial learning model means that when training an artificial neural network, training of two models is adversarial to one side in the direction of maximizing the objective function and the other side in the direction of minimizing the objective function. In other words, by inputting a latent variable with the characteristics of speech and noise as input, the decoder artificial neural network is trained in a direction that is as close to the target voice as possible, and the noise estimation artificial neural network is trained to be as similar to the original noise as possible, but GRL By changing the sign during error propagation through , learning proceeds to avoid estimating the noise as much as possible.

In the first and second operations, adversarial learning is a latent variable extracted by inverting the sign of the gradient through the gradient reversal layer during back-propagation in the second artificial neural network. learn to remove the noise component.

That is, through a structure that allows the latent variable output through the encoder to pass through the gradient inversion layer, it is learned in a direction that makes it difficult to distinguish between noises in the backpropagation process, and the noise characteristic is removed.

Therefore, the first artificial neural network learns in a direction to well estimate the original sound, which is a clean voice, and in the second artificial neural network, learns in a direction that does not estimate the noise well, and the two kinds of learning are performed antagonistically.

Accordingly, the encoder is trained to extract the latent variable from which the noise component is removed and only the voice component remains.

In this case, the first operation may decode the extracted latent variable to estimate the noise-removed speech, and output the magnitude spectrum of the estimated speech as a speech feature vector.

After that, the learning is completed and in the actual application step, the latent variable output from the encoder artificial neural network is output with the noise component removed and the voice component remaining ( S160 ). In this case, a speech magnitude spectrum may be output using the decoder artificial neural network.

Thereafter, the reconstructed speech is generated by reconstructing the output estimated speech magnitude spectrum as a speech signal in the time domain ( S170 ).

2 is a block diagram schematically illustrating the configuration of a voice enhancement training apparatus according to an embodiment of the present invention.

The voice enhancement training apparatus 100 according to an embodiment of the present invention includes a feature vector extractor 110 , an encoder 120 , a decoder 130 , a noise latent variable remover 140 , and a voice and a restoration unit 150 .

The voice enhancement training apparatus 100 is a kind of computing device and may include a processor (not shown) capable of processing and processing data. The processor includes a micro processing unit (MPU), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU) or a tensor processing unit (TPU), a cache memory, and a data bus. It may include a hardware configuration such as In addition, it may further include an operating system, a software configuration of an application for performing a specific purpose.

The feature vector extractor 110 is configured to use a noise-mixed voice (hereinafter, noise voice) as an input of a deep learning model.

Specifically, the feature vector extractor 110 divides the noise speech input to the speech enhancement training apparatus 100 into frame signals in units of time, and converts each divided frame signal into a frequency domain signal to obtain a speech feature vector. A magnitude spectrum can be generated.

The encoder 120 is configured to implement an encoder artificial neural network (a first artificial neural network) that receives the generated magnitude spectrum from the feature vector extractor 110 and outputs a latent variable. The encoder 120 is also referred to as a recognition network, and in simple terms, serves to transform an input into an internal representation.

The decoder 130 receives the latent variable output from the encoder 120, decodes the input latent variable, and estimates the noise-free speech. The decoder 130 may implement a decoder artificial neural network (second artificial neural network) capable of outputting the magnitude spectrum of the estimated original sound.

The decoder 130 is also referred to as a generative network, and in simple terms, serves to convert an internal representation into an output. The decoder 130 may calculate a latent variable for which frame to pay attention to, and estimate a speech feature vector according to the attention level.

The noise latent variable removal unit 140 is configured to receive the latent variable output from the encoder 120 and implement a noise estimation artificial neural network (third artificial neural network) that outputs the magnitude spectrum of the estimated noise.

Here, the decoder 130 and the noise latent variable removal unit 140 may train the artificial neural network through adversarial learning.

Specifically, the decoder 130 tries to estimate a clean voice, that is, an original sound from the input latent variable, and the noise latent variable remover 140 tries to estimate the noise from the input latent variable.

That is, as a generative model, the decoder 130 is trained to make a voice that is as similar to the original sound as possible, and the noise latent variable remover 140 is learned not to distinguish characteristics between input noises as much as possible, while learning to be hostile to each other. do.

In this case, the noise estimation artificial neural network of the noise latent variable removal unit 140 includes a gradient reversal layer, and reversely converts the sign of the gradient in the back-propagation process through the gradient reversal layer. will do

That is, the noise estimation artificial neural network of the noise latent variable removal unit 140 is trained in a direction that does not estimate the noise component well, and as a result, the input latent variable does not have a noise component.

Through such adversarial learning, the encoder 120 learns and outputs a latent variable having a characteristic in which noise components other than the original sound component are removed.

The latent variable learned through this process and output from the encoder 120 outputs a magnitude spectrum in which only the characteristics of the original sound are preserved through the decoder 130 .

After the hostile learning training is finished, in the actual estimation step, the operation of the noise latent variable removal unit 140 is omitted, and voice enhancement can be performed only by the operations of the encoder 120 and the decoder 130 .

That is, after learning, only generative models among adversarial learning models are used.

The voice restoration unit 150 may generate the restored voice by restoring the magnitude spectrum of the voice feature vector output through the decoder 130 back to the time domain voice signal.

According to the above-described various embodiments, by directly accessing the latent variable of the noisy speech in the middle hidden layer of the deep learning model, which was not presented in the existing method, the noise feature is removed and only the speech feature is left, so that the decoder is better than the existing model. By more effectively removing noise components, voice enhancement performance can be improved.

The control method according to the above-described various embodiments may be implemented as a program and stored in various recording media. That is, a computer program that is processed by various processors to execute the above-described noise removal method may be used in a state stored in the recording medium.

As an example, i) extracting a feature vector from a voice mixed with noise, ii) extracting a latent variable by using the extracted feature vector as an input of a first artificial neural network (eg, an encoder artificial neural network), iii) the extracted A first operation of outputting an estimated speech feature vector by taking the latent variable as an input of a second artificial neural network (eg, a decoder artificial neural network) and the extracted latent variable as an input of a third artificial neural network (eg, a noise estimation artificial neural network) a second operation of outputting the estimated noise feature vector is performed, the first operation and the second operation are trained by adversarial learning, and iv) outputting a latent variable from which the noise component is removed by learning. A non-transitory computer readable medium in which a program for performing steps and v) generating a reconstructed speech based on the output latent variable is stored may be provided.

The non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specifically, the above-described various applications or programs may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

On the other hand, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention as claimed in the claims Various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

100: speech enhancement training apparatus 110: feature vector extraction unit
120: encoder 130: decoder
140: noise latent variable removal unit 150: voice restoration unit

Claims (8)

In a robust voice enhancement training method using an adversarial training model,
extracting a feature vector from the noisy speech;
extracting a first latent variable having a feature in which noise components excluding original sound components are removed by using the extracted feature vector as an input to a first artificial neural network;
A first operation of outputting an estimated original speech feature vector by using the extracted first latent variable as an input of a second artificial neural network, and using the extracted first latent variable as an input of a third artificial neural network, the first latent variable performing a second operation of outputting a noise feature vector from which a noise component has been removed, and the second artificial neural network and the third artificial neural network are trained by adversarial learning;
receiving, by the second artificial neural network, the learning completed, the first latent variable, and outputting a second latent variable in which a noise component is removed from the first latent variable; and
generating a reconstructed speech based on the output second latent variable; A voice enhancement training method comprising a. According to claim 1,
The adversarial learning is
During back-propagation in the third artificial neural network, the sign of the gradient is reversed through the gradient reversal layer included in the adversarial learning model, and the extracted first latent variable is A voice enhancement training method, characterized in that learning to remove noise characteristics. According to claim 1,
The first operation is
A speech enhancement training method, comprising: estimating an original sound by decoding the extracted first latent variable, and outputting a magnitude spectrum of the estimated original sound from the original sound feature vector. In the voice enhancement training apparatus using an adversarial training model,
a feature vector extractor for extracting a feature vector from a noisy voice;
an encoder that uses the extracted feature vector as an input to a first artificial neural network and extracts a first latent variable having a feature in which noise components other than the original sound component are removed;
a decoder for outputting an estimated original speech feature vector using the extracted first latent variable as an input to a second artificial neural network;
a noise latent variable removal unit that uses the extracted first latent variable as an input to a third artificial neural network and outputs a noise feature vector from which a noise component is removed from the first latent variable; and
a voice restoration unit for generating a restored voice; including,
The second artificial neural network and the third artificial neural network are
adversarial learning,
The decoder is
The second artificial neural network on which the learning is completed receives the first latent variable and outputs a second latent variable from which the noise component is removed from the first latent variable,
The voice restoration unit,
Voice enhancement training apparatus, characterized in that generating the reconstructed speech based on the output second latent variable. 5. The method of claim 4,
The adversarial learning is
During back-propagation in the third artificial neural network, the sign of the gradient is reversed through the gradient reversal layer included in the adversarial learning model, and the extracted first latent variable is A voice enhancement training apparatus, characterized in that it learns to remove the noise characteristic. 6. The method of claim 5,
The decoder is
The apparatus for estimating an original sound by decoding the extracted first latent variable, and outputting a magnitude spectrum of the estimated original sound from the original sound feature vector. As a computer-readable recording medium storing a computer program,
The computer program, when executed by a processor,
extracting a feature vector from the noisy speech;
extracting a first latent variable having a feature in which noise components excluding original sound components are removed by using the extracted feature vector as an input to a first artificial neural network;
A first operation of outputting an estimated original speech feature vector by using the extracted first latent variable as an input of a second artificial neural network, and using the extracted first latent variable as an input of a third artificial neural network, the first latent variable performing a second operation of outputting a noise feature vector from which the noise component is removed, and the second artificial neural network and the third artificial neural network are trained by adversarial learning;
receiving, by the second artificial neural network, the learning completed, the first latent variable, and outputting a second latent variable in which a noise component is removed from the first latent variable; and
generating a reconstructed speech based on the output second latent variable; A computer-readable recording medium comprising instructions for causing the processor to perform a voice enhancement training method comprising a. As a computer program stored in a computer-readable recording medium,
The computer program, when executed by a processor,
extracting a feature vector from the noisy speech;
extracting a first latent variable having a feature in which noise components excluding original sound components are removed by using the extracted feature vector as an input to a first artificial neural network;
A first operation of outputting an estimated original speech feature vector by using the extracted first latent variable as an input of a second artificial neural network, and using the extracted first latent variable as an input of a third artificial neural network, the first latent variable performing a second operation of outputting a noise feature vector from which the noise component is removed, and the second artificial neural network and the third artificial neural network are trained by adversarial learning;
receiving, by the second artificial neural network, the learning completed, the first latent variable, and outputting a second latent variable in which a noise component is removed from the first latent variable; and
Generating a reconstructed reconstructed speech based on the output second latent variable; A computer program comprising instructions for causing the processor to perform a speech enhancement training method comprising a.

Images