KR102410850B1 - Method and apparatus for extracting reverberant environment embedding using dereverberation autoencoder - Google Patents

Abstract

A method and apparatus for extracting reverberation environment embeddings using a reverberation cancellation auto-encoder are presented. A method for extracting a reverberation environment embedding using a reverberation removal auto-encoder according to an embodiment includes: generating a reverberation speech for training from a speech database (DB) and an indoor impulse response (RIR) database (DB); performing multi-linear prediction on the generated reverberation speech and outputting a multi-linear prediction result; estimating a weight for a weighted sum of the multiple linear prediction results; estimating a final reverberation cancellation voice from the multiple linear prediction results and the estimated weight; and extracting a reverberation environment embedding vector from a weight estimation model for estimating the weight.

Description

Method and apparatus for extracting reverberation environment embedding using reverberation cancellation auto encoder

The following embodiments relate to a method and apparatus for extracting reverberation environment embeddings using a reverberation removal auto-encoder, and more particularly, a method for extracting reverberation environment embeddings using a reverberation removal auto-encoder to increase recognition performance for a voice uttered in a reverberation environment. and devices.

In the process of transmitting sound signals including voices, reverberation occurs, which is absorbed and reflected by various objects, such as walls, ceilings, and floors, as well as in a straight path. At this time, the generated reverberation component reduces the intelligibility of the voice, and the automatic speech recognition (ASR) system also has a phenomenon that the recognition performance of the voice with strong reverberation is deteriorated. Methods to improve speech recognition performance for reverberated speech can be largely divided into a method that directly removes the reverberation effect from speech and a model adaptation method that trains an automatic speech recognition (ASR) model to recognize the speech itself with reverberation well. have.

The reverberation cancellation method is a method of improving intelligibility or speech recognition performance by removing a reverberation effect from an input voice. Linear prediction-based reverberation cancellation is a representative method, and the reverberation effect is removed through linear combination of consecutive values on the time axis. It is based on the fact that reverberant speech can be modeled as a convolution of speech and Room Impulse Response (RIR). This method is known to have excellent performance when applied to each frequency axis of a local Short-Time Fourier Transform (STFT) domain spectrogram, especially when applied to multi-channel voices collected through a microphone array. have. However, there is a disadvantage in that the coefficients of the linear prediction filter for each target utterance must be newly predicted each time through an iterative algorithm.

The reverberation cancellation autoencoder is a kind of deep neural network (DNN), and is trained to output a voice from which the reverberation is removed when a reverberated voice is input (Non-Patent Document 1). A pair of reverberated speech and reverberated speech is required as training data, but it is not easy to secure a large-scale data pair of a real environment. Therefore, reverberation speech is artificially generated through convolution of recorded speech and indoor impulse response (RIR) in a quiet environment, and is often paired with the original speech and used as training data. Unlike the linear prediction-based method, since the model is pre-trained with various reverberation environment data, there is an advantage in that there is no need to additionally predict parameters for the target voice in the reverberation removal step. However, in the case of a DNN regression model trained through the L1 and L2 loss functions, the overall trend is well represented, but information on detailed elements is not saved. This phenomenon distorts the spectrogram and in some cases makes speech recognition performance worse even though the reverberation effect is removed.

The most widely known model adaptation method for reverberant speech recognition is multi-condition training (MCT) through data augmentation. Training an acoustic model through voices recorded in various environments improves voice recognition performance for more diverse environments, but in practice, it is not easy to sufficiently secure voices recorded in various environments. Instead, as with the training data generation of the reverberation cancellation auto-encoder, the reverberation environment speech recognition performance is improved by generating training data in various environments through a data augmentation method that artificially generates reverberated speech through clean speech and room impulse response (RIR). can do.

As another model adaptation method, there is a method of using an auxiliary feature R-vector containing reverberation information (Non-Patent Document 2). This method is derived from a speaker adaptation method that improves speech recognition performance by using an x-vector that extracts speaker information from a utterance as an auxiliary feature. The x-vector is extracted from the hidden layer of the trained DNN model through the process of classifying from which speaker the input speech was uttered. Similarly, R-vector extracts information about the reverberation environment with a trained model through the process of classifying which room impulse response (RIR) the reverberated speech generated through convolution is generated. If the R-vector obtained in this way is used as an auxiliary feature of the existing acoustic model, the reverberation speech recognition performance can be improved.

X. Feng, Y. Zhang and J. Glass, “Speech feature denoising and dereverberation via deep autoencoders for noisy reverberant speech recognition,” 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, 2014, pp. 1759-1763, doi: 10.1109/ICASSP.2014.6853900. Y. Khokhlov, A. Zatvornitskiy, I. Medennikov et al., "R-vectors: New technique for adaptation to room acoustics," INTERSPEECH, 2019.

The embodiments describe a method and apparatus for extracting a reverberation environment embedding using a reverberation removal auto-encoder, and more specifically, extracting a reverberation environment embedding using a reverberation removal auto-encoder, and a reverberation environment through a model adaptation method with auxiliary features. It provides a technology to improve speech recognition performance in

A method for extracting a reverberation environment embedding using a reverberation removal auto-encoder according to an embodiment includes: generating a reverberation speech for training from a speech database (DB) and an indoor impulse response (RIR) database (DB); performing multi-linear prediction on the generated reverberation speech and outputting a multi-linear prediction result; estimating a weight for a weighted sum of the multiple linear prediction results; estimating a final reverberation cancellation voice from the multiple linear prediction results and the estimated weight; and extracting a reverberation environment embedding vector from a weight estimation model for estimating the weight.

The method may further include training the entire model through a loss function of the estimated final reverberation-removed speech and the original speech before reverberation is added.

The outputting of the multiple linear prediction result includes converting the reverberation speech into a mel spectrogram, inputting it to a linear prediction model, applying zero padding, and a convolutional neural network composed of linear prediction coefficients. (Convolutional Neural Network) may be passed to output the multi-linear prediction result.

The step of estimating the weight includes converting the reverberation speech into a log mel spectrogram, inputting it into a weight estimation model, applying zero padding, passing it through a convolutional neural network to obtain an output, and each time - By applying a softmax function to the values belonging to the frequency index, it can be converted into a weight having a value between 0 and 1 and the sum is 1.

In the estimating of the final reverberation-removed speech, the linear prediction results and weights for each time-frequency index are obtained by element-wise multiplication of the multiple linear prediction results and the estimated weights. After the multiplication result is obtained, an estimated value of the final dereverberation speech may be obtained by adding C values of a set of linear prediction coefficients in each time-frequency index.

In the step of extracting the reverberation environment embedding vector, the target speech signal is converted into a log Mel spectrogram, input to the weight estimation model to obtain an output, and frame values for each channel-frequency from the weight obtained are obtained. A matrix may be obtained by taking the average, and the reverberation environment embedding vector may be extracted by rearranging the elements of the matrix.

In the step of training the entire model through the loss function, the obtained final reverberation cancellation voice and the original voice of the voice database (DB) are converted into log Mel spectrograms, respectively, and then the loss between the two log Mel spectrograms is performed. Training the neural network through the backpropagation algorithm can be repeated until the function converges.

An apparatus for extracting a reverberation environment embedding using an auto-encoder for removing reverberation according to another embodiment includes: a reverberation speech generator for generating a reverberation speech for training from a speech database (DB) and an indoor impulse response (RIR) database (DB); a linear prediction model for outputting multiple linear prediction results by performing multiple linear prediction on the generated reverberation speech; a weight estimation model for estimating a weight for a weighted sum of the multiple linear prediction results; a dereverberation speech estimator for estimating a final dereverberation speech from the multiple linear prediction results and the estimated weight; and an embedding extractor configured to extract a reverberation environment embedding vector from a weight estimation model for estimating the weight.

The method may further include a dereverberation auto-encoder training unit that trains the entire model through the estimated final dereverberation speech and the loss function of the original speech before reverberation is added.

The embedding extractor converts a target speech signal into a log Mel spectrogram, inputs it to the weight estimation model to obtain an output, and obtains a matrix by averaging frame values for each channel-frequency from the obtained weight. and rearranging the elements of the matrix to extract the reverberation environment embedding vector.

According to embodiments, the reverberation environment embedding extraction method using the reverberation removal auto-encoder extracts the reverberation environment embedding using the reverberation removal auto-encoder, and improves the speech recognition performance in the reverberation environment through a model adaptation method having the auxiliary feature. and devices may be provided.

According to embodiments, a reverberation environment embedding extraction method using a reverberation cancellation auto-encoder capable of naturally reflecting characteristics of each of the indoor impulse responses and their relationship to each other through a reverberation cancellation auto-encoder instead of an indoor impulse response (RIR) classifier and devices may be provided.

In addition, according to embodiments, from the viewpoint of the reverberation cancellation auto-encoder, instead of directly applying to the input feature of the acoustic model, an embedding vector is extracted and used as an auxiliary input, thereby compensating for the limitation of generating distortion due to inability to restore detailed information. A method and apparatus for extracting reverberation environment embeddings using a removal auto-encoder may be provided.

1 is a diagram illustrating an electronic device according to example embodiments.
2 is a block diagram illustrating an apparatus for extracting a reverberation environment embedding using a reverberation removal auto-encoder according to an exemplary embodiment.
3 is a flowchart illustrating a method of extracting a reverberation environment embedding using a reverberation cancellation auto-encoder according to an exemplary embodiment.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the described embodiments may be modified in various other forms, and the scope of the present invention is not limited by the embodiments described below. In addition, various embodiments are provided in order to more completely explain the present invention to those of ordinary skill in the art. The shapes and sizes of elements in the drawings may be exaggerated for clearer description.

The following embodiments deal with a method for improving speech recognition performance in a reverberation environment through a model adaptation method that extracts a reverberation environment embedding using a reverberation cancellation auto-encoder and uses the reverberation environment embedding as an auxiliary feature. In the case of the conventional R-vector, the reverberation environment embedding classifies which room impulse response (RIR) out of tens of thousands of reverberant speech was generated, which does not take into account any relationship between the room impulse response (RIR) at all. . This is because, due to the characteristics of the DNN classifier trained through cross entropy, both cases with very similar and large differences between the respective indoor impulse responses (RIRs) are treated the same. However, in the proposed method, the characteristics of each indoor impulse response (RIR) and their relationship to each other can be naturally reflected through the reverberation cancellation auto-encoder instead of the indoor impulse response (RIR) classifier. In addition, from the point of view of the reverberation cancellation auto-encoder, instead of directly applying to the input features of the acoustic model, an embedding vector is extracted and used as an auxiliary input, thereby compensating for the limitation of generating distortion due to inability to restore detailed information.

The structure of the reverberation cancellation auto encoder can be configured by adopting and supplementing the linear prediction-based reverberation cancellation method. Instead of estimating the linear prediction parameters for the target speech every time, a plurality of linear prediction parameters may be set in the DNN model and a final result may be obtained through a weighted sum thereof. This method has the advantage of being able to perform reverberation cancellation through a pre-trained model without estimating parameters each time while using a method that reflects the characteristics of reverberant speech well called linear prediction. The weights for the weighted summation are estimated through a separate network, and each time and frequency index of the spectrogram has different values. This has flexibility compared to the existing linear prediction method in which the parameters are fixed within the utterance. By averaging the weights over time, a vector of a fixed length can be obtained regardless of the length of the utterance, and this is used as a reverberation environment embedding vector.

1 is a diagram illustrating an electronic device according to example embodiments.

Referring to FIG. 1 , an electronic device 100 according to embodiments may include at least one of an input module 110 , an output module 120 , a memory 130 , and a processor 140 .

The input module 110 may receive a command or data to be used for a component of the electronic device 100 from the outside of the electronic device 100 . The input module 110 is at least one of an input device configured to allow a user to directly input a command or data to the electronic device 100 or a communication device configured to receive a command or data by wire or wireless communication with an external electronic device may include any one. For example, the communication device may include at least one of a wired communication device and a wireless communication device, and the wireless communication device may include at least one of a short-range communication device and a long-distance communication device.

The output module 120 may provide information to the outside of the electronic device 100 . The output module 120 is at least one of an audio output device configured to audibly output information, a display device configured to visually output information, or a communication device configured to transmit information by wire or wireless communication with an external electronic device may include any one. For example, the communication device may include at least one of a wired communication device and a wireless communication device, and the wireless communication device may include at least one of a short-range communication device and a long-distance communication device.

The memory 130 may store data used by components of the electronic device 100 . The data may include input data or output data for a program or instructions related thereto. For example, the memory 130 may include at least one of a volatile memory and a non-volatile memory.

The processor 140 may execute a program in the memory 130 to control the components of the electronic device 100 , and may process data or perform an operation. In this case, the processor 140 may include a reverberation speech generator, a linear prediction model, a weight estimation model, a dereverberation speech estimator, and an embedding extractor, and may further include a reverberation removal auto-encoder training unit according to an embodiment. . Through this, the processor 140 may perform the reverberation environment embedding extraction using the reverberation cancellation auto encoder.

2 is a block diagram illustrating an apparatus for extracting a reverberation environment embedding using a reverberation removal auto-encoder according to an exemplary embodiment.

Referring to FIG. 2 , the apparatus 200 for extracting reverberation environment embeddings using a reverberation removal auto-encoder according to an embodiment includes a reverberation speech generator 210 , a linear prediction model 220 , a weight estimation model 230 , and reverberation removal. It may include a voice estimator 240 and an embedding extractor 260 . According to an embodiment, the apparatus 200 for extracting the reverberation environment embedding using the reverberation removal auto-encoder may further include a reverberation removal auto-encoder training unit 250 . Here, the apparatus 200 for extracting the reverberation environment embedding using the reverberation removal auto-encoder may be included in the processor 140 of FIG. 1 .

The reverberation speech generator 210 may generate a reverberation speech for training from the speech database DB and the indoor impulse response (RIR) database DB.

The linear prediction model 220 may output multiple linear prediction results by performing multiple linear prediction on the generated reverberation speech. More specifically, after converting the reverberation speech into a mel spectrogram, it is input to the linear prediction model 220 to apply zero padding, and a convolutional neural network composed of linear prediction coefficients. It is possible to output multiple linear prediction results by passing .

The weight estimation model 230 may estimate a weight for a weighted sum of multiple linear prediction results. More specifically, after converting the reverberation speech into a log mel spectrogram, it is input to the weight estimation model 230 to apply zero padding, passes through a convolutional neural network to obtain an output, and each time-frequency By applying a softmax function to values belonging to the index, it can be converted into a weight having a value between 0 and 1 and the sum of which is 1.

The dereverberation speech estimator 240 may estimate the final dereverberation speech from the multiple linear prediction results and the estimated weight. More specifically, as the reverberation removal speech estimator 240 multiplies the multiple linear prediction results and the estimated weights by element (element-wise multiplication), the linear prediction result and the weight for each time-frequency index are After the multiplication result is obtained, an estimated value of the final dereverberation speech may be obtained by adding C values of a set of linear prediction coefficients in each time-frequency index.

The reverberation cancellation auto encoder training unit 250 may train the entire model through the loss function of the estimated final reverberation cancellation voice and the original voice before reverberation is added. More specifically, the reverberation cancellation auto encoder training unit 250 converts the obtained final reverberation cancellation voice and the original voice of the voice database (DB) into log Mel spectrograms, respectively, and then the loss function between the two log Mel spectrograms is Training the neural network through the backpropagation algorithm can be repeated until convergence.

The embedding extractor 260 may extract the reverberation environment embedding vector from the weight estimation model 230 for estimating the weight. More specifically, the embedding extraction unit 260 converts the target speech signal into a log Mel spectrogram, inputs it to the weight estimation model 230 to obtain an output, and from the obtained weights, frame values for each channel-frequency A matrix is obtained by taking the average of , and the reverberation environment embedding vector can be extracted by rearranging the elements of the matrix.

The apparatus 200 for extracting reverberation environment embeddings using the reverberation removal auto-encoder according to an embodiment may describe the method for extracting reverberation environment embeddings using the reverberation removal auto-encoder according to an embodiment, for example.

3 is a flowchart illustrating a method for extracting a reverberation environment embedding using a reverberation cancellation auto-encoder according to an exemplary embodiment.

Referring to FIG. 3 , the method for extracting reverberation environment embeddings using a reverberation cancellation auto-encoder according to an embodiment includes generating a reverberation speech for training from a speech database (DB) and an indoor impulse response (RIR) database (DB) (S110), performing multiple linear prediction on the generated reverberation speech and outputting a multiple linear prediction result (S120), estimating a weight for a weighted sum of multiple linear prediction results (S130), multiple linear prediction result The steps of estimating the final reverberation-removed speech from the values and the estimated weights (S140), and extracting the reverberation environment embedding vector from the weight estimation model 230 for estimating the weights (S160) may be included.

The method may further include training the entire model through a loss function of the estimated final reverberation-removed speech and the original speech before reverberation is added ( S150 ).

Hereinafter, each step of the method for extracting the reverberation environment embedding using the reverberation cancellation auto-encoder according to an embodiment will be described as an example.

In step S110 , the reverberation speech generator 210 may generate a reverberation speech for training from the speech database DB and the indoor impulse response (RIR) database DB.

That is, it is a process of generating a reverberation voice from the voice DB and the indoor impulse response (RIR) DB. The voice DB consists of voices recorded in a quiet environment free from noise and reverberation. The indoor impulse response (RIR) DB consists of an indoor impulse response (RIR) recorded in a real reverberation environment or an indoor impulse response (RIR) created in a virtual space through imaging methods. Reverberant speech is generated through convolution of an indoor impulse response (RIR) and speech. For each indoor impulse response (RIR), M reverberated speech is generated by convolution with M speeches randomly selected from the speech DB. Finally, if reverberation is generated for the entire indoor impulse response (RIR) DB composed of N indoor impulse responses (RIR), the total number of reverberated speeches becomes N * M.

In operation S120 , the linear prediction model 220 may perform multi-linear prediction on the generated reverberation speech to output a multi-linear prediction result.

That is, the linear prediction model 220 is a model that outputs multiple linear prediction results for the input reverberation speech. First, the input reverberation speech signal is converted into a K x N-dimensional mel spectrogram composed of K frequency bins and N frames through local Fourier transform (STFT) and mel filterbank filtering. The linear prediction model 220 includes C sets of linear prediction coefficients of length L. Through this, C linear predictions are performed for each of the K frequency bins, and a C x K x N-dimensional Mel spectrogram output is finally obtained. This process can be expressed by the following equation.

[Equation 1]

는

번째 선형 예측 계수를 나타낸다. 이 과정은 입력 스펙트로그램에 적절한 제로 패딩을 적용하고, 선형 예측 계수들로 이루어진 합성곱 층(convolution layer)을 통과하는 인공 신경망 구조로 구현 가능하다.where x is the Mel spectrogram of the input reverberation voice,

Is

represents the th linear prediction coefficient. This process can be implemented as an artificial neural network structure that applies appropriate zero padding to the input spectrogram and passes through a convolution layer made of linear prediction coefficients.

In step S130 , the weight estimation model 230 may estimate a weight for a weighted sum of multiple linear prediction results.

That is, the weight estimation model 230 is a model for estimating a weight for a weighted sum of multiple linear prediction results. The input of the weight estimation model 230 is used by taking the logarithm of the K x N-dimensional Mel spectrogram obtained in the same way as the linear prediction model 220 . Appropriate zero padding is applied here, and convolutional neural networks are passed through to obtain a C x K x N dimensional output. Then, the softmax function is applied to the C values belonging to each time-frequency index, which has a value between 0 and 1, and is converted into a weight whose sum is 1.

In operation S140 , the dereverberation speech estimator 240 may estimate the final dereverberation speech from the multiple linear prediction results and the estimated weight.

That is, it is a process of obtaining the final dereverberation speech from the linear prediction results and the estimated weight. In steps S120 and S130 , the outputs of the linear prediction model 220 and the weight estimation model 230 all have C x K x N dimensions. Element-wise multiplication of these results in the same C x K x N dimension. Since this is the result of multiplying the linear prediction result and the weight for each time-frequency index, adding C values in each time-frequency index finally gives the estimated value of the reverberation cancellation voice expressed as a K x N-dimensional spectrogram. can be obtained This can be expressed as a mathematical expression as follows.

[Equation 2]

는 가중치 추정 모델(230)에서 구한 가중치,

는 선형 예측 모델(220)에서 구한 선형 예측 결과이며,

은 각각 선형 예측, 주파수, 프레임 인덱스를 나타낸다. here,

is the weight obtained from the weight estimation model 230,

is the linear prediction result obtained from the linear prediction model 220,

denotes a linear prediction, a frequency, and a frame index, respectively.

In step S150 , the reverberation reduction auto encoder training unit 250 may train the entire model through the loss function of the estimated final dereverberation speech and the original speech before reverberation is added.

That is, it is a process of training a model by minimizing the difference between the estimated reverberation-removed speech and the original speech before reverberation is added. The reverberation voice generator 210 converts the original voice of the voice DB used for synthesizing the reverberation voice into a K x N-dimensional log Mel spectrogram. After taking a log of the result of the reverberation cancellation voice estimator 240, an L1 loss function value between two log Mel spectrograms is obtained. Construct a mini-batch from the training data and repeat the neural network training through the backpropagation algorithm until the loss function obtained through the above process converges.

In operation S160 , the embedding extraction unit 260 may extract the reverberation environment embedding vector from the weight estimation model 230 for estimating the weight.

That is, it is a process of obtaining a reverberation environment embedding vector from the output of the trained weight estimation model 230 . The target speech signal is converted into a log Mel spectrogram, and this is used as an input of the weight estimation model 230 to obtain an output. From the obtained C x K x N-dimensional weights, the average of N frame values for each channel-frequency is taken to obtain a C x K-dimensional matrix. Rearranging the elements of the matrix yields a C * K dimension vector, which becomes the reverberation environment embedding vector. The process of extracting the embedding vector from the weight can be expressed as the following equation.

[Equation 3]

는 잔향 환경 임베딩 벡터 e의 인덱스이다.here,

is the index of the reverberation environment embedding vector e.

As described above, according to embodiments, characteristics of each of the indoor impulse responses and their relationship may be naturally reflected through the reverberation cancellation auto-encoder instead of the indoor impulse response (RIR) classifier.

In addition, according to embodiments, from the viewpoint of the reverberation cancellation auto-encoder, instead of directly applying to the input feature of the acoustic model, an embedding vector is extracted and used as an auxiliary input, thereby compensating for the limitation of generating distortion due to inability to restore detailed information. have.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (10)

In the reverberation environment embedding extraction method using the reverberation removal auto encoder,
generating a reverberation voice for training from a voice database (DB) and an indoor impulse response (RIR) database (DB);
outputting multiple linear prediction results by performing multiple linear prediction on the reverberated speech generated through a linear prediction model;
estimating a weight for a weighted sum of the multiple linear prediction results through a weight estimation model;
estimating a final reverberation cancellation voice from the multiple linear prediction results and the estimated weight; and
extracting a reverberation environment embedding vector from the weight estimation model for estimating the weight
including,
Training the entire model through the estimated final dereverberation speech and the loss function of the original speech before reverberation is added.
further comprising,
The step of estimating the weight is
The weights for the weighted sum are estimated through the separate weight estimation model, and have different values in each time-frequency index of the spectrogram,
The step of extracting the reverberation environment embedding vector comprises:
After converting the target speech signal into a log Mel spectrogram, it is input to the weight estimation model to obtain an output, and from the obtained weights, the average of frame values for each channel-frequency is taken to obtain a matrix, As the elements are rearranged to extract the reverberation environment embedding vector, the reverberation environment embedding vector of a fixed length independent of the length of a utterance is obtained through the time average of the weights, and the reverberation environment embedding vector is used to sound What to use as an auxiliary input to the model
Characterized in, the reverberation environment embedding extraction method. delete According to claim 1,
The step of outputting the multiple linear prediction result comprises:
After the reverberation speech is converted into a mel spectrogram, it is input to a linear prediction model, zero padding is applied, and the multi-linear outputting the prediction result
Characterized in, the reverberation environment embedding extraction method. According to claim 1,
The step of estimating the weight is
After converting the reverberation speech into a log mel spectrogram, it is input to a weight estimation model, applied zero padding, passed through a convolutional neural network to obtain an output, and is soft to values belonging to each time-frequency index. Converting to weights that have a value between 0 and 1 and sum to 1 by applying the softmax function
Characterized in, the reverberation environment embedding extraction method. According to claim 1,
The step of estimating the final reverberation cancellation voice includes:
By element-wise multiplication of the multiple linear prediction results and the estimated weights, a result obtained by multiplying a linear prediction result and a weight for each time-frequency index is obtained, and then at each time - Obtaining the estimated value of the final dereverberation speech by adding C values of a set of linear prediction coefficients in the frequency index
Characterized in, the reverberation environment embedding extraction method. delete According to claim 1,
The step of training the entire model through the loss function is,
After converting the acquired final reverberation cancellation voice and the original voice of the voice database (DB) into log Mel spectrograms, respectively, training the neural network through the back propagation algorithm until the loss function between the two log Mel spectrograms converges. repeating
Characterized in, the reverberation environment embedding extraction method. In the reverberation environment embedding extraction apparatus using the reverberation removal auto encoder,
a reverberation speech generator for generating a reverberation speech for training from a speech database (DB) and an indoor impulse response (RIR) database (DB);
a linear prediction model for outputting multiple linear prediction results by performing multiple linear prediction on the generated reverberation speech;
a weight estimation model for estimating a weight for a weighted sum of the multiple linear prediction results;
a dereverberation speech estimator for estimating a final dereverberation speech from the multiple linear prediction results and the estimated weight; and
An embedding extraction unit for extracting a reverberation environment embedding vector from a weight estimation model for estimating the weight
including,
An autoencoder training unit for dereverberation that trains the entire model through the estimated final dereverberation speech and the loss function of the original speech before reverberation is added.
further comprising,
The weights for the weighted sum are estimated through the separate weight estimation model, and have different values in each time-frequency index of the spectrogram,
The embedding extraction unit,
After converting the target speech signal into a log Mel spectrogram, it is input to the weight estimation model to obtain an output, and from the obtained weights, the average of frame values for each channel-frequency is taken to obtain a matrix, As the elements are rearranged to extract the reverberation environment embedding vector, the reverberation environment embedding vector of a fixed length independent of the length of a utterance is obtained through the time average of the weights, and the reverberation environment embedding vector is used to sound What to use as an auxiliary input to the model
Characterized in, the reverberation environment embedding extraction device. delete delete

Images