JP7026357B2 - Time frequency mask estimator learning device, time frequency mask estimator learning method, program - Google Patents

Description

The present invention relates to a time-frequency mask estimator learning device for learning a time-frequency mask estimator using deep learning, a time-frequency mask estimator learning method, and a program.

The present invention can be used for all acoustic signal processing techniques utilizing deep learning, but in the present specification, sound enhancement will be described as an example.

<Signal analysis and speech enhancement using STFT>
In order to perform acoustic signal processing, it is first necessary to observe the sound using a microphone. The observed sound contains noise in addition to the target sound to be processed. Speech enhancement refers to signal processing that extracts the target sound from the observation signal containing noise.

Define speech enhancement. Let x _k be the observation signal of the microphone, and let x _k be a mixed signal of the target sound _sk and the noise _nk .
x _k = s _k + n _k … (1)
Where k is an index of time in the time domain. In order to extract the target sound from the observation signal, the observation signal in the time domain is used.

Consider analyzing L points for each point. After that, the t ∈ {0,…, T} th signal that summarizes the observed signals in that way.

Is expressed as the observation signal in the t-frame. Where ^T represents transpose. Then, the observation signal at the t-frame can be described as follows from Eq. (1).
x _t = s _t + n _t … (3)

Here, _{st and n t are} the target sound and noise of the t-frame defined as in Eq. (2), respectively. In the time-frequency analysis of signals using STFT, STFT is applied to the observed signal in each time frame. The signal after STFT satisfies the following properties.

here

Is the analysis result obtained as a result of STFT of the observation signal at the t-frame.

Time-frequency mask processing is one of the typical methods in speech enhancement. In this process, the time-frequency mask is applied to the observed signal after STFT.

By multiplying by, the estimated value of the target sound after STFT is obtained as follows.

Here, ○ is the Hadamard product. Lastly,

By executing reverse STFT (ISTFT: inverse-STFT), the estimated value of the target sound in the time domain is obtained.

Now, a function with the parameter θ _G that estimates G _t from the observed signal

And put. Then, G _t is defined as follows.

Here, φ _t is an acoustic feature extracted from X _t , and the amplitude spectrum of X _t or the like is used. In addition, in speech enhancement using deep learning, which has been actively studied in recent years,

The mainstream method is to design a deep neural network (DNN). After that,

Is implemented using DNN.

C. Trabelsi, O. Bilaniuk, Y. Zhang, D. Serdyuk, S. Subramanian, JF Santos, S. Mehri, N. Rostamzadeh, Y. Bengio, and CJ Pal, "Deep complex networks," in Int. Conf. Learn. Representat., 2018.

<Problems of signal analysis and speech enhancement using STFT>
In audio / acoustic signal processing, it is rare to handle the waveform as it is, and in many cases, as described above, the observed signal is Fourier transformed (STFT) every short time interval to emphasize or identify the signal. times. However, STFT is a conversion from a real number to a complex number, and deep learning using a complex number complicates the learning, so that only the amplitude information of the STFT spectrum is often used or controlled. Also, when inputting to the DNN, only the amplitude information is often used as the feature quantity. For example, when the above-mentioned φ _t is an amplitude spectrum of X _t , this means that the information of the phase spectrum is ignored, so it can be said that the information obtained from the observation signal is fully utilized. do not have. However, since many DNN modules are designed on the assumption that they process real numbers, there is a problem in inputting X _t as it is. Although CNN and BN using complex numbers are proposed in Non-Patent Document 1, they are proposed as a general theory of machine learning and are not specialized in acoustic signal processing.

Therefore, an object of the present invention is to provide a time-frequency mask estimator learning device for learning a time-frequency mask estimator using the real part and the imaginary part of the STFT of the observed signal.

The time-frequency mask estimator learning device of the present invention includes a time-frequency mask estimator and a learning unit, and the time-frequency mask estimator is a time for estimating a time-frequency mask for estimating a target sound from an arbitrary observation signal. It is a frequency mask estimator and includes a CNN processing unit, a BN processing unit, and a GLU processing unit.

The CNN processing unit executes convolutional neural network processing on the real part of the FTFT spectrum of the observation signal and the value in which the corresponding imaginary part is regarded as a real number. The BN processing unit executes batch normalization processing using parameters common to norm operations for values in which the real part of the FTFT spectrum of the observation signal and the corresponding imaginary part are regarded as real numbers. The GLU processing unit executes gate linear unit processing on a value obtained by combining a real part of the FTFT spectrum of the observation signal and a value in which the corresponding imaginary part is regarded as a real number. The learning unit uses the time frequency so that the cost function between the known observation signal obtained by superimposing the known target sound and the known noise multiplied by the time frequency mask and the known target sound is minimized. Learn the parameters of the mask estimator.

According to the time-frequency mask estimator learning device of the present invention, it is possible to learn a time-frequency mask estimator using the real part and the imaginary part of the STFT of the observed signal.

The block diagram which shows the structure of the time frequency mask estimator learning apparatus of embodiment. The flowchart which shows the operation of the time frequency mask estimator learning apparatus of embodiment. The flowchart which shows the detailed operation of the time frequency mask estimator of embodiment. The figure which shows the configuration example of the time frequency mask estimator which combined three modules and auxiliary operation.

Hereinafter, embodiments of the present invention will be described in detail. The components having the same function are given the same number, and duplicate explanations are omitted.

In Example 1, as DNN modules based on the characteristics of the phase spectrum, (1) constrained complex convolutional neural network (CNN), (2) simplified complex batch normalization (BN: batch normalization), (3). ) Disclose three modules of complex gate linear unit (GLU). In the following, I = √ (-1), and () _Re and () _Im represent the real and imaginary parts of complex numbers, respectively. Note that () _Im refers to a value in which the imaginary part is regarded as a real number (value obtained by removing the complex unit from the imaginary part).

<Restricted complex CNN>
In the first embodiment, a complex CNN having the following format is executed.
W * z = (W _Re * z _{Re --W Im} * z _Im ) + I (W _Re * z _Im + W _Im * z _Re )… (8)

Where W represents the convolutional layer of the weight parameter W. As a variation, it is also possible to limit the convolution weight to a real number.
W _Re * z = (W _Re * z _Re ) + I (W _Re * z _Im )… (9)

By the way, the phase spectrum of an audio signal has a property of continuously changing in the time direction. Therefore, in this embodiment, unlike Non-Patent Document 1, the stride of Eqs. (8) and (9) in the time direction is restricted to 1.

<Simplified complex BN>
BN is a process that sets the mean and variance in the mini-batch used for learning DNN to 0 and 1, respectively. It is also defined as follows using learnable expansion and translation parameters.

here

Is the mean and variance in the mini-batch, and ε> 0 is a small positive constant. For example, ε may be set to about 10 ^-5 . This works well for real numbers, but it doesn't make sense to apply it straight to complex numbers. This is because, for example, if BN is applied separately to the real part and the imaginary part, the relation of the norm of the complex variable (corresponding to the relation of the amplitude of each frequency) and the phase spectrum are destroyed. Processing based on principal component analysis as proposed in Non-Patent Document 1 is also not valid for the same reason. On the other hand, the process of subtracting the average value from the set of complex numbers is equivalent to a high-pass filter, and the process of multiplying the scalar in the set of complex numbers can be regarded as the adjustment of the filter gain. Therefore, in this embodiment, the following simplified complex BN is proposed.

In equation (11), the subtraction of the mean value is performed separately for the real part and the imaginary part, but the norm operation uses the same parameters (the part corresponding to the denominator of equation (11)) for the real part and the imaginary part. The point is different from the conventional method. Also, the learnable parameters γ and β are not used for the same reason.

<Complex GLU>
Consider an activation function that is valid for complex spectra. Non-Patent Document 1 proposes the following three types of activation functions.

Where | | is an absolute value and Arg [] is a function that returns the argument of a complex number. In this embodiment, in order to preserve the phase spectrum, consider the activation function of the form of Eq. (14). That is, it is an activation function that manipulates only the amplitude spectrum.

A typical acoustic signal processing method that manipulates only the amplitude spectrum is the time-frequency mask in Eq. (5). Therefore, consider a CNN that inputs the real part and the imaginary part and outputs the time-frequency mask.

Where concat is a combination of two variables in the channel direction. This format is similar to GLU. GLU is an activation function that inputs a real value and outputs a mask, but in this embodiment, GLU (i.e. complex GLU) that combines complex values in the channel direction and outputs a mask is executed.

<Embodiment>
Hereinafter, as a specific embodiment, the time-frequency mask estimator learning device 1 will be described. As shown in FIG. 1, the time-frequency mask estimator learning device 1 has a configuration including a time-frequency mask estimator 11 and a learning unit 12, and includes a target sound DB 91 and a noise DB 92 outside or inside the device. It is a composition. Further, the time-frequency mask estimator 11 includes a CNN processing unit 111, a BN processing unit 112, a GLU processing unit 113, and an auxiliary calculation unit 114.

As shown in FIG. 2, the time-frequency mask estimator 11 randomly selects a target sound from the target sound DB 91 and noise from the noise DB 92, superimposes the noise, simulates an observation signal, and uses the observation signal as an STFT. On the other hand, CNN processing, BN processing, and GLU processing are executed to estimate a time-frequency mask for estimating a target sound from an arbitrary observation signal (S11). When the step S11 is executed for the first time, the parameter of the time frequency mask G is initialized with, for example, some random number. The TFT processing may be executed by the time-frequency mask estimator 11 or may be executed by the TFT processing unit (not shown).

The learning unit 12 has an arbitrary cost between a known observation signal obtained by superimposing a known target sound and a known noise multiplied by a time frequency mask (Equation (5)) and a known target sound. The parameters of the time-frequency mask estimator 11 are learned so that the function is minimized (S12). As the learning method, a stochastic steepest descent method may be used, and the learning rate may be set to about ^10-5 .

The time-frequency mask estimator learning device 1 makes a convergence test, and if it does not converge, the process returns to step S11. The convergence test rule may be, for example, whether or not S12 is repeated a certain number of times (for example, 100,000 times).

The details of step S11 are shown in FIG. The time-frequency mask estimator 11 repeatedly executes CNN processing (S111), BN processing (S112), GLU processing (S113), and other processing (S114) with respect to the FTFT of the observation signal in a predetermined order. Hereinafter, the details of steps S111 to S114 will be described.

The CNN processing unit 111 sets the stride in the time direction to 1 based on the equation (8) or (9) with respect to the value in which the real part of the FTFT spectrum of the observation signal and the corresponding imaginary part are regarded as real numbers. Convolutional neural network processing is executed with constraints (S111). The BN processing unit 112 is a parameter (1 / √ (σ ² _BRe + σ ² _BIm + ε)) common to the norm operation for the real part of the FTFT spectrum of the observation signal and the value in which the corresponding imaginary part is regarded as a real number. The batch normalization process is executed based on the equation (11) using the above (S112). Based on Eq. (15), the GLU processing unit 113 combines the real part z _Re of the SFT spectrum of the observed signal and the value z _Im which regards the corresponding imaginary part as a real number, and concat (z _Re , z _Im ). ), The gate linear unit processing is executed (S113). The auxiliary calculation unit 114 executes other processing (for example, concat or sigmoid) other than the above (S114).

FIG. 4 is a diagram showing a configuration example of a time-frequency mask estimator (neural network) in which three modules are combined. “Conv”, “BN”, “GLU”, “Concat”, and “sigmoid” represent the constrained complex CNN, the simplified complex BN, the complex GLU, the coupling in the channel direction of the input, and the operation of the sigmoid function, respectively. c, k, s, and p are the number of channels, kernel size, stride size, and padding size, respectively.

<Effect>
According to the time-frequency mask estimator learning device 1 of the first embodiment, as a DNN module for inputting a complex spectrum as it is, (1) a constrained convolutional neural network (CNN), and (2) a simplification. Since we introduced three types of complex normalization (BN: batch normalization) and (3) complex gate linear unit (GLU), the FFT spectrum remains complex and the processing according to the signal processing theory is performed by DNN. Can be realized with.

<Supplementary note>
The device of the present invention is, for example, as a single hardware entity, an input unit to which a keyboard or the like can be connected, an output unit to which a liquid crystal display or the like can be connected, and a communication device (for example, a communication cable) capable of communicating outside the hardware entity. Communication unit, CPU (Central Processing Unit, cache memory, registers, etc.) to which can be connected, RAM and ROM as memory, external storage device as hard hardware, and input, output, and communication units of these. , CPU, RAM, ROM, has a bus connecting so that data can be exchanged between external storage devices. Further, if necessary, a device (drive) or the like capable of reading and writing a recording medium such as a CD-ROM may be provided in the hardware entity. As a physical entity equipped with such hardware resources, there is a general-purpose computer or the like.

The external storage device of the hardware entity stores a program required to realize the above-mentioned functions and data required for processing of this program (not limited to the external storage device, for example, reading a program). It may be stored in a ROM, which is a dedicated storage device). Further, the data obtained by the processing of these programs is appropriately stored in a RAM, an external storage device, or the like.

In the hardware entity, each program stored in the external storage device (or ROM, etc.) and the data required for processing of each program are read into the memory as needed, and are appropriately interpreted and executed and processed by the CPU. .. As a result, the CPU realizes a predetermined function (each configuration requirement represented by the above, ... Department, ... means, etc.).

The present invention is not limited to the above-described embodiment, and can be appropriately modified without departing from the spirit of the present invention. Further, the processes described in the above-described embodiment are not only executed in chronological order according to the order described, but may also be executed in parallel or individually as required by the processing capacity of the device that executes the processes. ..

As described above, when the processing function in the hardware entity (device of the present invention) described in the above embodiment is realized by the computer, the processing content of the function that the hardware entity should have is described by the program. Then, by executing this program on the computer, the processing function in the above hardware entity is realized on the computer.

The program describing the processing content can be recorded on a computer-readable recording medium. The recording medium that can be read by a computer may be, for example, a magnetic recording device, an optical disk, a photomagnetic recording medium, a semiconductor memory, or the like. Specifically, for example, a hard disk device, a flexible disk, a magnetic tape or the like as a magnetic recording device, and a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read Only) as an optical disk. Memory), CD-R (Recordable) / RW (ReWritable), etc., MO (Magneto-Optical disc), etc. as a magneto-optical recording medium, EEP-ROM (Electronically Erasable and Programmable-Read Only Memory), etc. as a semiconductor memory. Can be used.

Further, the distribution of this program is performed, for example, by selling, transferring, renting, or the like a portable recording medium such as a DVD or a CD-ROM in which the program is recorded. Further, the program may be stored in the storage device of the server computer, and the program may be distributed by transferring the program from the server computer to another computer via the network.

A computer that executes such a program first temporarily stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. Then, when the process is executed, the computer reads the program stored in its own recording medium and executes the process according to the read program. Further, as another execution form of this program, a computer may read the program directly from a portable recording medium and execute processing according to the program, and further, the program is transferred from the server computer to this computer. You may execute the process according to the received program one by one each time. In addition, the above-mentioned processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and the result acquisition without transferring the program from the server computer to this computer. May be. The program in this embodiment includes information to be used for processing by a computer and equivalent to the program (data that is not a direct command to the computer but has a property that regulates the processing of the computer, etc.).

Further, in this form, the hardware entity is configured by executing a predetermined program on the computer, but at least a part of these processing contents may be realized in terms of hardware.

Claims (5)

A real part of the FTFT spectrum of the observed signal, a CNN processing part that performs convolutional neural network processing on the value that regards the corresponding imaginary part as a real number, and
A BN processing unit that executes batch normalization processing that uses parameters common to norm operations for values that regard the real part of the FTFT spectrum of the observation signal as a real number and the corresponding imaginary part.
Includes a GLU processing unit that performs gate linear unit processing on a value that combines the real part of the FTFT spectrum of the observed signal and the value that considers the corresponding imaginary part as a real number.
A time-frequency mask estimator that estimates the time-frequency mask for estimating the target sound from an arbitrary observation signal,
The time-frequency mask estimation so that the cost function between the value obtained by multiplying the known observation signal obtained by superimposing the known target sound and the known noise and the time-frequency mask and the known target sound is minimized. A time-frequency mask estimator learning device that includes a learning unit that learns instrument parameters. The time-frequency mask estimator learning device according to claim 1.
The CNN processing unit is a time-frequency mask estimator learning device that executes the convolutional neural network processing by limiting the stride in the time direction to 1. A time-frequency mask estimator that estimates a time-frequency mask for estimating a target sound from an arbitrary observation signal,
A step of executing a convolutional neural network process on a value in which the real part of the FTFT spectrum of the observed signal and the corresponding imaginary part are regarded as real numbers,
A step to execute a batch normalization process that uses parameters common to norm operations for values that regard the real part of the FTFT spectrum of the observed signal and the corresponding imaginary part as real numbers.
Perform the step of performing gate linear unit processing on the value obtained by combining the real part of the FTFT spectrum of the observation signal and the value in which the corresponding imaginary part is regarded as a real number.
The time-frequency mask estimator learning device
The time-frequency mask estimation so that the cost function between the value obtained by multiplying the known observation signal obtained by superimposing the known target sound and the known noise and the time-frequency mask and the known target sound is minimized. A time-frequency mask estimator learning method that performs the steps of learning instrument parameters. The time-frequency mask estimator learning method according to claim 3.
A time-frequency mask estimator learning method that executes the convolutional neural network process by limiting the stride in the time direction to 1. A program that causes a computer to function as the time-frequency mask estimator learning device according to claim 1.

Images