WO2020110228A1 - Information processing device, program and information processing method - Google Patents

Abstract

The present invention is provided with: a time/frequency conversion unit (104) which generates a first spectrum component and a second spectrum component by converting each of a first observation digital signal and a second observation digital signal, generated from observed sound, into a frequency domain signal; a mask generation unit (105) which, using a cross-correlation function of the first spectrum component and the second spectrum component and on the basis of a time difference between the time of arrival of the observed sound at a first microphone and the time of arrival of the observed sound at a second microphone, calculates a filtering coefficient for masking a spectrum component of sound arriving from a direction different from a first direction from which target sound arrives; and a masking filter unit (110) which separates the spectrum component by performing masking with respect to the first spectrum component using the filtering coefficient.

Description

Information processing apparatus, program, and information processing method

The present invention relates to an information processing device, a program, and an information processing method.

With the progress of digital signal processing technology in recent years, hands-free voice operation by voice recognition in the automobile or in the living room of the home, or hands-free call for making a hand-held call has become widespread. Further, an abnormal sound monitoring system has also been developed, which catches and detects an abnormal sound generated by a machine or a sound such as a scream of a person.

These hands-free voice operation systems, hands-free call systems, and abnormal sound monitoring systems are designed to produce a target sound such as a sound or abnormal sound under various noise environments such as in a moving vehicle, factory, office, or living room at home. A microphone is installed to collect the. However, such a microphone collects not only the target sound but also ambient noise other than the target sound and other voices (hereinafter, referred to as disturbing sound).

As a method of individually extracting the target sound from the voice, for example, when using a plurality of microphones, a method by beam forming such as directing the direction of the target sound by signal processing, or directing a blind spot to the disturbing sound, or , There is a method of estimating a mixing matrix by independent component analysis. However, although beamforming is excellent in suppressing noise, it is not so effective in separating voice, and independent component analysis has a problem that performance is deteriorated due to the influence of reverberation or noise. Further, generally, in a real environment, the number of noise sources of the interfering sound is not limited to one, and there is a constraint that it is difficult to separate more sound sources than the number of microphones.

On the other hand, under the assumption of sparseness that the target sound signal and the interfering sound signal do not overlap each other in the time frequency domain, frequency components other than the target sound are masked to separate the sound source signal, which is called binary masking. A method has been proposed. Binary masking is an effective method for suppressing directional interference that is easy to implement.

As a method based on this binary masking, there is a technique disclosed in Patent Document 1. Patent Document 1 discloses a method of increasing the accuracy of binary masking for mixed speech in which sparseness is not guaranteed by intentionally causing an amplitude difference between power spectra.

JP, 2010-239424, A

However, the conventional method has a problem that an error occurs in the mask coefficient because a power difference is intentionally generated between the power spectra of the main microphone input signal and the sub microphone input signal.

One or more aspects of the present invention have been made to solve such a problem, and an object thereof is to make it possible to easily obtain a high quality target signal.

An information processing apparatus according to one aspect of the present invention is based on a first observation analog signal generated by a first microphone based on an observation sound including a target sound coming from a first direction, and the observation sound. Receiving the input of the second observation analog signal generated by the second microphone and converting each of the first observation analog signal and the second observation analog signal into a digital signal, the first observation digital signal An analog/digital conversion unit that generates a signal and a second observed digital signal, and converts each of the first observed digital signal and the second observed digital signal into a frequency domain signal, By using a time/frequency conversion unit that generates a spectrum component and a second spectrum component and a cross-correlation function of the first spectrum component and the second spectrum component, the observed sound is transmitted to the first microphone. A mask generation unit that calculates a filtering coefficient for masking a spectral component of a sound coming from a direction different from the first direction, based on a time difference between the arrival time and the arrival time at the second microphone; By masking the first spectrum component using the filtering coefficient, a masking filter unit that separates the spectrum component, and by converting the separated spectrum component into a signal in the time domain, And a time/frequency inverse converter that generates an output digital signal.

A program according to one aspect of the present invention causes a computer to generate a first observation analog signal generated by a first microphone based on an observation sound including a target sound coming from a first direction, and the observation sound. Based on the input of the second observation analog signal generated by the second microphone based on the above, the first observation analog signal and the second observation analog signal are each converted into a digital signal, thereby performing the first observation. An analog/digital converter that generates a digital signal and a second observed digital signal, each of the first observed digital signal and the second observed digital signal is converted into a signal in the frequency domain, and The observed sound arrives at the first microphone using a time/frequency conversion unit that generates a spectral component and a second spectral component, and a cross-correlation function of the first spectral component and the second spectral component. A mask generation unit that calculates a filtering coefficient for masking a spectral component of a sound coming from a direction different from the first direction, based on a time difference between the time when the sound arrives at the second microphone and the time when the sound arrives at the second microphone; A masking filter unit that separates the spectral component of 1 by masking the spectral component by using the filtering coefficient, and by converting the separated spectral component into a signal in the time domain, output It is characterized in that it functions as a time/frequency inverse conversion unit that generates a digital signal.

An information processing method according to one aspect of the present invention is based on a first observed analog signal generated by a first microphone based on an observed sound including a target sound coming from a first direction, and the observed sound. Receiving the input of the second observation analog signal generated by the second microphone and converting each of the first observation analog signal and the second observation analog signal into a digital signal, the first observation digital signal Generating a signal and a second observed digital signal, and converting each of the first observed digital signal and the second observed digital signal into a signal in the frequency domain to obtain a first spectral component and a second spectral component. A spectral component is generated, and using the cross-correlation function of the first spectral component and the second spectral component, the time at which the observed sound arrives at the first microphone and the arrival at the second microphone. The filtering coefficient for masking the spectral component of the sound arriving from the direction different from the first direction is calculated according to the time difference from the time to be used, and the filtering coefficient is used for the first spectral component. An output digital signal is generated by separating the spectral components by performing masking and converting the separated spectral components into a signal in the time domain.

According to one or more aspects of the present invention, it is possible to easily obtain a high quality target signal.

It is a block diagram which shows roughly the structure of the sound source separation device which concerns on Embodiments 1 and 3. FIG. 4 is a block diagram schematically showing an internal configuration of a mask generation unit in the first to third embodiments. It is a schematic diagram for demonstrating arrangement|positioning of a 1st microphone and a 2nd microphone, and the arrival direction of a target sound. (A) to (C) are graphs for explaining the utterance amount ratio when the target speaker and the disturbing speaker speak. (A) And (B) is a graph for explaining the effect in the first embodiment. It is a block diagram which shows the 1st hardware structural example of a sound source separation apparatus. It is a block diagram which shows the 2nd hardware structural example of a sound source separation device. It is a flow chart which shows operation of a sound source separation device. 5 is a block diagram schematically showing a configuration of an information processing system including a sound source separation device according to a second embodiment. FIG. It is a schematic diagram which shows an example of the method of excluding the influence of noise other than a target sound and a disturbance sound.

Embodiment 1.
FIG. 1 is a block diagram schematically showing the configuration of a sound source separation device 100 as an information processing device according to the first embodiment.
The sound source separation device 100 includes an analog/digital conversion unit (hereinafter referred to as A/D conversion unit) 103, a time/frequency conversion unit (hereinafter referred to as T/F conversion unit) 104, a mask generation unit 105, and a masking filter. A unit 110, a time/frequency inverse conversion unit (hereinafter referred to as T/F inverse conversion unit) 111, and a digital/analog conversion unit (hereinafter referred to as D/A conversion unit) 112 are provided.
The sound source separation device 100 is connected to a first microphone 101 and a second microphone 102.

FIG. 2 is a block diagram schematically showing the internal configuration of the mask generation unit 105.
The mask generation unit 105 includes a mask coefficient calculation unit 106, an utterance amount ratio calculation unit 107, a gain calculation unit 108, and a mask correction unit 109.
Hereinafter, the configuration and operation principle of the sound source separation device 100 according to the first embodiment will be described with reference to FIGS. 1 and 2. The sound source separation device 100 forms a masking filter based on the signal in the frequency domain generated from the signals in the time domain acquired by the first microphone 101 and the second microphone 102, and uses the masking filter as the first microphone. By multiplying the signal in the frequency domain corresponding to the signal acquired in 101, the output signal of the target sound from which the interfering sound is removed is obtained.

Here, the first observed analog signal acquired by the first microphone 101 is also referred to as a first channel Ch1, and the second observed analog signal acquired by the second microphone 102 is also referred to as a second channel Ch2. ..
Further, in order to simplify the following description, as shown in FIG. 3, the first microphone 101 and the second microphone 102 are located on the same horizontal plane, and their positions are known. Yes, and does not change over time. Further, it is assumed that the direction range in which the target sound and the disturbing sound can arrive does not change with time. The direction in which the target sound arrives is also called the first direction, and the direction in which the disturbing sound arrives is also called the second direction.
Here, it is assumed that the target sound and the disturbing sound are voices from different single speakers.

The first microphone 101 generates a first observation analog signal by converting the observation sound into an electric signal. The first observed analog signal is given to the A/D conversion unit 103.
The second microphone 102 generates a second observed analog signal by converting the observed sound into an electric signal. The second observed analog signal is provided to the A/D conversion unit 103.

The A/D conversion unit 103 performs analog/digital conversion () on each of the first observed analog signal given from the first microphone 101 and the second observed analog signal given from the second microphone 102. Hereinafter, each is converted into a digital signal by performing A/D conversion) to generate a first observed digital signal and a second observed digital signal.

For example, the A/D conversion unit 103 samples the first observation analog signal given from the first microphone 101 at a predetermined sampling frequency and converts it into a digital signal divided into frame units. By doing so, the first observed digital signal is generated. Similarly, the A/D conversion unit 103 samples the second observation analog signal supplied from the second microphone 102 at a predetermined sampling frequency to obtain a digital signal divided into frame units. The second observed digital signal is generated by the conversion. Here, the sampling frequency is, for example, 16 kHz, and the frame unit is, for example, 16 ms.

The first observed digital signal generated from the first observed analog signal in the frame interval corresponding to the sample number t is represented by a code x ₁ (t), and the second observed digital signal in the frame interval corresponding to the sample number t is represented. The second observed digital signal generated from the observed analog signal is represented by the code x ₂ (t).
The first observed digital signal x ₁ (t) and the second observed digital signal x ₂ (t) are provided to the T/F conversion unit 104.

The T/F conversion unit 104 receives the first observed digital signal x ₁ (t) and the second observed digital signal x ₂ (t), and receives the first observed digital signal x ₁ (t) in the time domain and The second observed digital signal x ₂ (t) is converted into a first short-time spectrum component X ₁ (ω, τ) and a second short-time spectrum component X ₂ (ω, τ) in the frequency domain. However, ω represents a spectrum number that is a discrete frequency, and τ represents a frame number.

Specifically, the T/F conversion unit 104 performs, for example, Fast Fourier Transform of 512 points on the _first observed digital signal x ₁ (t), and thus the first short-time spectrum component X ₁ Generate (ω, τ). Similarly, the T/F conversion unit 104 generates a _second short-time spectrum component X ₂ (ω, τ) from the _second observed digital signal x ₂ (t).
In the following, unless otherwise specified, the short-time spectrum component of the current frame is simply described as a spectrum component and its description is omitted.

The mask generation unit 105 receives the first spectral component X ₁ (ω, τ) and the second spectral component X ₂ (ω, τ) and receives a time that is a filtering coefficient for performing masking for separating the target sound. The frequency filter coefficient b _mod (ω, τ) is calculated. For example, the mask generation unit 105 uses the cross-correlation function of the first spectral component X ₁ (ω, τ) and the second spectral component X ₂ (ω, τ) to determine that the observation sound is the first microphone 101. And a time difference between the second microphone 102 and the second microphone 102, a filtering coefficient for masking a spectral component of a sound coming from a direction different from the first direction in which the target sound arrives is calculated. ..

In _{obtaining the} time-frequency filter coefficient b _mod (ω, τ), as shown in FIG. 3, in the horizontal plane where the first microphone 101 and the second microphone 102 are provided, the first microphone 101 with respect to the vertical direction V ₂ of the vertical V ₁ and second microphones 102, from a direction included in a predetermined angle theta, it is assumed that the target sound comes. Incidentally, interference sound is the vertical direction V ₂ of the vertical V ₁ and second microphones 102 of the first microphone 101, it is assumed that the target sound coming from the opposite side.

Here, the vertical direction _{V 2} of the vertical _{V 1} and second microphones 102 of the first microphone 101, to the straight line connecting the first microphone 101 and second microphone 102, which are perpendicular And The vertical direction V ₂ of the vertical V ₁ and second microphones 102 of the first microphone 101, a reference direction is predetermined, not necessarily vertical.
The distance between the first microphone 101 and the second microphone 102 is the distance d.

In order to determine whether the sound collected by the first microphone 101 and the second microphone 102 is the target sound or the disturbing sound, the arrival of sound using the signals from the first microphone 101 and the second microphone 102. It is necessary to estimate whether the direction is within the desired range. Here, since the time difference that occurs between the signals from the first microphone 101 and the second microphone 102 is determined by the angle θ, the arrival direction can be estimated by using this time difference. This will be described below with reference to FIGS. 2 and 3.

The mask coefficient calculation unit 106 first calculates the cross-correlation function of the first spectral component X ₁ (ω, τ) and the second spectral component X ₂ (ω, τ) as shown in the following equation (1). The cross spectrum D(ω, τ) is calculated. Then, the mask coefficient calculation unit 106 gives the calculated cross spectrum D(ω, τ) to the utterance amount ratio calculation unit 107.

Next, the mask coefficient calculation unit 106 obtains the phase Θ _D (ω, τ) of the cross spectrum D(ω, τ) using the following equation (2).

Here, Q(ω, τ) and K(ω, τ) represent the imaginary part and the real part of the cross spectrum D(ω, τ), respectively.

The phase Θ _D (ω, τ) obtained by the above equation (2) means the phase angle for each spectral component of the first channel Ch1 and the second channel Ch2, which is defined by the discrete frequency ω. The division represents the time delay between the two signals. That is, the time difference δ(ω, τ) between the first channel Ch1 and the second channel Ch2 can be expressed by the following equation (3).

Then, sound theoretical value [delta] _theta time difference observed when coming from the direction of an angle theta, using the distance d, may be expressed as the following equation (4). However, c is the speed of sound.

Here, if the set of θ that satisfies θ>θ _th is set as a desired directional range, the theoretical time difference δ _{θ_th} and the time difference δ(ω, τ) of the observed analog signal are compared. , It is possible to estimate whether the voice comes from the desired directional range.
Therefore, the mask coefficient b(ω, τ) for performing masking for separating the target sound can be expressed by the following equation (5).

In other words, the mask coefficient calculation unit 106 uses the cross-correlation function of the first spectral component X ₁ (ω, τ) and the second spectral component X ₂ (ω, τ) to determine that the target sound is the first A first time difference between the time of arrival at the microphone 101 and the time of arrival at the second microphone 102, the time of arrival of an interfering sound at the first microphone 101, and the time of arrival at the second microphone 102. From the second time difference of, the sound that comes from the first range including the first direction in which the target sound arrives and the second range that includes the second direction in which the interfering sound arrives The mask coefficient for separating the spectrum component of the sound coming from the direction included in the first range from the spectrum component of the sound coming from the direction included in the second range is calculated.

The mask coefficient b(ω, τ) shown in the equation (5) is 1 when the target sound is estimated and is M when the disturbing sound is estimated. Here, when M=0, the mask coefficient has a binary value of 1 or 0 (binary), so a filter having such a mask coefficient is called a binary mask. A decimal other than binary may be used as the filter coefficient, and the filter in this case is also called a soft mask. However, the filter coefficient has a value of less than 1 for both the target sound and the disturbing sound. In this embodiment, for example, M=0.5 is used.
The mask coefficient calculation unit 106 gives the mask coefficient b(ω, τ) to the mask correction unit 109.

The utterance amount ratio calculation unit 107 calculates the first spectrum component X ₁ (ω, τ) of the first channel Ch1, the second spectrum component X ₂ (ω, τ) of the second channel Ch2, and the cross spectrum. By receiving D(ω, τ), the utterance amount ratio, which is the ratio between the utterance amount of the target sound speaker and the utterance amount of the disturbing sound speaker, is calculated. In other words, the utterance amount ratio is the interference of the amount of the spectral component of the sound coming from the first range including the first direction in which the target sound comes in the first spectral component X ₁ (ω, τ). It is the ratio to the amount of the spectral component of the sound coming from the second range including the second direction in which the sound comes.

First, the utterance amount ratio calculation unit 107 obtains the first power spectrum P ₁ (ω, τ) of the first channel Ch1 from the first spectrum component X ₁ (ω, τ) of the first channel Ch1. It is calculated from the following equation (6).

However, X _Re is the real part of the first spectral component X ₁ (ω, τ), and X _Im is the imaginary part of the first spectral component X ₁ (ω, τ).

Then, the utterance amount ratio calculation unit 107 uses the sign of the imaginary part Q(ω, τ) of the cross spectrum D(ω, τ) shown in the above equation (1) to observe the analog of the target voice. It is determined whether the signal comes from the target sound side or the interfering sound side. Then, the utterance amount ratio calculation unit 107 adds the first power spectrum P1(ω, τ) of the first channel Ch1 according to the code determination result, as shown in the following expression (7), The utterance amount s _Tgt (τ) of the _target speaker and the utterance amount s _Int (τ) of the disturbing speaker are respectively obtained.

Here, N is the total number of discrete frequency spectra, for example, N=256.

Then, the utterance amount ratio calculation unit 107 obtains the utterance amount ratio SR(τ) from the obtained two utterance amounts s _Tgt (τ) and s _Int (τ) by the following equation (8).

4A to 4C are graphs for explaining the utterance amount ratio SR(τ) when the target speaker and the disturbing speaker speak.
FIG. 4A is a graph showing an example of the time waveform of the observed analog signal acquired by the first microphone 101.
FIG. 4B is a graph showing an example of the time variation of the utterance amount between the target sound speaker and the disturbing sound speaker.
FIG. 4C is a graph showing an example of temporal variation of the utterance amount ratio SR(τ) obtained from the utterance amount of the target sound speaker and the utterance amount of the disturbing sound speaker.

As shown in FIG. 4C, in the case of a frame satisfying SR(τ)<0.3, there is a high possibility that only interfering sound is present, while in the frame satisfying SR(τ)>0.5, In the case, it can be seen that there is a high possibility that only the target sound is present.
When 0.3≦SR(τ)≦0.5, it can be considered that both the target sound and the disturbing sound are present.

Therefore, by controlling the masking strength according to the aspect of the observed analog signal by using the speech amount ratio SR(τ) obtained by the above equation (8), the target sound with high separation accuracy and less distortion can be obtained. Separation is possible. More specifically, for example, in a frame in which the utterance amount ratio SR(τ) is small, the masking filter coefficient is increased to strongly suppress the interfering sound and enhance the separation performance, thereby increasing the utterance amount ratio SR(τ). In the case of a large frame, it is possible to reduce the distortion of the target sound by reducing the numerical value of the masking filter coefficient.

Returning to FIG. 2, the gain calculation unit 108 uses the utterance amount ratio SR(τ) obtained by the above equation (8) to determine the constant in the mask coefficient b(ω,τ) of the above equation (5). A correction gain g(ω, τ) for correcting M is calculated by the following formula (9).

Here, G _Tgt , G _Int, and G _DT are predetermined modified gain constants, G _Tgt is a constant when there is a high possibility that the observed analog signal is only the target sound, and G _Int is the observed analog signal. Is a constant when there is a high possibility that only an interfering sound is present, and G _DT is a constant when there is a high possibility that both the target sound and the interfering sound are present in the observed analog signal. In the present embodiment, G _Tgt =1.5, G _DT =0.99, and G _Int =0.01 are preferable examples.

Then, when the possibility of the target sound is high, M in the above equation (5) is controlled to be large, in other words, the suppression amount of the mask is controlled to be small. However, the corrected M is limited to a value of 1 or less.
On the other hand, when the possibility of the disturbing sound is high, M in the above equation (5) is controlled to be further reduced, in other words, the amount of suppression of the disturbing sound is controlled to be further increased.
That is, the gain calculation unit 108 calculates a correction gain for correcting the mask coefficient so that the higher the utterance amount ratio is, the lower the masking strength is.

To calculate this modified gain, the calculation cost is low because only the utterance volume ratio obtained from a simple calculation of the power of the observed analog signal and the conditional expression based on the comparison of the utterance volume ratio are sufficient, and the mask coefficient is efficiently corrected. It is possible to

Further, K(ω) is a frequency correction coefficient represented by a positive number of 1 or less, and is set so that the value increases as the frequency increases, as shown in the following expression (10).

By performing the frequency correction with K(ω), the intensity of masking at high frequencies is relaxed, so that distortion of the target sound due to masking can be suppressed.

Note that the frequency correction coefficient of the equation (10) is corrected so that the value increases as the frequency increases, but the frequency correction coefficient of the equation (10) is not limited to such an example. , Can be changed as appropriate according to the characteristics of the observed analog signal. For example, when the acoustic signal targeted for sound source separation is speech, correction is performed to weaken the suppression of formant, which is an important frequency band component in speech, and to strengthen the suppression of other band components. Correction may be performed. As a result, the accuracy of mask control for voice is improved, so that the target sound can be efficiently separated.
Further, if the target of sound source separation is an abnormal sound of a machine, it is possible to efficiently separate the abnormal sound by changing the frequency correction coefficient of Expression (10) according to the frequency characteristic of the acoustic signal. ..

As a further effect of the correction by the frequency as described above, when environmental noise is mixed with the observed noise, masking to an acoustic signal other than the target sound or abnormal sound (for example, noise or music) As the influence of the noise is reduced, unpleasant artificial noise (musical tone) caused by unnecessary masking against environmental noise is reduced, and malfunction of the voice recognition device or abnormal sound monitoring device due to artificial noise is reduced, resulting in a hands-free call. It also has the side effect of reducing the unpleasant noise of time.

The constant value of the correction gain or the constant threshold value of the utterance amount ratio SR(τ) described above is not limited to the case of Expression (9), and is appropriately adjusted according to the mode of the target sound or the disturbing sound. can do. Further, the condition for determining the correction gain is not limited to three stages as in the equation (9), and may be set in more stages.

As shown in the following equation (11), the mask correction unit 109 applies the correction gain g obtained by the equation (9) to the mask coefficient b(ω, τ) obtained by the above equation (5). The correction is performed using (ω, τ) to obtain the time-frequency filter coefficient b _mod (ω, τ).

Returning to FIG. 1, the masking filter unit 110 uses the above formula ( ₁ ) as the first spectral component X ₁ (ω, τ) on the first microphone 101 side, as shown in the following formula (12). The time frequency filter coefficient b _mod (ω, τ) obtained in 11) is multiplied to calculate the spectral component Y(ω, τ). Then, the masking filter unit 110 sends the calculated spectral component Y(ω, τ) to the T/F inverse transform unit 111. The spectral component Y(ω, τ) separated here is also referred to as a target spectral component. The target spectrum component is a spectrum component including a target sound.

The T/F inverse transform unit 111 performs, for example, inverse fast Fourier transform on the spectral component Y(ω, τ) to calculate the output digital signal y(t). The T/F inverse conversion unit 111 gives the calculated output digital signal y(t) to the D/A conversion unit 112.

The D/A conversion unit 112 generates an output signal by converting the output digital signal y(t) into an analog signal. The generated output signal is output to an external device such as a voice recognition device, a hands-free communication device, or an abnormal sound monitoring device.

FIGS. 5A and 5B are graphs for explaining the effect in the first embodiment.
Similar to FIG. 4A, FIG. 5A is a graph showing an example of the time waveform of the observed analog signal acquired by the first microphone 101.
FIG. 5B is a graph showing an example of time variation of the output signal output from the D/A conversion unit 112.
As is clear from FIGS. 5A and 5B, it can be seen that the interfering sound is almost removed from the output signal and only the target sound is separated.

The hardware configuration of the sound source separation device 100 described above can be realized by a computer with a built-in CPU (Central Processing Unit), such as a tablet-type portable computer or a microcomputer for use in a device such as a car navigation system. Alternatively, the hardware configuration of the sound source separation apparatus 100 may be a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Integrated) integrated circuit (LGA) such as an FPGA (Field-Integrated Gate) integrated (LGA) integrated circuit (LGA). May be done.

FIG. 6 is a block diagram showing a hardware configuration example of the sound source separation device 100 configured by using an LSI such as DSP, ASIC or FPGA.

In the example of FIG. 6, the sound source separation device 100 includes a signal input/output unit 131, a signal processing circuit 132, a recording medium 133, and a signal path 134 such as a bus.
The signal input/output unit 131 is an interface circuit that realizes a connection function with the microphone circuit 140 and the external device 141. The microphone circuit 140 corresponds to the first microphone 101 and the second microphone 102, and for example, a device that captures acoustic vibration and converts it into an electric signal can be used.

Each function of the T/F conversion unit 104, the mask generation unit 105, the masking filter unit 110, and the T/F inverse conversion unit 111 shown in FIG. 1 can be realized by the signal processing circuit 132 and the recording medium 133. it can.
Further, the A/D conversion unit 103 and the D/A conversion unit 112 in FIG. 1 can be realized by the signal input/output unit 131.

The recording medium 133 is used to store various setting data of the signal processing circuit 132 and various data such as signal data. As the recording medium 133, for example, a volatile memory such as SDRAM (Synchronous Dynamic Random Access Memory), a non-volatile memory such as HDD (Hard Disk Drive) or SSD (Solid State Drive) can be used. The recording medium 133 can store the initial state of the sound source separation process, various setting data, constant data for control, and the like.

The output digital signal subjected to the sound source separation processing in the signal processing circuit 132 is sent from the signal input/output unit 131 to the external device 141. As the external device 141, for example, a voice recognition device, a hands-free call device, or It corresponds to an abnormal sound monitoring device.

FIG. 7 is a block diagram showing a hardware configuration example of the sound source separation device 100 configured by using a computing device such as a computer.
In the example of FIG. 7, the sound source separation device 100 includes a signal input/output unit 131, a processor 136 including a CPU 135, a memory 137, a recording medium 138, and a signal path 134 such as a bus.
The signal input/output unit 131 is an interface circuit that realizes a connection function with the microphone circuit 140 and the external device 141.

The memory 137 is a program memory that stores various programs for implementing sound source separation processing, a work memory used when the processor 136 performs data processing, and a ROM (Read Only) that is used as a memory that expands signal data. Memory) and RAM (Random Access Memory).

The functions of the T/F conversion unit 104, the mask generation unit 105, the masking filter unit 110, and the T/F inverse conversion unit 111 can be realized by the processor 136, the memory 137, and the recording medium 138.
Further, the A/D conversion unit 103 and the D/A conversion unit 112 can be realized by the signal input/output unit 131.

The recording medium 138 is used for accumulating various data such as various setting data and signal data of the processor 136. As the recording medium 138, for example, a volatile memory such as SDRAM or a non-volatile memory such as HDD or SSD can be used. It is possible to store programs including an OS (Operating System), various setting data, and various data such as audio signal data. The data in the memory 137 can be stored in the recording medium 138.

The processor 136 uses the memory 137 as a working memory, and operates according to the computer program read from the memory 137, whereby the T/F conversion unit 104, the mask generation unit 105, the masking filter unit 110, and the T/F inverse unit. It can function as the conversion unit 111.

The output signal generated by the sound source separation processing performed by the processor 136 is sent from the signal input/output unit 131 to the external device 141. Examples of the external device 141 include a voice recognition device, a hands-free call device, or It corresponds to an abnormal sound monitoring device.

The program executed by the processor 136 may be stored in a storage device inside a computer that executes the software program, or may be in a format distributed in a storage medium such as a CD-ROM. It is also possible to acquire the program from another computer through a wireless or wired network such as a LAN (Local Area Network). Such a program may be provided as a program product, for example.

Further, regarding the microphone circuit 140 and the external device 141, various data may be transmitted and received as digital signals through a wireless or wired network without passing through conversion of analog signals and digital signals.

Further, the program executed by the processor 136 is a program executed by the external device 141, for example, a program executed by the computer to cause the computer to function as a voice recognition device, a hands-free communication device, or an abnormal sound monitoring device. It is also possible that they are combined with each other and run on the same computer, or they can be distributed and run on multiple computers.

Note that the external device 141 may include the sound source separation device 100. That is, the voice recognition device, the hands-free communication device, or the abnormal sound monitoring device may be configured to include the sound source separation device 100.

Next, the operation of the sound source separation device 100 according to the first embodiment will be described.
FIG. 8 is a flowchart showing the operation of the sound source separation device 100.
First, the A/D conversion unit 103 sets each of the first observed analog signal and the second observed analog signal input from each of the first microphone 101 and the second microphone 102 in a predetermined frame. The first observed digital signal x ₁ (t) and the second observed digital signal x ₂ (t) are generated by taking in at intervals and A/D converting each, and the T/F conversion unit 104 generates them. (S10).
Then, the output from the A/D conversion unit 103 is repeatedly performed when the sample number t is smaller than the predetermined value T (No in S11).

In step S12, the T/F conversion unit 104 performs, for example, fast Fourier transform of 512 points on each of the first observed digital signal x ₁ (t) and the second observed digital signal x ₂ (t). Then, the first spectral component X ₁ (ω, τ) and the second spectral component X ₂ (ω, τ) are calculated. Then, the T/F conversion unit 104 gives the first spectral component X ₁ (ω, τ) and the second spectral component X ₂ (ω, τ) to the mask generation unit 105, and the first spectral component X ₁ (Ω, τ) is given to the masking filter unit 110.

The mask generation unit 105 uses the first frequency component X ₁ (ω, τ) and the second spectrum component X ₂ (ω, τ) to mask the time-frequency filter coefficient b _mod ( ω, τ) is calculated (S13). Hereinafter, detailed processing in step S13 will be described with reference to steps S13A to S13D.

In step S13A, the mask coefficient calculation unit 106 determines the cross spectrum D(ω, τ) from the cross-correlation function of the first spectrum component X ₁ (ω, τ) and the second spectrum component X ₂ (ω, τ). And the mask coefficient b(ω, τ) is calculated based on the obtained cross spectrum D(ω, τ). The mask coefficient calculation unit 106 gives the cross spectrum D(ω, τ) to the utterance amount ratio calculation unit 107, and gives the mask coefficient b(ω, τ) to the mask correction unit 109. Then, the process proceeds to step S13B.

In step S13B, the utterance amount ratio calculation unit 107 determines the target sound from the first spectrum component X ₁ (ω, τ), the second spectrum component X ₂ (ω, τ) and the cross spectrum D(ω, τ). An utterance amount ratio SR(τ) which is a ratio between the utterance amount of the speaker and the utterance amount of the interfering sound speaker is calculated. The utterance amount ratio calculation unit 107 gives the utterance amount ratio SR(τ) to the gain calculation unit 108. Then, the process proceeds to step S13C.

In step S13C, the gain calculation unit 108 calculates the correction gain g(ω, τ) for correcting the mask coefficient b(ω, τ) using the utterance amount ratio SR(τ). The gain calculation unit 108 gives the correction gain g(ω, τ) to the mask correction unit 109. Then, the process proceeds to step S13D.

In step S13D, the mask correction unit 109 corrects the mask coefficient b(ω, τ) using the correction gain g(ω, τ) to obtain the time frequency filter coefficient b _mod (ω, τ). Then, the mask correction unit 109 gives the time-frequency filter coefficient b _mod (ω, τ) to the masking filter unit 110.

The masking filter unit 110 multiplies the first spectral component X ₁ (ω, τ) by the time-frequency filter coefficient b _mod (ω, τ), and the spectral component Y(ω, τ) of the output digital signal y(t). ) Is calculated (S14). Then, the masking filter unit 110 provides the spectral component Y(ω, τ) to the T/F inverse transform unit 111.

The T/F inverse transform unit 111 transforms the spectral component Y(ω, τ) into an output digital signal y(t) in the time domain by performing an inverse fast Fourier transform on the spectral component Y(ω, τ). Yes (S15).

The D/A conversion unit 112 converts the output digital signal y(t) into an output signal which is an analog signal by D/A conversion, and outputs the output signal to the outside (S16).
Then, the output from the D/A conversion unit 112 is repeatedly performed when the sample number t is smaller than the predetermined value T (Yes in S17).

Next, if the sound source separation process is to be continued (Yes in S18), the process returns to step S10. On the other hand, when the sound source separation processing is not continued (No in S18), the sound source separation processing ends.

As described above, the sound source separation device 100 according to the first embodiment can create a masking filter with high separation performance at low calculation cost. Therefore, the target sound can be accurately acquired, and it is possible to provide a high-accuracy voice recognition device, a high-quality hands-free communication device, and an abnormal sound monitoring device with high detection accuracy.

Embodiment 2.
Although the first embodiment exemplifies the configuration based on the voice, the second embodiment will be described as an embodiment that can be applied to the case where there is noise other than the voice that becomes the disturbing sound.

FIG. 9 is a block diagram schematically showing a configuration of an information processing system 250 including the sound source separation device 200 according to the second embodiment. The information processing system 250 shown here is an example of a car navigation system, and shows a case where a speaker seated in a driver seat and a speaker seated in a passenger seat speak in a moving automobile. In the second embodiment, a speaker seated in the driver's seat will be referred to as a target sound speaker, and a speaker seated in the passenger seat will be referred to as an interfering sound speaker.

As shown in FIG. 9, the information processing system 250 includes a first microphone 101, a second microphone 102, a sound source separation device 200, and an external device 141.
The first microphone 101 and the second microphone 102 in the second embodiment are the same as the first microphone 101 and the second microphone 102 in the first embodiment. The external device 141 is similar to the external device 141 described with reference to FIG. 6 or 7.

As the input in the second embodiment, in addition to the voices of the target sound speaker and the disturbing sound speaker captured through the first microphone 101 and the second microphone 102, noise such as vehicle running noise, and during hands-free communication It is the received voice of the far-end speaker transmitted from the speaker, the guide voice transmitted by the car navigation, or the acoustic echo around which the music of the car audio or the like goes around. Voices other than the voices of the target speaker and the disturbing speaker are noise. In addition, the noise signal is a noise signal. Then, in the second embodiment, the sound arriving from a direction not included in the first range including the first direction in which the target sound arrives and a second range including the second direction in which the disturbing sound arrives The influence of noise is excluded by excluding the spectrum component and calculating the utterance amount ratio.

The external device 141 is, for example, a voice recognition device, a hands-free communication device, or an abnormal sound monitoring device, as described above. The external device 141 performs, for example, a voice recognition process, a hands-free call process, or an abnormal sound detection process, and obtains an output result according to each process.

The sound source separation device 200 includes an A/D conversion unit 103, a T/F conversion unit 104, a mask generation unit 205, a masking filter unit 110, and a T/F inverse conversion unit 111.
The A/D conversion unit 103, the T/F conversion unit 104, the masking filter unit 110, and the T/F inverse conversion unit 111 of the sound source separation device 200 according to the second embodiment are A of the sound source separation device 100 of the first embodiment. It is the same as the /D conversion unit 103, the T/F conversion unit 104, the masking filter unit 110, and the T/F inverse conversion unit 111.
However, in the sound source separation device 200 according to the second embodiment, the output digital signal y(t) generated by the T/F inverse conversion unit 111 is given to the external device 141.

As shown in FIG. 2, the mask generation unit 205 includes a mask coefficient calculation unit 106, an utterance amount ratio calculation unit 207, a gain calculation unit 108, and a mask correction unit 109.
The mask coefficient calculation unit 106, the gain calculation unit 108 and the mask correction unit 109 of the mask generation unit 205 according to the second embodiment are the same as the mask coefficient calculation unit 106, the gain calculation unit 108 and the mask correction of the mask generation unit 105 according to the first embodiment. It is similar to the unit 109.

The utterance amount ratio calculation unit 207 excludes the disturbing sound signal from the calculation of the utterance amount ratio SR(τ) by using the equation (13) obtained by modifying the equation (7) described in the first embodiment.
In the first embodiment, the arrival direction of the target sound is determined by the sign of the imaginary part Q(ω, τ) of the cross spectrum D(ω, τ) of the equation (1). In the conditional expression, by combining the time difference δ(ω, τ) of the first channel Ch1 and the second channel Ch2, which means the angle of the arrival direction, the target speaker and the interfering sound are calculated from the calculation of the utterance amount. The influence of noise other than the speaker can be excluded.

Here, δ _θDT and δ _θDN are threshold values of the time difference of the observed analog signal for excluding from the calculation of the utterance amount, and are predetermined constants obtained by converting the arrival direction angle into the time difference.
δ _{θ DT} is _{assumed when} the arrival time difference of observed analog signals is extremely small and it is difficult to determine whether the arrival direction is the target sound direction or the disturbing sound direction, or when noise is coming from the front direction. Is a threshold value for excluding from the calculation of the utterance amount.
δ _{θ DN} is highly likely to deviate from the expected arrival directions of the target sound and the interfering sound, in other words, the observed analog signal is directional noise such as wind noise mixed from a window, or from the speaker. This is a threshold value for excluding such a case from the calculation of the utterance amount when the possibility of released music or the like is high.

FIG. 10 is a schematic diagram illustrating an example of a method for excluding the influence of noise other than the target sound and the disturbing sound in Expression (13).
In the example of FIG. 10, the exclusion range is described based on the first channel Ch1.
As shown in FIG. 10, by setting an exclusion range in the calculation of the utterance amount, the influence of noise other than the target sound and the interfering sound can be excluded, so that the calculation accuracy of the utterance amount ratio is improved and the quality of the utterance amount is further improved. It is possible to configure a high sound source separation device.

Since the sound source separation device 200 according to the second embodiment is configured as described above, it is possible to create a masking filter with high separation performance at low calculation cost even under various noise conditions. Therefore, since the target sound can be accurately acquired even under the noise in the vehicle, a high-accuracy voice recognition device, a high-quality hands-free communication device, or an abnormal sound monitoring device for detecting an abnormal sound in the vehicle. Can be provided.

Embodiment 3.
In the first and second embodiments, only the current frame information is used for calculating the utterance amount ratio, but the embodiment is not limited to such an example, and the past frame information is used for the calculation. It is also possible.

As shown in FIG. 1, the sound source separation device 300 according to the third embodiment includes an A/D conversion unit 103, a T/F conversion unit 104, a mask generation unit 305, a masking filter unit 110, and The T/F inverse conversion unit 111 and the D/A conversion unit 112 are provided.
The A/D conversion unit 103, the T/F conversion unit 104, the masking filter unit 110, the T/F inverse conversion unit 111, and the D/A conversion unit 112 of the sound source separation device 300 according to the third embodiment are the same as those of the first embodiment. This is the same as the A/D conversion unit 103, the T/F conversion unit 104, the masking filter unit 110, the T/F inverse conversion unit 111, and the D/A conversion unit 112 of the sound source separation device 100 according to.

As shown in FIG. 2, the mask generation unit 305 according to the third embodiment includes a mask coefficient calculation unit 106, a speech amount ratio calculation unit 307, a gain calculation unit 108, and a mask correction unit 109.
The mask coefficient calculation unit 106, the gain calculation unit 108, and the mask correction unit 109 of the mask generation unit 305 according to the third embodiment are the same as the mask coefficient calculation unit 106, the gain calculation unit 108, and the mask correction unit of the mask generation unit 105 according to the first embodiment. It is similar to the unit 109.

The utterance amount ratio calculation unit 307 calculates the utterance amount ratio SR(τ) by using the above equation (8), and further calculates the calculated SR(τ) by using the following equation (14). Smoothing is performed with the utterance amount ratio SR(τ-1) before the frame.

Here, α is a smoothing coefficient, and α=0.9 is a suitable example in the third embodiment.

In this way, when calculating the speech volume ratio, the speech volume ratio calculated in the past is used to smooth the last calculated speech volume ratio, so that even if noise is mixed in the observed analog signal, it will be stable. It is possible to calculate the utterance amount ratio by using the above method, and it is possible to perform sound source separation with higher accuracy.

Further, in the second embodiment, the utterance amount ratio calculation unit 207 calculates the utterance amount of each signal by using the equation (13). However, as a modification, the utterance amount ratio calculation unit 207 performs the calculation. To a predetermined frame section, in other words, by calculating the integral value of the power spectrum of a predetermined frame section, the occupation ratio of the target sound and the disturbing sound in the predetermined frame section, specifically, , It is possible to analyze which one is speaking long or which one is loud. Therefore, it is possible to determine which voice is dominant in the double talk of the target sound and the disturbing sound, and more accurate sound source separation can be performed.

In the second embodiment described above, the case where the information processing system 250 is an example of the car navigation system has been described, but the second embodiment is not limited to this. For example, the information processing system 250 includes a remote voice recognition system such as a smart speaker or a TV installed in a general home or an office, a voice call system of a TV conference system, a voice recognition dialogue system of a robot, or an abnormal sound of a factory. It can also be applied to monitoring systems. Even in such a case, the effects described in the second embodiment are similarly exerted on noise or acoustic echo generated in these acoustic environments.

Also, in the first to third embodiments described above, the frequency bandwidth of the input signal is 16 kHz, but the first to third embodiments are not limited to such an example. For example, the first to third embodiments can be applied to an acoustic signal in a wider band such as 24 kHz.

In addition to the above, in the first to third embodiments, it is possible to modify any constituent element or omit any constituent element.

As described above, since the sound source separation devices 100 to 300 according to the first to third embodiments can perform high quality sound source separation at low calculation cost, any one of the voice recognition system, the voice communication system, and the abnormal sound monitoring system can be used. Can be introduced to. As a result, it is possible to improve the recognition rate of a remote voice recognition system such as a car navigation system or a television, and to improve the quality of a hands-free call system such as a mobile phone or an intercom system, a TV conference system or an abnormal sound monitoring system.

100, 200, 300 sound source separation device, 101 first microphone, 102 second microphone, 103 A/D conversion unit, 104 T/F conversion unit, 105, 205, 305 mask generation unit, 106 mask coefficient calculation unit, 107, 207, 307 utterance volume ratio calculation unit, 108 gain calculation unit, 109 mask correction unit, 110 masking filter unit, 111 T/F inverse conversion unit, 112 D/A conversion unit, 250 information processing system.

Claims (6)

1. A first observation analog signal generated by a first microphone based on an observation sound including a target sound coming from a first direction, and a second observation analog signal generated by a second microphone based on the observation sound. An analog that receives an observation analog signal and converts each of the first observation analog signal and the second observation analog signal into a digital signal to generate a first observation digital signal and a second observation digital signal / Digital converter,
   A time/frequency conversion unit that generates a first spectral component and a second spectral component by converting each of the first observed digital signal and the second observed digital signal into a frequency domain signal,
   By using a cross-correlation function of the first spectral component and the second spectral component, a time difference between a time when the observed sound reaches the first microphone and a time when the observed sound reaches the second microphone. A mask generation unit that calculates a filtering coefficient for masking a spectral component of a sound coming from a direction different from the first direction,
   A masking filter unit that separates spectral components by masking the first spectral components using the filtering coefficient;
   An information processing apparatus, comprising: a time/frequency inverse conversion unit that generates the output digital signal by converting the separated spectral component into a signal in the time domain.
2. The observation sound includes an interfering sound coming from a second direction different from the first direction,
   The mask generation unit,
   Using the cross-correlation function of the first spectral component and the second spectral component, a first time of arrival of the target sound at the first microphone and a second time of arrival of the target sound at the second microphone. And a second time difference between the time when the disturbing sound arrives at the first microphone and the time when the disturbing sound arrives at the second microphone, including the first direction of the observation sound. A sound coming from the first range is distinguished from a sound coming from the second range that includes the second direction and does not overlap the first range. A mask coefficient calculating unit for calculating a mask coefficient for separating the spectrum component from the spectrum component of the sound coming from the second range;
   An utterance amount ratio calculator that calculates a ratio of the amount of the spectrum component of the sound coming from the first range to the amount of the spectrum component of the sound coming from the second range among the first spectrum components; ,
   A gain calculation unit that calculates a correction gain for correcting the mask coefficient so that the higher the ratio is, the lower the intensity at which the masking is performed,
   The information processing apparatus according to claim 1, further comprising a mask correction unit that calculates the filtering coefficient by correcting the mask coefficient with the correction gain.
3. The observation sound includes an interfering sound coming from a second direction different from the first direction,
   The mask generation unit,
   Using the cross-correlation function of the first spectral component and the second spectral component, a first time of arrival of the target sound at the first microphone and a second time of arrival of the target sound at the second microphone. And a second time difference between the time when the disturbing sound arrives at the first microphone and the time when the disturbing sound arrives at the second microphone, including the first direction of the observation sound. A sound coming from the first range is distinguished from a sound coming from the second range that includes the second direction and does not overlap the first range. A mask coefficient calculating unit for calculating a mask coefficient for separating the spectrum component from the spectrum component of the sound coming from the second range;
   Of the first spectral component, the ratio of the amount of the spectral component of the sound arriving from the first range to the amount of the spectral component of the sound arriving from the second range is calculated as a function of time. Together with the utterance amount ratio calculation unit that smoothes the ratio calculated last using the ratio calculated in the past,
   A gain calculation unit that calculates a correction gain for correcting the mask coefficient, such that the higher the smoothed ratio, the lower the intensity with which the masking is performed,
   The information processing apparatus according to claim 1, further comprising a mask correction unit that calculates the filtering coefficient by correcting the mask coefficient with the correction gain.
4. The utterance amount ratio calculation unit calculates the ratio by excluding a spectrum component of a sound coming from a direction that is not included in the first range and the second range. Alternatively, the information processing device according to item 3.
5. Computer,
   A first observation analog signal generated by a first microphone based on an observation sound including a target sound coming from a first direction, and a second observation analog signal generated by a second microphone based on the observation sound. An analog that receives an observation analog signal and converts each of the first observation analog signal and the second observation analog signal into a digital signal to generate a first observation digital signal and a second observation digital signal / Digital converter,
   A time/frequency conversion unit that generates a first spectral component and a second spectral component by converting each of the first observed digital signal and the second observed digital signal into a frequency domain signal,
   By using a cross-correlation function of the first spectral component and the second spectral component, a time difference between a time when the observed sound reaches the first microphone and a time when the observed sound reaches the second microphone. A mask generator for calculating a filtering coefficient for masking a spectral component of a sound coming from a direction different from the first direction,
   A masking filter unit that separates spectral components by performing masking on the first spectral components using the filtering coefficient; and
   A program, which functions as a time/frequency inverse conversion unit that generates an output digital signal by converting the separated spectral components into a time domain signal.
6. A first observation analog signal generated by a first microphone based on an observation sound including a target sound coming from a first direction, and a second observation analog signal generated by a second microphone based on the observation sound. By receiving the observation analog signal and converting each of the first observation analog signal and the second observation analog signal into a digital signal, a first observation digital signal and a second observation digital signal are generated,
   By converting each of the first observed digital signal and the second observed digital signal into a frequency domain signal, a first spectral component and a second spectral component are generated,
   By using a cross-correlation function of the first spectral component and the second spectral component, a time difference between a time when the observed sound reaches the first microphone and a time when the observed sound reaches the second microphone. , Calculating a filtering coefficient for masking a spectral component of a sound coming from a direction different from the first direction,
   By masking the first spectral component using the filtering coefficient, the spectral component is separated,
   An information processing method comprising: generating an output digital signal by converting the separated spectral component into a signal in a time domain.

Images