JP2003140671A - Separating device for mixed sound - Google Patents

Abstract

PROBLEM TO BE SOLVED: To separate a desired sound from a mixed input signal in which unsteady noise and a plurality of sounds are superposed one over another. SOLUTION: A sound separating device as an embodiment of the invention has a frequency analyzing means which takes a frequency analysis of the mixed input signal generated by mixing sound signals generated by different sound sources with a target signal to calculates a spectrum and an amplitude maximum point at each time, a feature extracting means which has a narrow-band layer analyzing narrow-band feature parameters by using the spectrum and amplitude maximum point and one or more wide-band layers analyzing wide- band feature parameters by using the feature parameters extracted by the narrow-band layer and extracts feature parameters associated with the target signal, and a signal composing means which composes the target signal according to the extracted featured parameters.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a signal separating apparatus for separating a target signal from a mixed input signal, and more particularly to a desired sound from a mixed input signal on which non-stationary noise or a plurality of sounds are superposed. An apparatus for separating signals.

[0002]

2. Description of the Related Art Heretofore, a spectral subtraction method has been known as a method for separating a required audio signal from a mixed input signal input to a sensor or the like. With this technique,
The target signal is separated by subtracting the noise spectrum from the spectrum of the mixed input signal containing the noise and the target signal.

Specifically, for example, from the spectrum of a mixed input signal containing noise and a target signal, a spectrum obtained from a section in which it is clear that the target signal does not exist and contains only noise is called the noise spectrum. Assuming and subtracting this from the spectrum of the mixed input signal in the section including the target signal, the spectrum of the target signal is separated.
As another example, prepare a microphone that collects only background noise separately from a microphone that collects the mixed input signal, and subtract the spectrum obtained from the latter from the spectrum of the former to obtain the target signal from the mixed input signal. To separate.

As another method, there is a method of extracting only a target signal by utilizing a harmonic structure which is a structural feature of a vowel of a voice or a musical instrument sound. As an example, only a signal having a harmonic structure can be extracted by applying a comb filter having a spectral pass characteristic corresponding to the fundamental frequency of a vowel and its harmonics to a mixed input signal.

As still another method, in the ft map in which the frequency spectra of the mixed input signals are arranged in time series, the maximum amplitude point in the frequency direction is scanned and extracted, and this is extracted as a candidate point of a point that should constitute a frequency component. However, a method is known in which the frequency components of the target signal are extracted by sequentially connecting these maximum points in the time direction. For example,
The purpose is to compare the amplitude maximum points at a certain time on the ft map with the amplitude maximum points at the next time, and connect the maximum points that show continuity in frequency, power, sound source direction, etc. sequentially in the time direction. Reproduce the signal.

Several methods are known in which a plurality of signal separation methods are combined. Japanese Patent Laid-Open No. 9-257559 discloses a method of collectively extracting amplitude maximum points using local structure information. In this method, the maximum amplitude point of the spectrum is set as a frequency component candidate point, and it is determined whether or not a frequency component is formed from the relationship between each frequency component candidate point and the neighboring points located in the neighborhood. For the frequency component candidate points, the continuity with respect to time, frequency, and power value is determined, points having continuity are connected, and the frequency component is extracted.

[0007]

However, each of the sound separation methods described above has the following problems.

First, in the spectral subtraction method,
Only stationary noise can be separated, and it is not possible to separate one audio signal from an input signal on which multiple audio signals are superimposed, or to separate sudden noise such as door opening / closing sound. .

The method using the comb filter is effective when the audio signal has a stationary fundamental frequency. However, since the fundamental frequency of the audio signal generally changes dynamically, there are few practical cases where this method is effective.

In the method of extracting the frequency component at the maximum amplitude point, it is difficult to uniquely determine the continuity of the maximum amplitude point in the time direction. In particular, when the S / N ratio is high, the maximum points that are candidates increase and the polysemy becomes high. Further, when the energy of another signal exists near the frequency component of the target signal and the amplitude maximum points are close to each other, those signals cannot be distinguished. When a method such as discrete Fourier transform is used to obtain the maximum amplitude point, the fundamental frequency of the acoustic component contained in the input signal and the resolution of the discrete Fourier transform are different, or the acoustic component contained in the input signal is modulated. However, if the amplitude maximum points of different sound sources are close to each other, the frequency of the amplitude maximum point cannot be accurately determined, so that actual frequency component extraction becomes difficult.

Therefore, it is an object of the present invention to provide a sound separation method capable of separating non-stationary noise and a plurality of superimposed voice signals. In addition, the target signal can be separated even when the fundamental frequency and amplitude of the target signal dynamically change,
It is another object of the present invention to provide a sound separation device that can effectively separate a target signal even when the frequency components of the target signal and noise are close to each other (that is, when the S / N ratio is high).

[0012]

The sound separating device of the present invention comprises:
A sound separation device for separating the target signal from a mixed input signal in which acoustic signals emitted from different sound sources and a target signal are mixed, wherein the spectrum and amplitude maximum points at each time are analyzed by frequency analysis of the mixed input signal. A frequency analysis means for calculating, a narrow layer for analyzing a narrow characteristic parameter using the spectrum and the amplitude maximum point, and a wide range characteristic parameter using the characteristic parameter extracted by the narrow layer. And a signal synthesizing unit for synthesizing the target signal based on the extracted feature parameter, the feature extracting unit including one or more wide-area layers to be analyzed, the feature extracting unit extracting the feature parameter associated with the target signal. Composed.

According to the present invention, the feature extraction means handles both the narrow range feature parameter and the wide range feature parameter so that the accuracy of separating the target signal does not depend on the accuracy of extracting the particular feature parameter. . The characteristic parameters to be extracted include frequency / amplitude values of frequency component candidate points included in the input signal and their time series data such as changes, harmonics, pitch continuity, and intonation, as well as onset / offset. , Sound source direction, etc. are also included.
Further, the number of layers provided in the feature extraction means can be variable according to the type of the feature parameter to be extracted.

In another aspect of the present invention, the narrow area layer and the wide area layer mutually supply the characteristic parameters analyzed in the respective layers, and the characteristic parameters of the respective layers based on the supplied characteristic parameters. Is configured to update.

According to this aspect, the feature parameters analyzed in each layer of the feature extracting means are mutually supplied, so that the feature parameters can be matched with each other, and thus the feature parameter extraction accuracy can be improved. .

[0016] In still another form of the present invention, the narrow layer is an instantaneous coding layer for calculating a frequency and a change thereof and an amplitude and a change thereof of the frequency component candidate point.

According to this aspect, it is possible to follow a gradual change in the amplitude and frequency of the same sound source signal by utilizing the instantaneous time change information.

[0018] In still another mode of the present invention, the wide-area layer groups frequency component candidate points having a harmonic structure based on the frequencies of the frequency component candidate points and changes thereof, and determines the fundamental frequency of the harmonic structure and its It includes a harmonic calculation layer for calculating changes and a pitch continuity calculation layer for calculating signal continuity from the fundamental frequency and changes thereof at a plurality of times.

As an example of the change to be calculated, there is a time change rate, but other than this, a second derivative or the like may be used as long as it can catch the change of the frequency component candidate point.

According to this aspect, the target signal in the non-stationary noise can be separated by utilizing its consistency, and the amplitude of the fundamental frequency and the change of the frequency are tracked according to the global characteristic parameter. be able to.

In one embodiment of the present invention, the hierarchy is composed of one or a plurality of calculation elements that respectively perform similar processing to calculate a characteristic parameter, and the calculation elements are connected to an upper hierarchy and a lower hierarchy. It is configured to mutually supply each calculation element included in the connected hierarchy and the calculated characteristic parameter.

According to this aspect, the independence of the features to be extracted is increased, and flexible updating of the feature parameters is realized.
Here, the calculation element is an information processing element that is generated one-to-one corresponding to the characteristic parameter, performs the same processing, and has a function of mutually supplying the characteristic parameter to another calculation element. It does not mean a simple element.

[0023] In still another form of the present invention, the calculation element has a first degree indicating a degree of matching between the characteristic parameter supplied from the calculation element included in a higher-level connected layer and the calculated characteristic parameter. And calculating a second consistency function indicating a degree of matching between the feature parameter supplied from the calculation element included in the lower connecting hierarchy and the calculated feature parameter, The feature parameters are configured to be updated so as to maximize the validity index represented by the product of the consistency functions of the.

According to this aspect, it is possible to mutually refer to the characteristic parameters among the calculation elements and improve the consistency between the characteristic parameters.

In yet another form of the present invention, the validity index is supplied to a computing element included in the lower hierarchy.

According to this aspect, the binding force of the upper hierarchy with respect to the calculation element is increased to shorten the calculation convergence time,
On the contrary, the restraint force can be weakened to reduce the influence. As a result, it is possible to retain many feature parameters while the number of calculations is small, and tighten the survival conditions as the layers become more consistent and increase the precision of the feature parameters. Like Furthermore, a threshold value is calculated every time the validity index of the upper layer is updated, and when the value of the validity index falls below the threshold value, unnecessary feature parameters can be removed early by eliminating the calculation element. Further, when the validity index is larger than a predetermined value, it is possible to flexibly update data, such as creating a new calculation element in the layer one level below.

[0027]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

<First Embodiment> FIG. 1 is a block diagram showing the overall structure of a sound separation device 100 according to a first embodiment of the present invention. The sound separation device 100 includes a signal input unit 101,
Frequency analysis unit 102, feature extraction unit 103, and signal synthesis unit 104
Is included. The sound separation device 100 analyzes various features included in a mixed input signal on which noise and signals emitted from various sound sources are superimposed, sorts out the consistency between the features, and separates a target signal. The main part of the sound separation device 100 is realized by, for example, executing software including the features of the present invention on a computer or workstation equipped with an input / output device, a CPU, a memory, an external storage device, etc. The unit can also be realized by hardware. Based on this, FIG. 1 expresses the configuration with functional blocks.

A mixed input signal to be the target of sound separation is input to the signal input unit 101. The signal input unit 101 is specifically an acoustic input terminal such as a microphone, and directly collects mixed input signals. In this case, the sound input terminal is 1
The number is not limited to two, and two or more can be used. When there are two or more sound input terminals, a mode in which the sound source direction is used as a feature of the target signal can be implemented as described later. In another embodiment, the mixed input signal is a sound signal file prepared in advance, and in this case, the signal input unit 101 performs a process of taking the sound signal file.

The frequency analysis unit 102 subjects the signal input to the signal input unit 101 to A / D conversion, frequency-analyzes the digitized signal at appropriate time intervals, and obtains a frequency spectrum at each time. Create an ft map in which spectra are arranged in chronological order. The frequency analysis is performed by using a known Fourier transform, wavelet transform, band division by a filter bank, or the like. Further, the maximum amplitude point of the obtained spectrum is obtained.

The feature extraction unit 103 is separated from the frequency analysis unit 102 by ft.
The map is received, the characteristic parameters are extracted from each spectrum and its maximum amplitude point, and the characteristic parameters of the target signal are estimated from them.

The signal synthesizer 104 reconstructs the waveform of the target signal from the estimated characteristic parameters. Specifically, the waveform of the target signal is reconstructed using a template waveform such as a sine wave from various characteristic parameters estimated at each time.

The target signal thus extracted and reconstructed from the mixed audio signal is sent to a speaker (not shown) for reproduction, or sent to a display (not shown) to display the spectrum of the target signal. .

<Structure of Feature Extraction Unit> In the mixed input signal,
It contains various characteristic parameters of the signal emitted from each sound source that constitutes the input signal. These characteristic parameters can be classified into several types. For example, there are those that appear broadly in the time-frequency domain such as pitch, modulation, and intonation, those that appear narrowly like sound source position information, and those that appear instantaneously such as the maximum point of the frequency spectrum and its instantaneous change. It can be represented hierarchically. Further, the characteristic parameters of the signals emitted from the same sound source should be related to each other. In the present invention, paying attention to this, the feature extraction unit has a hierarchical structure and is configured to process different feature parameters in each hierarchy.
The characteristic parameters in each layer are updated so that the layers are most consistent.

FIG. 2 shows a sound separation device 100 in which the feature extraction unit 103 has a three-layer hierarchical structure. As shown in the figure, there are a local feature extraction layer 106 and an intermediate feature extraction layer 1 in the hierarchy.
07, a global feature extraction layer 108 is included. It should be noted that the hierarchical structure can be provided in four layers or more, or in two layers, depending on the type of the characteristic parameter to be extracted. When the number of layers is four or more, the number of intermediate feature extraction layers increases by a corresponding number. Further, some layers may be arranged in parallel, which will be described later in connection with the second and third embodiments.

Each layer of the feature extraction unit 103 analyzes different feature parameters. The local feature extraction layer 106 and the intermediate feature extraction layer 107, and the intermediate feature extraction layer 107 and the global feature extraction layer 108 are logically connected to each other. The ft map created by the frequency analysis unit 102 is passed to the local feature extraction layer 106 in the feature extraction unit 103.

Each layer first calculates the characteristic parameter extracted by itself based on the characteristic parameter passed from the lower layer. The calculated characteristic parameters are passed to the upper and lower layers. The characteristic parameters of the upper and lower layers are constraint conditions, and the characteristic parameters are updated so that the characteristic parameters of the upper and lower layers connected to each other and the characteristic parameters of the own layer are matched.

When the feature parameters in each layer and the feature parameters in the upper and lower layers are best matched, the feature extraction unit 103 determines that the optimum solution is obtained, and analyzes the feature parameters that can reconstruct the target signal. Output as a result.

FIG. 3 is a block diagram showing an example of a combination of feature parameters extracted by each layer in the feature extraction unit 103 and a flow of processing in each layer. In this embodiment, the local feature extraction layer 106 performs instantaneous encoding, the intermediate feature extraction layer 107 performs harmonic calculation, and the global feature extraction layer 108.
Respectively, the pitch continuity is calculated.

The instantaneous coding layer (local feature extraction layer) is f-
Based on the t map, the frequencies and amplitudes of the frequency component candidate points included in the input signal and their time change rates are calculated. This calculation can be realized by the instantaneous encoding method disclosed in Japanese Patent Application No. 2001-16055, for example. In particular,
After subjecting the input signal to A / D conversion and multiplying by the window function, discrete Fourier transform is executed to calculate the spectrum of the input signal.
Further, the power spectrum of the input signal is calculated, and a single signal or a plurality of unit signals corresponding to the amplitude maximum points are generated.
Each unit signal has a frequency, an amplitude, and their rate of change over time as parameters. Each unit signal is A / D converted and the spectrum is calculated by the discrete Fourier transform. If there are multiple unit signals, they are added together. The squared error in the amplitude / phase space of the spectrum of the input signal and the spectrum of the unit signal sum is calculated, and finally the input signal is changed by changing the number of unit signals and the parameters of each unit signal to minimize the error. It is possible to obtain the frequency and amplitude of the included frequency component candidate points and the time change rate thereof.

The characteristic parameter of the harmonic structure calculated in the harmonic calculation layer is input to the instantaneous coding layer, and the consistency with the characteristic parameter of the instantaneous information obtained in its own layer is verified.

The harmonic calculation layer (intermediate feature extraction layer) calculates from the frequency calculated in the instantaneous coding layer and its rate of change over time,
Calculate the harmonic nature of the signal at each time. That is, a frequency component candidate point group having a frequency that is an integer multiple (n × f ₀ ) of a certain fundamental frequency f ₀ and a change rate that is an integer multiple (n × df ₀ ) of a certain change rate df ₀ is set as one harmonic structure. Group as sound frequency components. The output of the harmonic calculation layer is the fundamental frequency of the harmonic structure sound and its rate of change. The fundamental frequency information at each time calculated by the pitch continuity calculation layer is input to the harmonic calculation layer, and the consistency with the characteristic parameters obtained by the own layer is verified.

Since the harmonic calculation layer selects the harmonic structured sound at each time, it is not necessary to store the fundamental frequency in advance like a comb filter. Further, even if the fundamental frequency fluctuates, since the harmonic structure exists at each time, the harmonic structure sound can be detected.

Pitch Continuity Calculation Layer (Global Feature Extraction Layer)
Calculates a temporally continuous pitch flow from the fundamental frequency obtained by the harmonic calculation layer and its rate of change over time. For example, if the pitch frequency at a certain time and its change rate are obtained, the rough value of the pitch at the time before and after that can be predicted. Those in which the error between the predicted pitch and the pitch actually existing at that time are within a certain range are grouped as a pitch flow of a block. The output of the pitch continuity calculation layer is the pitch flow and the amplitudes of the frequency component candidate points forming the flow.

Next, the flow of processing in each layer will be described.

First, instantaneous coding calculation is performed on the ft map obtained from the frequency analysis unit to calculate the frequency f of the frequency component candidate points included in the input signal as a characteristic parameter and its time change rate df ( 301). The frequency f and the time change rate df are sent to the harmonic calculation layer.

The harmonic calculation layer calculates the relationship between the frequencies f corresponding to the frequency component candidate points at each time and the rate of change d with time.
By examining the relationship between f, the frequency component candidate point groups having a harmonic relationship, that is, having a harmonic structure are grouped (hereinafter referred to as “harmonic group”), and the fundamental frequency f ₀ of each group is used as a characteristic parameter. And its change rate df ₀ (30
2). At this stage, there can be multiple harmonic groups.

The fundamental frequency f ₀ of the harmonic group and its rate of change df ₀ calculated at each time are passed to the pitch continuity calculation layer. The pitch continuity calculation layer compares the fundamental frequency f ₀ and the rate of change df ₀ at each time over a certain period of time, and estimates a pitch continuity curve that can smoothly connect these (303). The characteristic parameters are the frequency of the continuous pitch curve and its rate of change. Pitch continuous curve is 1
When noise is mixed in two target signals, one f-
Only one should be calculated for the t-map, but in a real environment it is unlikely that the pitch continuous curve will be uniquely determined, as described later with reference to FIG. 4, so multiple pitch continuous curves are candidates. Presumed. Further, when separating a mixed signal including two or more voice signals, two or more pitch continuous curves are estimated.

When the characteristic parameters are obtained in the harmonic calculation layer and the pitch continuity calculation layer in this manner, consistency calculation is performed in each layer (304). Specifically, the instantaneous coding layer receives the characteristic parameter from the harmonic calculation layer and calculates the consistency with the characteristic parameter of its own layer. The harmonic calculation layer receives the characteristic parameters from the instantaneous coding layer and the pitch continuity calculation layer, and calculates the consistency with the characteristic parameters of its own layer. The pitch continuity calculation layer receives the characteristic parameter from the harmonicity calculation layer and calculates the consistency with the characteristic parameter of its own layer. These consistency calculations proceed in parallel in each layer. By performing the calculation at the same time, the consistency between the characteristic parameters of each layer can be obtained.

Each layer updates the characteristic parameters of its own layer based on the calculated consistency. The updated characteristic parameters are further passed to the upper and lower layers as indicated by the arrow in the figure,
Consistency calculations are performed (305).

When all layers are consistent, the calculation ends (306). Next, each layer is the fundamental parameter f ₀ of the harmonic structure and the harmonics nf
₀ (n is an integer), its rate of change dnf ₀ , the amplitude a (nf ₀ , t) and the phase θnf ₀ are output at each time (307). By using this result to reconstruct the signal, the target audio signal is separated. As described above, it is possible to robustly separate a harmonic structured sound having a complicated structure by the method of performing the entire calculation in parallel based on the consistency between various characteristic parameters.

In the above description, the harmonic structures are grouped on the ft map for simplification, but this grouping is based on the feature space of four dimensions or more depending on the number of features extracted in the instantaneous coding layer. But you can do it too. For example, in addition to the frequency of each frequency component candidate point and its change rate, the amplitude of each frequency component candidate point and its change rate are used to perform grouping so that the frequency and amplitude of the frequency component candidate point change continuously. be able to. This corresponds to the fact that the amplitudes of the signals from the same sound source are continuous, just as the pitches of the signals from the same sound source are continuous. The same applies to other instantaneous coding features.

The method of performing sound separation by paying attention to the local structure of the audio signal as in the embodiment described above is described in the above-mentioned Japanese Patent Laid-Open No.
Several proposals have been made so far, such as Japanese Patent No. 9-257559. The problem with such a conventional method is that it is not uniquely determined to which amplitude maximum point at a next time a certain amplitude maximum point should be connected. This point will be described with reference to FIG.

FIG. 4 is an example of the ft map obtained by frequency analysis of mixed input signals. It is assumed that the mixed input signal contains two consecutive audio signals and is instantaneously noisy. The black circles in the figure represent the maximum amplitude points of the spectrum of the mixed input signal. (a) shows the estimation result of pitch continuity by the conventional method. In this method, the flow of sound is estimated by connecting the amplitude maximum point in the frequency direction at a certain time with the amplitude maximum point at the next time. However, as shown in the figure, there are many possible connectable flows and they are not uniquely determined. Especially when the S / N ratio is low, the number of candidate points connected to the vicinity of the target signal increases, and the problem becomes more difficult.

On the other hand, in the above-described embodiment, there is a possibility that there is a deviation from the actual frequency component due to the deviation of the discrete Fourier transform resolution, the modulation of the input signal, and the proximity of the frequency component due to the instantaneous encoding as shown in (b). Since the frequency component candidate point and its rate of change are obtained instead of the amplitude maximum point with
You can see in which direction the frequency changes, as indicated by the arrow on the ft map. Therefore, the sound flow becomes clear as shown by the solid line and the dotted line in FIG. 7B, and the frequency component candidate points such as the two arrows marked with X are separated as noise.

Further, in this embodiment, attention is paid to the fact that the acoustic features included in the voice signals emitted from the same sound source are related to each other and the properties thereof are not abruptly changed but have consistency. Therefore, a voice signal in non-stationary noise can be separated by utilizing the consistency of the voice signal, and it is possible to follow the gradual amplitude and frequency changes of the same sound source signal from the global characteristic parameters. it can.

Further, by simultaneously extracting and associating various characteristic parameters having different properties, even in the case of an input signal for which the characteristic extraction accuracy of a single unit cannot be ensured, the uncertainties can be complemented and the characteristic extraction accuracy can be improved as a whole. .

<Computing Element> In the embodiment of the present invention, each layer is composed of one or more computing elements. In the present specification, a “calculation element” is an information processing element that is generated in a one-to-one correspondence with a characteristic parameter, performs the same processing, and has a function of mutually supplying the characteristic parameter with another calculation element. Yes, it does not mean a physical element.

FIG. 5 is a diagram showing an example of the configuration of the calculation elements in each layer. The configurations of the calculation elements corresponding to the global feature extraction layer, the intermediate feature extraction layer, and the local feature extraction layer are shown in order from the top. Here, although FIG. 5 is described for the combination of the features of the above-described embodiment as shown in the parentheses of FIG. 5, the same applies to other combinations of the features. 501 is an example of the ft map supplied by the frequency analysis unit, which is 5, ₃ , ₅ , and 5 amplitude maximum points (black points in the figure) for _four times t ₁ , t ₂ , t ₃ , and t ₄ , respectively. Represents)
Is detected.

In the local feature extraction layer, the vibration on the f-t map is
A calculation element corresponding to the maximum width point is generated. In Figure 5
The computational element is shown as a black square (eg 503)
ing. In the intermediate feature extraction layer, they are in a harmonic relationship with each other.
One for each group of computational elements in the local feature extraction layer
The calculation element of is generated. In FIG. 5, time t₁, T₃, T
_FourSince a harmonic structure is recognized for each of
Three calculation elements j-2, j, j + 1 are generated in the feature extraction layer.
It These are shown as black boxes (e.g. 504) in the figure.
Has been. Time t₂For, between the frequency component candidate points
Since there were few numbers and no harmonic structure was recognized,
At this time, the calculation element j-1 is not generated.

In the global feature extraction layer, from the fundamental frequency calculated by the harmonic calculation and the rate of change thereof, calculation elements are provided for the group that seems to have pitch continuity from time t ₁ to t _4. Is generated. In FIG. 5, the calculation element j-
Since it is recognized that there is pitch continuity for 2, j and j + 1, the calculation element i is generated. This is shown in FIG. 7 by one laterally long rectangular parallelepiped (505).

As the consistency calculation progresses and the validity of the calculation element i becomes stronger, the validity of the calculation element in the intermediate feature extraction layer corresponding to the time t ₂ becomes stronger. Is generated. This is indicated by a hollow rectangular parallelepiped 506 in the figure. If the validity of the calculation elements j-2, j-1, j + 1 becomes stronger by continuing the consistency calculation, the calculation of the part indicated by a white square (eg 502) in the local feature extraction layer is performed. Since the validity of the existence of the element becomes stronger,
Corresponding computing elements are generated.

In actual sound separation, there is an amplitude maximum point of a voice signal other than the target signal and noise on the ft map,
Also for these, a calculation element is generated in the local feature extraction layer, and a calculation element corresponding to the intermediate feature extraction layer is generated for a group having a harmonic relationship among them. In particular, multiple harmonic groups are often recognized at the beginning of the consistency calculation. The same applies to the global feature extraction layer. However, such a calculation element is determined to be less appropriate as the consistency calculation progresses, and disappears. In this way, the calculation element corresponding to the characteristic parameter of the target signal is selected.

It should be noted that the configuration of each layer by the calculation element shown in FIG. 5 is merely an example, and the configuration of the calculation element always changes as the consistency calculation progresses. This is because, as described above, calculation elements are generated for all the amplitude maximum points on the ft map at the start of calculation, but as the calculation progresses, calculation elements with low validity disappear and calculation with high validity is performed. This is because only the elements survive and the calculation converges. It can be considered that FIG. 5 corresponds to the case where only one harmonic structure is observed at each time, or the case where the calculation element corresponding to the harmonic structure of low validity disappears due to the progress of the consistency calculation. it can.

FIG. 6 is a functional block diagram showing an example of the configuration of the calculation element 600. In the following description, the layer including the calculation element is N layer, and the layer one lower layer is (N-1).
The layer and the next higher layer will be referred to as (N + 1) layer. Further, the number of the calculation element of the (N + 1) layer is represented by i, the number of the calculation element of the N layer is represented by j, and the number of the calculation element of the (N−1) layer is represented by k.

The lower consistency calculation unit 604 finds a feature parameter set P _N-1 calculated in the (N-1) layer that matches the feature extracted in its own layer, and calculates the parameter P _Nj . Then, the consistency R with the characteristic parameter P _Nj of the N layer
_Nj is calculated by the Bottom-Up function (BUF) of the following equation.

[0067]

[Equation 1]

The upper consistency calculation unit 601 determines the upper (N + 1)
A set P of characteristic parameters calculated by each calculation element of the layer
The consistency Q _Nj between _{(N + 1) i} and the characteristic parameter P _Nj of the N layer is calculated by the Top-Down function (TDF) of the following equation.

[0069]

[Equation 2] Here, S _{(N + 1) i} is the validity index of the (N + 1) layer (the validity index will be described later).

The number of parameters corresponds to the number of calculation elements included in each layer. For the computing element in the intermediate feature extraction layer of FIG. 6, the number of parameters supplied from the (N-1) layer is k, and the number of parameters supplied from the (N + 1) layer is one.

The matching functions Q _Nj and R _Nj respectively calculated by the matching calculation units 601 and 604 are multiplied by the multiplication unit 602 to calculate the validity index S _Nj . The validity index S _Nj is
It is a parameter that represents the certainty of the parameter P _Nj of the calculation element j in the N layer, and is expressed as an overlapping portion of the matching functions Q _Nj and R _{Nj in} the parameter space.

The threshold calculator 603 calculates the threshold S _th by the threshold calculation function (TCF) for all the calculation elements in the N layer. The threshold value S _th is set to a relatively small value in the initial stage of calculation while referring to the validity index S _{(N + 1) i} of the upper layer, and is set to a large value as the calculation converges.
The threshold value calculation unit 603 is not included in the calculation element 600.

The threshold comparing section 605 compares the threshold S _th with the validity index S _Nj . If the validity index S _Nj is lower than the threshold value S _th , it means that the validity of the existence of this calculation element is low, and therefore the calculation element disappears.

The parameter updating unit 606 uses the validity index S _Nj.
The parameter P _Nj is updated so that The updated parameter P _Nj is passed to the calculation elements of the (N + 1) layer and the (N−1) layer in the next calculation cycle.

At the highest hierarchy in the feature extraction section, the configuration of the calculation element itself is the same as that shown in FIG. 6, but the parameters input to the calculation element are as shown in FIG. In this case, instead of the validity index from the upper layer, the index (S _win ) of the most valid element among the calculation elements in the global feature extraction layer is used. In addition, instead of the parameters from the upper layer, the parameters from the lower layer are calculated by the parameter estimation function (PPF) 607 (P
_predict) was used to calculate the consistency Q _Nj and the threshold S _{t h.} Therefore, the TDF is as follows.

[0076]

[Equation 3]

A calculation element having a high validity index S _Nj has a strong influence on the TDF of the calculation element in the lower layer (N-1) layer, and has an effect of increasing each validity index. On the contrary, the calculation element with the low validity index S _Nj has a small influence, and disappears when S _Nj falls below the threshold value S _th . Threshold S _th
Is calculated every time the validity index of the (N + 1) layer changes,
Furthermore, the TCF is not fixed but changes depending on the progress of calculation.
This allows many calculation elements while the number of calculations is small.
(I.e., the corresponding feature parameter) is left, and the survival condition can be tightened as the levels of matching become more consistent, so the precision of the feature parameter can be improved compared to when the threshold value is fixed. it can.

FIG. 8 is a flow chart for explaining the flow of calculation in the feature extraction section having the (N-1) layer, N layer, and (N + 1) layer formed by the above-mentioned calculation elements.

When the calculation is started, first, necessary initialization is performed (801). Then, (N-1) layer, N layer, (N +
1) In each of the layers, the parameter update value of the calculation element of each layer is calculated based on the parameter data input from the layer to be connected (803), and the parameter of the calculation element of each layer is updated (805). In addition, the validity index is calculated (807).

Based on the calculated parameters, each layer updates the connection relation with the layer to which each layer is connected (809). At this time, the calculation element whose validity index is below the threshold disappears (811). In addition, a necessary calculation element is newly generated (813).

When the parameter update values of all the calculation elements fall below the set values (815), it is determined that the layers have been matched, and the calculation ends. If there is a parameter update value that exceeds the set value among the calculation elements, the update value is calculated again (80
3), and the same calculation is repeated thereafter.

<Second Embodiment> The feature parameters extracted in each layer are not limited to the combinations described in connection with the first embodiment, and may be local, intermediate, or global depending on the type of feature to be adopted. A configuration can be adopted in which each target feature extraction layer is assigned. Other available features include onset / offset and intonation. The feature parameters are extracted by an appropriate method, and the feature parameters are exchanged between layers so as to achieve matching, as in the first embodiment described above.

In the second embodiment of the present invention, as shown in FIG. 9, two sound input terminals can be provided to use the sound source direction as a feature. In this case, a sound source direction analysis unit 911 is separately provided as shown in the figure,
The sound source direction information is supplied to the feature extraction unit 915. The method of sound source direction analysis is a well-known technique, for example, a method of analyzing the sound source direction from the time difference or sound pressure difference of the sound reaching the microphone, or frequency analysis of the input signal, and the arrival time difference and / or sound pressure difference for each frequency. A method such as analyzing the direction of the sound source may be used.

In order to analyze the direction of the sound source, the mixed input signal is input to a plurality of acoustic input terminals (microphone L901 in this embodiment).
And microphone R903)). Frequency analysis section
In 905, the signals collected by the microphone L901 and the microphone R903 are separately analyzed by a method such as FFT, and a frequency spectrum is obtained.

The feature extraction unit 915 is provided with the same number of instantaneous coding layers as the microphones. In this embodiment, an instantaneous coding layer (L) 917 and an instantaneous coding layer (R) 919 corresponding to the microphone L and the microphone R are provided to receive the spectrum.
The instantaneous coding layers 917 and 919 calculate the frequency and amplitude of the frequency component candidate point and its change over time based on the received frequency spectrum.

The instantaneous coding layers 917 and 919 also use the harmonic information calculated by the harmonic calculation layer 923 to verify the consistency with the calculated frequency component candidate points.

To the sound source direction analysis unit 911, the mixed input signal collected by the microphone L901 and the microphone R903 is input. The input signal is F in the sound source direction analysis unit 911.
It is cut out with the same time window width as FT, the cross-correlation of the two signals is calculated, and the maximum point thereof is obtained (black points shown in FIG. 10).

The feature extraction unit 915 is provided with a sound source direction estimation layer 921. The sound source direction estimation layer 921 includes a sound source direction analysis unit 911.
Of the obtained cross-correlation peaks, the one with an error smaller than a constant value with the line drawn in the time direction is estimated to be the time difference due to the difference in the sound source direction (in the case of FIG. 10, τ1,
Three are estimated, τ2 and τ3). The arrival time difference of each target signal due to the sound source direction difference estimated in this way is passed to the harmonic calculation layer 923.

The sound source direction estimation layer 921 also includes the harmonic calculation layer 9
Using the time difference of the harmonic information obtained from 23, the consistency with each estimated arrival time difference is verified.

The harmonic calculation layer 923 is an instantaneous coding layer (L) 917.
And the local peaks obtained from the instantaneous coding layer (R) 919 are added together with the respective arrival time differences obtained from the sound source direction estimation layer 921 being shifted and added. Specifically, τ1, on the left and right microphones 901, 903,
Since signals of similar waveforms whose arrival times are deviated by τ2 and τ3 are input, it is used that the outputs from the instantaneous coding layers 917 and 919 have the same frequency component candidate points deviated by τ1, τ2, and τ3, respectively. Then, the frequency component of the target signal arriving from the same sound source is emphasized. In this way the sound separation device 900
By configuring the above, it is possible to improve the separation accuracy of the mixed input signal including a plurality of target signals.

The operations of the pitch continuity calculating layer 925 of the feature extracting unit 915 and the signal synthesizing unit 927 are the same as those already described with reference to FIG. Similarly, each layer is composed of calculation elements, but the calculation element in the harmonic calculation layer 923 receives characteristic parameters from a plurality of layers (that is, the instantaneous coding layer and the sound source direction estimation layer). It is configured to calculate a feature parameter and pass the calculated feature parameter to multiple layers.

<Third Embodiment> FIG. 11 shows a sound separating device according to a third embodiment of the present invention.

The mixed input signal is transmitted through a plurality of acoustic input terminals (two microphones L1001 and R1003 in this embodiment).
Input). In the frequency analysis unit 1005, the microphone L10
The signals input by 01 and the microphone R1003 are separately analyzed by a method such as FFT to obtain a frequency spectrum.

The feature extraction unit 1015 is provided with as many instantaneous coding layers as microphones. In this embodiment, an instantaneous coding layer (L) 1017 and an instantaneous coding layer (R) 1019 corresponding to the microphone L and the microphone R are provided to receive the spectrum.
Instantaneous coding layers 1017, 1019, based on the frequency spectrum received respectively, the frequency and amplitude of the frequency component candidate point,
The change over time is calculated.

The instantaneous coding layers 1017 and 1019 also use harmonic information calculated by the harmonic calculation layer 1023 to verify the consistency with the estimated frequency component candidate points.

The sound source direction analysis unit 1011 includes a frequency analysis unit 1005.
The cross-correlation in each frequency channel is calculated from the FFT performed by the above, and the maximum point is obtained (black points shown in FIG. 12). Also, the sound pressure difference of each frequency channel is calculated.

The feature extraction unit 1015 has a sound source direction estimation layer 1021.
Is provided. The sound source direction estimation layer 1021 is a sound source direction analysis unit.
The maximum points are roughly grouped by sound source by obtaining the cross-correlation of the signals of the respective frequency channels obtained from 1011 and the maximum points thereof, and the sound pressure difference of each channel. The arrival time difference of each target signal due to the sound source direction difference estimated in this way is passed to the harmonic calculation layer 1023.

The sound source direction estimation layer 1021 is also a harmonic calculation layer.
By using the harmonic structure information obtained from 1023, the consistency between the estimated arrival time difference and the sound source group is verified.

The harmonic calculation layer 1023 is the instantaneous coding layer (L) 10
17 and the frequency component candidate points obtained from the instantaneous coding layer (R) 1019 are shifted by the respective arrival time differences obtained from the sound source direction estimation layer 1021 and added, and the sound source direction estimation layer 10 is further added.
By using the same sound source information obtained from 21,
Calculate harmonics.

The operations of the pitch continuity calculating layer 1025 of the feature extracting unit 1015 and the signal synthesizing unit 1027 are the same as those already described with reference to FIG. Similarly, each layer is composed of calculation elements, but the calculation element in the harmonic calculation layer 1023 receives characteristic parameters from multiple layers (that is, the instantaneous coding layer and the sound source direction estimation layer). It is configured to calculate a feature parameter and pass the calculated feature parameter to multiple layers.

[0101]

FIG. 13 to FIG. 15 show the results of separating a target signal from an input signal in which noise is mixed in the target signal using the sound separation device 100 according to the first embodiment of the present invention described above.
Shown in. In each figure, (a) is the spectrum of the target signal,
(b) shows the spectrum of the input signal mixed with noise, and (c) shows the spectrum of the output signal separated from the noise. The horizontal axis of each figure is time (msec), and the vertical axis is frequency.
(Hz) An ATR voice database was used as the input signal.

FIG. 13 shows the case where intermittent noise is mixed in the target signal. The target signal in (a) is a part of the "Family Restaurant" uttered by a woman, "Familyless", and white noise was mixed for 15 ms every 200 ms of the target signal, and was used as the input signal in (b). The output signal in (c) was created by synthesizing the waveforms from the characteristic parameters extracted from the input signal. As is clear from the figure, white noise is almost completely removed.

FIG. 14 shows the result when noise is continuously mixed in the target signal. The target signal in (a) is a part of the female utterance "Finally," and the target signal added with white noise with an S / N ratio of 20 dB was used as the input signal in (b). The output signal in (c) was created by synthesizing the waveforms from the characteristic parameters extracted from the input signal. It can be seen that the spectral shape of the target signal is reproduced with high accuracy.

FIG. 15 shows the separation result when another voice signal is mixed in the target signal. The target signal in (a) is a part of the female utterance "Finally", and the target signal added with the male utterance "Uyamau" with an S / N ratio of 20 dB was used as the input signal in (b). The output signal in (c) was created by synthesizing the waveforms from the characteristic parameters extracted from the input signal. Although the spectrum is slightly different compared to the target signal in (a),
The target signal has been reproduced to a level that is audibly unproblematic.

Although the present invention has been described above with reference to specific embodiments, the present invention is not limited to these, and various changes and substitutions can be made. For example, the feature parameters used in each of the described embodiments are for exemplification, and new feature parameters discovered in future research and relationships between feature parameters can also be used in the present invention. Moreover, although the time change rate is used as the change of the frequency component candidate points, a second derivative or the like may be used.

[0106]

According to the present invention, a target sound is robustly separated in an environment in which non-stationary noise is mixed by extracting and utilizing a dynamic feature amount such as a temporal change rate of parameters of a mixed input signal. can do. Also, without preparing a template in advance, the global and local features of the signal can be evaluated in parallel while interacting with each other, so that the target sound with complex frequency and amplitude changes can be separated flexibly. can do.

[Brief description of drawings]

FIG. 1 is a block diagram showing an overall configuration of a sound separation device according to a first embodiment of the present invention.

FIG. 2 is a diagram showing a hierarchical structure of a feature extraction unit.

FIG. 3 is a diagram showing processing performed in each layer of a feature extraction unit.

FIG. 4 is a diagram illustrating detection of pitch continuity by a conventional method and the sound separation device of the present invention.

FIG. 5 is a diagram showing an example of a configuration of a feature extraction unit using a calculation element.

FIG. 6 illustrates an embodiment of a computing element.

FIG. 7 is a diagram showing an embodiment of a computing element.

FIG. 8 is a flowchart illustrating a process in a feature extraction unit shown in FIG.

FIG. 9 is a block diagram showing a configuration of a sound separation device according to a second embodiment of the present invention.

FIG. 10 is a graph for explaining estimation of a sound source direction.

FIG. 11 is a block diagram showing a configuration of a sound separation device according to a third embodiment of the present invention.

FIG. 12 is a graph for explaining estimation of a sound source direction.

FIG. 13 is a spectrum diagram showing a result of audio signal separation performed by the sound separation device according to the first embodiment.

FIG. 14 is a spectrum diagram showing a result of audio signal separation performed by the sound separation device according to the first embodiment.

FIG. 15 is a spectrum diagram showing a result of audio signal separation performed by the sound separation device according to the first embodiment.

[Explanation of symbols]

100, 900, 1000 Sound separation device 101 Signal input unit 102, 905, 1005 Frequency analysis unit 103, 915, 1015 Feature extraction unit 104, 927, 1027 Signal synthesis unit 106, 917, 919, 1017, 1019 Local feature extraction layer (Instantaneous coding layer) 107, 923, 1023 Intermediate feature extraction layer (harmonic calculation layer) 108, 925, 1025 Global feature extraction layer (pitch continuity calculation layer) 600 Computation element 601 Upper consistency calculation unit 603 Threshold calculation unit 604 Lower consistency calculation unit 605 Threshold comparison unit 606 Parameter update unit 901, 903, 1001, 1003 Microphone 911, 1011 Sound source direction analysis unit 921, 1021 Sound source direction estimation layer

Claims (12)

[Claims] 1. A sound separation device for separating the target signal from a mixed input signal in which acoustic signals emitted from different sound sources and a target signal are mixed, wherein the mixed input signal is frequency-analyzed at each time. Frequency analysis means for calculating a spectrum and an amplitude maximum point, a narrow layer for analyzing a narrow characteristic parameter using the spectrum and the amplitude maximum point, and a wide area using the feature parameter extracted by the narrow layer Characteristic extracting means for extracting characteristic parameters associated with the target signal, the signal synthesizing section synthesizing the target signal based on the extracted characteristic parameters. And a sound separating device including the means. 2. The narrow and wide layers mutually supply the characteristic parameters analyzed in the respective layers, and update the characteristic parameters of the respective layers based on the supplied characteristic parameters. 1. The sound separation device according to 1. 3. The narrow band layer is an instantaneous coding layer for calculating a frequency of a candidate point that constitutes a frequency component included in an input signal and its change, and an amplitude and its change. Item 2. The sound separation device according to item 2. 4. The wide-area layer groups the frequency component candidate points having a harmonic structure based on the frequencies of the frequency component candidate points and their changes, and groups the fundamental frequency of the harmonic structure and the harmonics included in the harmonic structure. The sound according to claim 2, further comprising: a harmonic calculation layer that calculates a wave and its change, and a pitch continuity calculation layer that calculates signal continuity from the fundamental frequency and changes thereof at a plurality of times. Separation device. 5. The sound separation device according to claim 3, wherein the wide area layer further includes a sound source direction estimation layer that estimates a direction of a sound source that has emitted the mixed input signal. 6. The wide-area layer groups frequency component candidate points having a harmonic structure based on the frequency of the frequency component candidate point and its change and the estimated sound source direction, and defines the fundamental frequency of the harmonic structure as a group. And a harmonic continuity calculation layer that calculates harmonics included in the harmonic structure and changes thereof, and a pitch continuity calculation layer that calculates signal continuity from the fundamental frequency and changes thereof at a plurality of times. Item 5. The sound separation device according to item 5. 7. The sound separation device according to claim 3, wherein a time change rate is used as the change. 8. The hierarchy is composed of one or a plurality of calculation elements that respectively perform similar processing to calculate a characteristic parameter, and the calculation elements are included in an upper connection layer and a lower connection layer. 8. The sound separation device according to claim 1, wherein the calculated calculation elements and the calculated characteristic parameters are mutually supplied. 9. The first calculation element indicates a degree of matching between the characteristic parameter supplied from the calculation element included in a higher-level connected layer and the calculated characteristic parameter.
And calculating a second consistency function indicating a degree of matching between the feature parameter supplied from the calculation element included in the lower connecting hierarchy and the calculated feature parameter, 9. The sound separation device according to claim 8, wherein the feature parameter is updated so as to maximize the validity index represented by the product of the consistency functions of. 10. The sound separation device according to claim 9, wherein the validity index is supplied to a calculation element included in the lower hierarchy. 11. The sound separating device according to claim 10, wherein a threshold value is calculated based on the supplied validity index, and the calculation element is extinguished when the value of the validity index falls below the threshold value. 12. A new calculation element is generated in a lower hierarchy when the validity index is larger than a predetermined value.
The sound separation device according to 0.