JP2010206449A - Speech direction estimation device and method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a speech direction estimation device which does not request to arrange many microphones so as to enclose a speaker, and can appropriately estimate the speech direction even under environment in which a reverberation time is long. <P>SOLUTION: A correlation matrix calculation unit sequentially generates and outputs an M×M correlation matrix representing correlation between digital sound signals as expected values by frames, and a base matrix generation unit inputs the digital sound signals and information on a speaker position, and generates a base matrix having a base number D. A correlation matrix decomposition unit inputs correlation matrixes and base matrixes by the frames and decomposes the correlation matrixes into component columns projected on the base matrixes, and a frequency averaging processing unit inputs the component columns and outputs a frequency averaged component column averaged with frequencies by the frames. A front-lateral direction decision unit inputs the frequency averaged component column, and estimates whether the speaker has spoken in the front direction by the frames. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technique for estimating the utterance direction of a speaker from an audio signal input to a microphone.

  A system for exchanging voice information such as a telephone or a voice conference terminal is generally called a voice communication system. In the video conference system, the video is added to the audio information and presented, so that the situation of the place is easily transmitted, but in the audio communication system, it is difficult to grasp the situation of the other party. One of the information on the other party's situation is utterance direction information. By receiving this information from the other party, it is possible to grasp the direction in which the speaker is speaking and to facilitate communication.

Conventional techniques for estimating such speech direction information are disclosed in Non-Patent Documents 1 and 2 and the like, and a configuration example is shown in FIG. The speech direction estimation apparatus 10 in this configuration example estimates speech direction information as follows.
(i) The voice from the speaker 1 is picked up using M (M is an integer of 2 or more) microphones 11-1,. The collected analog signal is converted into a digital signal vx (t) = [x ₁ (t),..., X _M (t)] ^T by the AD converter 12. Here, t represents an index of discrete time.

(ii) The frequency domain transform unit 13 receives a set (frame) of the digital signals composed of a plurality of samples as an input, and performs frequency domain signal VX (ω, n) = [X ₁ (ω, n) by fast Fourier transform or the like. _{, ···, X M (ω,} n)] to convert to ^T. Here, ω represents a frequency index, and the total number of frequency indexes is Ω. N represents the index of the frame.
(iii) The fixed beamformer design unit 14 sets the fixed beamformer VG (ω, r, θ) = [G ₁ (ω, r, θ),..., G _M (for each speaker position and direction. ω, r, θ)] ^T is designed. G _i (ω, r, θ) is a coefficient to be multiplied by the frequency component x _i (ω, n) of the i-th microphone in order to emphasize / suppress the sound source at the speaker position r and the speech direction θ.

In designing, acoustic propagation characteristics between a sound source and a microphone VH (ω, r, θ) = [H ₁ (ω, r, θ),..., H _{M for} each predetermined speaker position and direction. (ω, r, θ)] ^T is obtained using simulation values or actual measurement values. Here, H _i (ω, r, θ) represents an acoustic propagation characteristic between the sound source at the speaker position r and the speech direction θ and the i-th microphone.

The fixed beamformer VG (ω, r, θ) is designed as a value that satisfies the expressions (1) and (2) representing the relationship with the acoustic propagation characteristics.
VH (ω, r _T , θ _T ) ^H · VG (ω, r _T , θ _T ) = 1 (1)
_{VH (ω, r U, θ} U) H · VG (ω, r T, θ T) = 0 (2)
Expressions (1) and (2) emphasize the output power of the speaker position r _T and the speech direction θ _T and suppress the output power of the other speaker positions r _U and the speech direction θ _U. It shows that VG (ω, r, θ) is designed.

(iv) In the product-sum calculation unit 15, the frequency domain signal VX (ω, n) = [X ₁ (ω, n),..., X _M (ω, n)] ^T and the fixed beamformer VG (ω , r, θ) = [G ₁ (ω, r, θ),..., G _M (ω, r, θ)] ^T as an input, for each frequency ω, speaker position r, and speech direction θ. Multiplying the frequency component X _i (ω, n) corresponding to each microphone by the coefficient G _i (ω, r, θ) of the fixed beamformer, and adding the obtained M components, the output Y (ω, n, r, θ) is calculated. This calculation is synonymous with calculating Y (ω, n, r, θ) = VG (ω, r, θ) ^H · VX (ω, n).

(v) The power calculator 16 calculates and outputs power | Y (ω, n, r, θ) | ² from the output Y (ω, n, r, θ) from the product-sum calculator 15.
(vi) The frequency averaging processing unit 17 averages the power | Y (ω, n, r, θ) | ² output from the power calculation unit 16 by frequency, and calculates AY (n, r, θ). obtain. In this calculation, if F ₀ is defined as the frequency index used in the averaging process, and | F ₀ | is defined as the total number of frequency indexes,

Is equivalent to calculating Note that F ₀ satisfies Ω ≧ | F ₀ |.
(vii) The sound source direction selection unit 18 searches for the speaker position r and the speech direction θ where the power AY (n, r, θ) averaged by frequency for each frame is maximum, and the power AY (n , r, θ) is determined as the estimated speech direction θ _out (n).

Hiroshi Nakajima, “Expanded Beamforming for Estimating Sound Source Direction”, Proceedings of the Acoustical Society of Japan, September 2005, p.619-620 Hiroshi Nakajima, 8 others, "Sound source directivity estimation using extended beamforming", Proceedings of the Acoustical Society of Japan, September 2005, p.621-622

The following two points can be cited as problems of the prior art.
(i) To deal with utterances at any position and to estimate the direction of utterance with high accuracy, a large number of microphones are required and the position of the microphones must be devised. As the output power | Y (ω, n, r, θ) | ² of the fixed beamformer designed for each position and speech direction is different, the speech direction can be estimated with higher accuracy. However, assuming a sound source with a strong directivity in front of the mouth, such as a sound wave radiated from the mouth of the speaker, as shown in FIG. 16, it is necessary to collect sound so as to surround the speaker with a large number of microphones. Depending on the speaker position / speech direction, there is no difference in the output power of the fixed beamformer, and the speech direction estimation error increases (for example, in the experiment of Non-Patent Document 2, 64 microphones are used). Therefore, in order to reduce the error, a large number of microphones are required, the apparatus becomes large, and it is difficult to use it by attaching it to a portable apparatus such as a telephone or an audio conference terminal.

(ii) Reverberation time (time from the arrival of the direct wave until 60 dB attenuation from the collected power of the direct wave)
High speech direction estimation performance cannot be obtained in a reverberation environment with a reverberation time of 250 msec or more. In a reverberation environment with a reverberation time of 250 msec or more, the sound propagation characteristics VH (ω, r, θ) are accurate because many strong reflected waves are mixed. It is difficult to design well. Therefore, ambiguity occurs in the output of the fixed beamformer, and the estimation accuracy deteriorates. For example, in an actual environment room that is not subjected to low reverberation processing, reverberation time is generally about 250 to 500 msec, so that accurate estimation is difficult.

  An object of the present invention is to estimate the speech direction, which does not require a large number of microphones to be placed so as to surround the speaker, and can appropriately estimate the speech direction even in a reverberant environment with a reverberation time of 250 msec or more. To provide an apparatus, a method, and a program.

  The speech direction estimation apparatus of the present invention includes an AD conversion unit, a frequency domain conversion unit, a correlation matrix calculation unit, a base matrix generation unit, a correlation matrix decomposition unit, a frequency averaging processing unit, and a front / lateral direction determination unit. It comprises. The AD conversion unit converts analog audio signals collected by M microphones (M is an integer of 2 or more) into digital audio signals. The frequency domain transform unit transforms the digital audio signal from the time domain to the frequency domain in units of frames composed of a plurality of samples. The correlation matrix calculation unit sequentially generates and outputs an M × M correlation matrix representing the correlation between the respective digital audio signals converted into the frequency domain for each frequency as an expected value for each of a plurality of frames. The base matrix generation unit receives a digital speech signal and speaker position information as input, and generates a base matrix composed of D bases (D is an integer of 2 or more). The correlation matrix decomposition unit inputs a correlation matrix and a base matrix for each frame, and decomposes the correlation matrix into component columns projected onto the base matrix. The frequency averaging processing unit receives the component sequence and outputs a frequency averaged component sequence averaged by frequency for each frame. The front / side orientation determination unit estimates whether or not the direction of the speaker's utterance is the front direction for each frame, using the frequency averaged component sequence as an input.

  According to the speech direction estimating apparatus of the present invention, it is not necessary to arrange a large number of microphones so as to surround a speaker, and it is possible to appropriately estimate the speech direction even in a reverberant environment where the reverberation time is 250 msec or more. It becomes.

The figure which shows the function structural example of the speech direction estimation apparatus 100 of 1st Embodiment. The figure which shows the example of a processing flow of the utterance direction estimation apparatus 100 of 1st Embodiment. The image figure which shows the positional relationship of a microphone, a speaker, and a speech direction. The image figure about the method of determining left-right orientation. The figure which shows the propagation characteristic of an audio | voice signal in a time domain. The image figure which shows the correlation with an utterance direction, a base vector, and an eigenvalue / eigenvector. The image figure which shows the relationship between an utterance direction and an eigenvalue. The figure which shows the function structural example of the utterance direction estimation apparatus 200 of 2nd Embodiment. The figure which shows the example of a processing flow of the speech direction estimation apparatus 200 of 2nd Embodiment. The image figure about the method of judging front direction and sideways. The figure which shows the function structural example of the speech direction estimation apparatus 300 of 3rd Embodiment. The figure which shows the verification environment and conditions of an effect. The figure which shows the verification result about the relationship between an utterance direction and an eigenvalue. The figure which shows the verification result of left-right direction estimation. The figure which shows the service structural example which incorporated this invention in the audio conference terminal. The figure which shows the function structural example of the speech direction estimation apparatus by a prior art. The image figure which shows the relationship between the microphone and speaker by a prior art.

[First Embodiment]
FIG. 1 shows a functional configuration example of the speech direction estimating apparatus 100 of the present invention, and FIG. 2 shows a processing flow example thereof. The utterance direction estimation apparatus 100 estimates whether the utterance direction is the front direction, the left direction, or the right direction with respect to the microphone array.

  The utterance direction estimation apparatus 100 includes a microphone array 101 including M microphones (M is an integer of 2 or more) 101-1 to 101-M, an AD conversion unit 12, a frequency domain conversion unit 13, and a correlation matrix calculation unit. 102, an eigenvalue decomposition unit 201, a first eigenvector power calculation unit 104, a first frequency averaging processing unit 105, a left / right orientation determination unit 106, a second frequency averaging processing unit 202, and a front / side orientation determination unit 203.

  In the prior art, as shown in FIG. 16, it was necessary to arrange a large number of microphones so as to surround the speaker. However, in the present invention, M microphones 101-1 to 101-M can be provided. It is sufficient to arrange them closely. Although the number of microphones constituting such a microphone array 101 is not too large, according to the configuration of the present invention described below, it is possible to estimate the utterance direction with two or more microphones. The arrangement may be two-dimensional or three-dimensional. By adopting a configuration in which a small number of microphones are arranged densely in this way, it can be attached to a portable device such as a telephone or an audio conference terminal, and the direction of speech by surrounding speakers can be estimated It becomes. The speaker speaks around the microphone array 101. FIG. 3 shows an image of a speaker talking around a microphone array 101 composed of seven microphones as viewed from above, and the direction of the arrow is the direction of speech. FIG. 3A shows a state where the speaker speaks in front of each position, and FIG. 3B shows a state where the speaker speaks sideways.

The AD conversion unit 12 converts the analog voice signals uttered by the speaker 1 collected by the M microphones 101-1 to 101 -M into digital voice signals x ₁ (t),..., X _M (t (S1). Here, t represents an index of discrete time.

The frequency domain transform unit 13 receives as input a set (frame) of the above digital speech signals composed of a plurality of discrete time samples, and performs frequency domain digital speech signals X ₁ (ω, n),. It is converted into X _M (ω, n) and output (S2). Here, n represents a frame index, and ω represents a frequency index. The total number of frequency indexes is Ω.

The correlation matrix calculation unit 102 receives the digital audio signals X ₁ (ω, n),..., X _M (ω, n) in the frequency domain as inputs, and an M × M correlation matrix R representing the correlation between the signals. (ω, k) is sequentially generated and output for each frequency ω by equation (3) (S3).
R (ω, k) = E [VX (ω, n) · VX ^H (ω, n)] (3)

Here, VX (ω, n) = [X ₁ (ω, n),..., X _M (ω, n)] ^T where H represents a conjugate transpose and E represents VX (ω, n) An operator that calculates an expected value for each L frame by averaging processing after calculating VX (ω, n) ^H for each frame. That is, the correlation matrix is sequentially output at a rate of once per L frame, and k represents an output index of the correlation matrix. L is preferably an integer greater than or equal to M.

The eigenvalue decomposition unit 201 receives the correlation matrix R (ω, k) as an input, and first, M eigenvalues λ ₁ (ω, k),..., Λ _M (ω, k) so as to satisfy Expression (4). ) Eigenvalue matrix Λ (ω, k), which is a diagonal matrix with each square as a diagonal element, and M eigenvectors vv ₁ (ω, k),..., Vv _M (ω, k) Is decomposed into eigenvector matrix V (ω, k) by the eigenvalue decomposition method.
R (ω, k) = V (ω, k) · Λ (ω, k) · V ^H (ω, k) (4)
Where Λ (ω, k) = diag [λ ₁ ² (ω, k),..., Λ _M ² (ω, k)]
λ ₁ (ω, k) ≧ λ ₂ (ω, k) ≧ ・ ・ ・ ≧ λ _M (ω, k)
V (ω, k) = [vv ₁ (ω, k),..., Vv _M (ω, k)] ^T
vv _i (ω, k) = [v _{i, 1} (ω, k),..., v _{i, M} (ω, k)]
Note that diag [•] is an operator having the components in [•] as elements of a diagonal matrix.
Then, the first eigenvector vv ₁ (ω, k) corresponding to the first eigenvalue λ ₁ (ω, k) which is the maximum eigenvalue.
k) is output (S4).

The first eigenvector power calculation unit 104 receives the first eigenvector vv ₁ (ω, k) as an input, and v _1,1 (ω, k),... Constituting the first eigenvector vv ₁ (ω, k). For M elements of v _{1, M} (ω, k), the power is calculated by the equation (5), respectively, and M power elements pv _1,1 (ω, k),...
, Pv _{1, M} (ω, k) is output (S5).
pv _{1, i} (ω, k) = | v _{1, i} (ω, k) | (5)
The first frequency averaging processing unit 105 formulas _M power elements pv _1,1 (ω, k),..., Pv _{1, M} (ω, k) generated for each frequency ω, respectively. The average value is calculated according to (6), and M averaged power elements apv _1,1 (k),..., Apv _{1, M} (k) are output (S6).

F ₁ is an index of frequencies used for averaging, | F ₁ | is a total number of frequency indexes, and F ₁ is appropriately set so as to satisfy Ω ≧ | F ₁ |.
The left / right direction determination unit 106 receives M averaged power elements apv _1,1 (k),..., Apv _{1, M} (k) as input, and determines whether the left direction is spoken or the right direction is spoken. The result is output (S7).

The left-right orientation determination is performed by using two averaged power elements apv _{1, α} (k), apv corresponding to two microphones 101-α and 101-β among the M microphones constituting the microphone array 101. This is done by taking a ratio of _{1, β} (k) and comparing it to a predetermined threshold value thr1. FIG. 4 illustrates an example of the determination image in the horizontal direction. In this example, the case where the speech is directed toward the midpoint between the microphone α and the microphone β shown in FIG. 4A is apv _{1, α} (k) / apv _{1, β} (k) = thr1, and this direction is set as follows. The reference speech direction in the left-right direction is determined based on the magnitude relationship between apv _{1, α} (k) / apv _{1, β} (k) and thr1. Specifically, since the rate of attenuation of apv _{1, β} (k) is larger than apv _{1, α} (k) as it goes to the left, apv _{1, α} (k) / apv _{1, β} (k )> Thr1 can be determined to be leftward (FIG. 4 (b)), and the more rightward is, the greater the rate at which apv _{1, α} (k) attenuates than apv _{1, β} (k). Therefore, when apv _{1, α} (k) / apv _{1, β} (k) <thr1, it can be determined to be rightward (FIG. 4 (c)). Note that the two microphones have the widest left-right spacing with respect to the speaker's position so that the average power elements apv _{1, α} (k) and apv _{1, β} (k) tend to differ. It is desirable to choose.

A theoretical background capable of determining the left-right direction of speech with such a configuration will be described.
FIG. 5 shows the propagation characteristics of the audio signal in the time domain. Propagation characteristics can be broadly divided into three types: direct wave, initial reverberation, and rear reverberation. In the time zone in which direct wave and initial reverberation are observed, the directivity with respect to the microphone array composed of a plurality of microphones. It is known that waves with In particular, in the initial reverberation time zone (after the arrival of the direct wave, the time from the direct wave pickup power to the attenuation of 10 dB), a strong reflected wave with directionality is mixed, but the power of this reflected wave depends on the direction of speech. Change. Specifically, since the direct wave power increases as the utterance direction is the front direction, the reflected wave power decreases, and the direct wave power decreases as it is in the horizontal direction. The power of will increase. The present invention uses such a property to estimate the speech direction.

This will be described below in accordance with the configuration of the present invention. FIG. 6 shows how the utterance state in front and side affects each eigenvalue λ _i (ω, k) of the correlation matrix R (ω, k). Here, a case where a microphone array is configured by three microphones is illustrated. When facing the front, many direct waves reach the microphone array, and the arrival rate of the reflected waves is relatively low. Therefore, as shown in FIG. Compared to the basis vector group to be expressed, it has a large power. At this time, the first eigenvalue λ ₁ (ω, k) is significantly larger than the second eigenvalue λ ₂ (ω, k) and the third eigenvalue λ ₃ (ω, k). On the other hand, in the case of the horizontal orientation, the direct waves that reach the microphone array are reduced, so that more reflected waves arrive accordingly. For this reason, as shown in FIG. 6B, the power of the basis vectors expressing the direct wave decreases, and the power of the basis vector group expressing the reflected wave increases. At this time, the first eigenvalue λ ₁ (ω, k) is smaller than that in the front direction, and conversely, the second eigenvalue λ ₂ (ω, k) and the third eigenvalue λ ₃ (ω, k) are in the front direction. Larger than the case. FIG. 7 shows an image of the difference that occurs in each eigenvalue between the case of facing forward and the case of facing sideways.

From the above, it can be seen that the power of the basis vector expressing the direct wave is significantly reflected in the first eigenvalue. And if so, M of the first eigenvector corresponding to the first eigenvalue
The value of each power element corresponding to a single microphone can be considered as a measure of how strong a direct wave reaches each of the M microphones. The power at which direct waves reach each microphone varies depending on the direction of speech. Therefore, as described above, the ratio of the power elements of any two microphones in the reference utterance direction is the threshold (thr1).
By comparing the threshold value and the ratio of the values of the power elements of the two microphones for a certain utterance direction, the utterance is uttered to the left or the right direction with respect to the reference utterance direction. It is possible to determine whether you have spoken.
The eigenvalue decomposition unit 201 further normalizes the first eigenvalue λ ₁ (ω, k) as described above using Expression (7), and outputs a first normalized eigenvalue nλ ₁ (ω, k) (S8).

The second frequency averaging processing unit 202 outputs the first normalized eigenvalue nλ ₁ (ω, obtained for each frequency ω.
The average value of k) is calculated according to the equation (8), and the first averaged eigenvalue aλ ₁ (k) is output (S9).

F ₂ is an index of frequencies used for averaging, | F ₂ | is the total number of frequency indexes, and F ₂ is appropriately set so as to satisfy Ω ≧ | F ₂ |.

The front / horizontal direction determination unit 203 receives the determination result from the left / right direction determination unit 106 and the first averaged eigenvalue aλ ₁ (k), and uses the first averaged eigenvalue aλ ₁ (k) as a predetermined threshold value thr2. If aλ ₁ (k) <thr2, the speaker determines that the speaker is facing sideways with respect to the microphone array, and otherwise determines that the speaker is facing frontward. If it is determined to be in the horizontal direction, the determination result in the left-right direction determination unit 106 is output as it is, and if not, the determination result that it is in the front direction is output (S10). FIG. 10 shows an example of the front / side orientation determination image. Here, thr2 may be arbitrarily set according to the environment and the position of the speaker. Since the first averaged eigenvalue aλ ₁ (k) is obtained for each frame group k, the determination result is also output for each frame group k.

  A theoretical background that can determine whether the speech direction is the front direction or the horizontal direction with such a configuration will be described. The power of the basis vector representing the direct wave is significantly reflected in the first eigenvalue. Specifically, in the case of the front direction, the power of the base vector expressing the direct wave shows a large value and the first eigenvalue also shows a large value. The first eigenvalue is also smaller than when facing the front. Therefore, by using the first eigenvalue as a front / side determination parameter, it is possible to appropriately determine that the first eigenvalue is front-facing if it is larger than a certain threshold value and that it is lateral if it is smaller.

Thus, according to the utterance direction estimation apparatus of the first embodiment, the utterance direction is the front direction,
Since it is possible to determine whether it is facing left or right, it is possible to more smoothly communicate with the other party via the network.
In the first embodiment, the example in which the spatial correlation matrix R (ω, k) is decomposed by the eigenvalue decomposition method and the direction of the speaker is estimated has been described. Next, an alternative method will be described in which other methods can be considered.

[Second Embodiment]
FIG. 8 shows a functional configuration example of the speaker orientation estimating apparatus 200 according to the second embodiment, and FIG. 9 shows a processing flow thereof. The utterance direction estimation apparatus 200 includes a microphone array 101 including M (M is an integer of 2 or more) microphones 101-1 to 101-M, an AD converter 12, a frequency domain converter 13, and a correlation matrix calculator. 102, an utterance position estimation unit 405, a base matrix generation unit 400, a correlation matrix decomposition unit 410, a frequency averaging processing unit 420, and a front / horizontal generation unit 430. Among these, the microphone array 101, the AD conversion unit 12, the frequency domain conversion unit 13, and the correlation matrix calculation unit 102 are the same as those described in the first embodiment. Description of is omitted.

  The speaker position estimation unit 405 estimates the position r of the speaker. There are a plurality of methods for estimating the position r of the speaker. For example, it may be estimated from the time difference between the microphone outputs output from the AD converter 12, or image information is obtained from an imaging device (not shown) and the image is analyzed. And may be estimated. Further, information on the position r of the speaker may be obtained from the outside.

[Base matrix generator]
The base matrix generation unit 400 receives the position information r and the frequency domain signal X _M (ω, n) of each microphone output from the frequency domain conversion unit 13 as input, and the basis number D that is an axis for matrix decomposition. A base matrix is generated (S400). Here, the base number D does not need to be the same as the number of microphones M, and is an integer of 2 or more. Hereinafter, a method for generating the basis matrix UU (ω, k) will be described.

First, the first basis uu ₁ (ω, k) = [uu ₁₁ (ω, k),..., Uu _1M (ω, k)] is designed for each frequency ω and frame k. Several methods are conceivable for designing uu ₁ (ω, k). Two types of methods will be described.

One possible method is to simulate the transfer characteristics of the direct sound path. The method uses a frequency expression H ^ H (ω, r) = [H ^ H ₁ (ω, r),..., H ^ H _M (ω , R)]. _{Here, H ^ H 1 ~H ^ H} M is the transfer function representing the transmission characteristic between the sound sources and each microphone. For each frame k, the speaker's position r estimated by the speaker position estimation unit 405 is used as an input, and uu ′ ₁ (ω, r) = [H ^ H ₁ (ω, r),. , H ^ H _M (ω, r)]. Here, as the transfer function, an actual measurement value prepared in advance may be used, or a value calculated by simulation may be used. Finally, uu ₁ (ω, k) is obtained by normalization as in equation (9).

Another method using an area enhancement filter is also conceivable. The method obtains position information r estimated by the speaker position estimation unit 405 for each frame k, and filters u ^to u ₁ (ω, k) = [u ^to u ₁₁ ( ω, k),..., u ^to u _1M (ω, k)] ^T is designed for each frame k and frequency ω. The area enhancement filters u ^to u ₁ (ω, k) can be obtained by, for example, calculating Expression (10).

Here, h ^to h ₁₁ (ω) = [h ^to h ₁₁₁ (ω),..., H ^to h _11M (ω)] ^T is a typical transfer characteristic of an area to be emphasized, and h ^to h _1q (ω) = [h ^to h _1q1 (ω),..., h ^to h _1qM (ω)] ^T (q is an integer of 2 or more and L or less) is a typical transfer characteristic of an area that is not desired to be emphasized, and L−1 are prepared. To do. L is an integer of 2 or more. Further, when L = M is not satisfied, the inverse matrix operation is not regular, and thus becomes a pseudo inverse matrix operation. Finally, uu ₁ (ω, k) is obtained by normalization as in equation (11).

Next, uu ₂ (ω, k),..., Uu _D (ω, k) of the basis number D is calculated based on the _first basis uu ₁ (ω, k) obtained by either of the above methods. Then, a basis matrix UU (ω, k) is generated. A method for generating this basis matrix UU (ω, k) will now be described.
First, a method of generating a basis matrix UU (ω, k) with orthonormal basis is conceivable. One of the methods generates a base orthogonal to the vector uu ₁ (ω, k) by, for example, the Gram Schmitt method. It is possible to generate D−1 vectors uu ₂ (ω, k),..., Uu _D (ω, k) by repeating the calculations of Expression (12) and Expression (13) D−1 times.

Here, χ _n (ω, k) is an initial value of the n-th basis uu _n (ω, k), and may be given randomly so that ‖u _n (ω, k) ‖ ≠ 0. In addition, uu ₂ (ω, k) is sequentially generated up to uu _M (ω, k).
In addition, the base matrix UU (ω, k) can be generated using an area enhancement filter. The method generates a filter that emphasizes an area different from the area emphasized by the first basis uu ₁ (ω, k) already generated, and calculates the basis uu ₂ (ω, k) by the calculation of Expression (14). ,..., Uu _D (ω, k) is obtained.

Here, h ^to h _n1 (ω) = [h ^to h _n11 (ω),..., H ^to h _n1M (ω)] ^T is a typical transfer characteristic of an area to be emphasized by the nth base, h ^to h _nq (ω) = [h ^to h _nq1 (ω),..., h ^to h _nqM (ω)] ^T (q is an integer of 2 or more and L or less) is a representative area where the nth base is not emphasized L-1 are prepared. L is an integer of 2 or more. Further, when L = M is not satisfied, the inverse matrix operation is not regular, and thus becomes a pseudo inverse matrix operation. Then, uu _n (ω, k) is obtained by normalizing as in Expression (15).

[Correlation matrix decomposition unit]
Correlation matrix decomposition section 410 has correlation matrix R (ω, k) and basis matrix UU (ω, k) = [uu ₁ (ω, k),..., Uu _D (ω, k) for each frequency ω and frame k. )] With ^T as an input, a component sequence L (ω, k) = [L ₁ (ω, k),..., L _D (ω, k)] is output (S410).
The component sequence L (ω, k) is a feature quantity effective for front / side determination obtained by projecting the correlation matrix R (ω, k) onto the base matrix UU (ω, k). Projection is to calculate the degree of similarity between the correlation matrix R (ω, k) and the base matrix UU (ω, k). That is, the elements L ₁ (ω, k),..., L _D (ω, k) constituting the component sequence L (ω, k) are obtained by the calculation of Expression (16).

Here, H means complex conjugate transpose.
[Frequency averaging processing section]
The frequency averaging processing unit 420 receives the component sequence L (ω, k) output from the correlation matrix decomposing unit 410 and receives the frequency averaged component sequence L ⁻ (k) averaged at the frequency ω for each frame k. ) = [L ₁ ⁻ (k),..., L _D ⁻ (k)] is output (S420). Frequency averaging component column ^L - each element of (k) can be expressed by equation (17).

Here, | F ₃ | is the total number of frequency indexes used in the frequency averaging process.
Front-lateral determining unit 430, frequency averaged frequency averaged component column L ^- as input (k), the speaker for each frame k is whether a state of "frontal", or the "horizontal" Whether it is in a state is determined and output (S430). The front / side orientation determination unit 430 receives the first averaged eigenvalue aλ ₁ (k) only as an input from the frequency averaged component sequence L ⁻ (k). 203. Processing and theoretical background are the same as those of the front / side orientation determination unit 203 of the first embodiment.

  As described above, it is possible to estimate the speaker direction without using the method of decomposing the spatial correlation matrix R (ω, k) using the eigenvalue decomposition method. The second embodiment is an example in which the speaker is only determined to be “front-facing” or “landscape-oriented”, but it is also possible to determine the left-right orientation. Next, a description will be given of a third embodiment in which the horizontal direction can be determined.

[Third Embodiment]
FIG. 11 shows a functional configuration example of the speaker orientation estimating apparatus 300 according to the third embodiment. The utterance direction estimation apparatus 300 includes, in the functional configuration of the utterer direction estimation apparatus 200 of the second embodiment, an eigenvalue decomposition unit 201 of the utterer direction estimation apparatus 100 of the first embodiment, a first eigenvector power calculation unit 104, Functional configurations of a first frequency averaging processing unit 105 and a left / right direction determination unit 106 are added. Further, the front / side orientation determination unit 110 is different from the speaker orientation estimation device 200 in that the left / right direction of the speaker's utterance direction is also determined by using the left / right direction determination result as an input.

  Although the eigenvalue decomposition unit 201, the first eigenvector power calculation unit 104, the first frequency averaging processing unit 105, and the left / right direction determination unit 106 have been described, the configuration of the third embodiment is clearly described. Therefore, only the function of each part will be described again, and the description referring to the processing flow will be omitted.

  The eigenvalue decomposition unit 201 includes a correlation matrix R (ω, k), an eigenvalue matrix that is a diagonal matrix having the squares of M eigenvalues as diagonal elements, and M eigenvectors corresponding to the eigenvalues. The eigenvector matrix is decomposed and an eigenvector corresponding to the maximum eigenvalue (hereinafter referred to as “first eigenvector”) is output.

  The first eigenvector power calculation unit 104 calculates the power for each of M elements constituting the first eigenvector, and outputs M power elements. The first frequency averaging processing unit 105 calculates an average value for each of the M power elements generated for each frequency, and outputs M averaged power elements. The left / right direction determination unit 106 calculates a ratio of two averaged power elements corresponding to two arbitrary microphones among the M averaged power elements, and compares the ratio with a predetermined threshold value. Then, it is determined whether the speaker speaks leftward or rightward with respect to the microphone array, and the left / right direction determination result is output.

  The front / horizontal direction determination unit 110 receives the left / right direction determination result output from the left / right direction determination unit 106 as an input, and determines whether or not the speaker's utterance direction is the front direction for each frame and whether the utterance is leftward or rightward. To do.

When the configuration of the utterance direction estimation device of each of the above embodiments is realized by a computer, the processing contents of the functions that each device should have are described by a program. The processing functions are realized on the computer by executing the program on the computer. In this case, at least a part of the processing content may be realized by hardware.
Further, the various processes described above are not only executed in time series according to the description, but may be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. In addition, it can change suitably in the range which does not deviate from the meaning of this invention.

[Verification of effects]
In the sound collection environment shown in FIG. 12 (a), the effect of the present invention was verified under the conditions shown in FIG. 12 (b). The definition of the utterance direction is as shown in FIG.

FIG. 13 shows the relationship between each utterance direction and the first to seventh averaged eigenvalues aλ _i (k). Here, the second to seventh averaged eigenvalues are obtained by the same method as the first averaged eigenvalue. FIG. 13A shows the case where the reverberation time is 250 msec, and FIG. 13B shows the case where the reverberation time is 400 msec. In any reverberation time, the first averaged eigenvalue varies greatly depending on the speech direction, but the second to seventh averaged eigenvalues have a small difference depending on the speech direction. The first eigenvalue is the largest when it is 0 ° (frontward) and the smallest when it is ± 90 ° (laterally). From this verification result, it can be seen that the configuration of the present invention for determining the front / sideways orientation based on the first averaged eigenvalue shown in the second embodiment is valid and effective.

  FIG. 14 shows a comparison between the utterance direction in the left-right direction estimated by the configuration of the first embodiment and the actual utterance direction. FIG. 14A shows the case where the reverberation time is 250 msec, and FIG. 14B shows the case where the reverberation time is 400 msec. In any reverberation time, a correct answer rate of 75% or more was obtained. From this verification result, it can be seen that the configuration of the present invention for determining the left-right direction from the two power elements of the first eigenvector shown in the first embodiment is valid and effective.

[Service application example]
FIG. 15 is a structural example of a service in which the present invention is incorporated into an audio conference terminal. Assume that the conference hall A and the conference hall B are connected by a voice terminal through a network. Speech direction information is extracted from a voice signal picked up by a microphone attached to the voice conference terminal, and transmitted to the other party along with the voice information. By presenting the utterance direction information as visual information on the other party side, it is possible to convey the situation of the place that is difficult to convey only with the voice information.

  In addition, the speaker orientation estimation technology can be applied to a minutes system that records the state of communication in a conference using video and audio. The speaker orientation estimation technology can tag the recorded voice and video as to who spoke to whom, which is useful for organizing minutes. In addition, services performed by detecting face orientation in images, for example, surveillance / crime prevention services used with surveillance cameras and intercoms, and services that determine whether or not attention is paid to advertisements using digital signage (digital signage) Etc. can also be replaced with an audio signal output.

Claims (9)

An AD converter for converting analog audio signals picked up by M microphones (M is an integer of 2 or more) into digital audio signals;
A frequency domain transforming unit that transforms the digital audio signal from a time domain to a frequency domain in units of frames of a plurality of samples;
A correlation matrix calculation unit that sequentially generates and outputs an M × M correlation matrix representing the correlation between the digital audio signals converted into the frequency domain for each frequency as an expected value for each of a plurality of frames;
A base matrix generation unit that generates a base matrix composed of D (D is an integer of 2 or more) bases by using the digital speech signal and speaker position estimation information as inputs;
A correlation matrix decomposing unit that receives the correlation matrix and the base matrix for each frame, and projects the correlation matrix onto the base matrix to obtain a component sequence that is an effective feature quantity for front / lateral determination;
With the above component sequence as an input, a frequency averaging processing unit that outputs a frequency averaged component sequence averaged by frequency for each frame;
Using the frequency averaged component sequence as an input, a front / side orientation determination unit that estimates whether the direction of the speaker's utterance is the front direction for each frame, and
An utterance direction estimation device comprising: In the utterance direction estimation apparatus according to claim 1,
Furthermore,
The correlation matrix is decomposed into an eigenvalue matrix that is a diagonal matrix having the squares of the M eigenvalues as diagonal elements, and an eigenvector matrix composed of M eigenvectors corresponding to the eigenvalues, and the maximum eigenvalue is obtained. An eigenvalue decomposition unit that outputs a corresponding eigenvector (hereinafter referred to as “first eigenvector”);
A first eigenvector power calculator that calculates power for each of M elements constituting the first eigenvector and outputs M power elements;
A first frequency averaging processor that calculates an average value for each of the M power elements generated for each frequency and outputs M averaged power elements;
Of the M averaged power elements, the ratio of two averaged power elements corresponding to any two microphones is taken and compared with a predetermined threshold value so that the speaker can A left-right orientation determination unit that determines whether the speech is directed leftward or rightward with respect to the array and outputs a left-right orientation determination result;
Comprising
The front / side orientation determination unit receives the left / right orientation determination result as input and determines whether or not the direction of the speaker's utterance is the front direction for each frame and whether the utterance is uttered leftward or rightward. An utterance direction estimation device characterized by being. In the utterance direction estimation apparatus according to claim 1 or 2,
The basis matrix generation unit obtains a first basis of the basis matrix by a method of simulating transfer characteristics of a direct sound path, obtains a plurality of basis vectors orthogonal to each other based on the first basis, An utterance direction estimation apparatus characterized by generating a base matrix for decomposing a correlation matrix. In the utterance direction estimation apparatus according to claim 1 or 2,
The basis matrix generation unit obtains a first basis of the basis matrix by an area enhancement filter, and provides a filter for enhancing sounds in an area different from the first basis, and decomposes the correlation matrix An utterance direction estimation apparatus characterized by generating a base matrix using the filter. An AD conversion process in which an AD conversion unit converts analog audio signals picked up by M (M is an integer of 2 or more) microphones into digital audio signals;
A frequency domain transforming process in which the frequency domain transforming unit transforms the digital audio signal from the time domain to the frequency domain in units of frames composed of a plurality of samples;
Correlation matrix calculation unit that sequentially generates and outputs an M × M correlation matrix representing the correlation between the digital audio signals converted into the frequency domain for each frequency as an expected value for each of a plurality of frames. Process,
A basis matrix generation process in which a basis matrix generation unit generates a basis matrix having a basis number D (D is an integer of 2 or more) by using the digital speech signal and speaker position estimation information as inputs;
Correlation matrix decomposition process for obtaining a component sequence that is an effective feature quantity for front / horizontal determination by inputting the correlation matrix and the base matrix for each frame and projecting the correlation matrix onto the base matrix for each frame When,
A frequency averaging processing step in which the frequency averaging processing unit outputs the frequency averaged component sequence averaged by frequency for each frame, using the component sequence as an input;
A front / side orientation determination unit that receives the frequency averaged component sequence as an input and estimates whether the direction of the speaker's utterance is the front direction for each frame;
Speech direction estimation method including In the speech direction estimation method according to claim 5,
Furthermore,
An eigenvalue decomposition unit decomposes the correlation matrix into an eigenvalue matrix that is a diagonal matrix whose diagonal elements are the squares of the M eigenvalues and an eigenvector matrix composed of M eigenvectors corresponding to the eigenvalues. Eigenvalue decomposition process for outputting an eigenvector corresponding to the largest eigenvalue (hereinafter referred to as “first eigenvector”);
A first eigenvector power calculation unit that calculates power for each of M elements constituting the first eigenvector and outputs M power elements;
A first frequency averaging processor that calculates an average value for each of the M power elements generated for each frequency and outputs M averaged power elements;
The left-right direction determination unit takes a ratio of two averaged power elements corresponding to two arbitrary microphones out of the M averaged power elements, and compares it with a predetermined threshold value. , A left-right direction determination process for determining whether the speaker speaks leftward or rightward with respect to the microphone array and outputs a left-right direction determination result;
Including
The front / horizontal determination process is a process of determining whether the direction of the speaker's utterance is the front direction for each frame with the result of the determination of the left / right direction as an input, and whether the speaker is speaking leftward or rightward. A speech direction estimation method characterized by being. In the speech direction estimation method according to claim 5 or 6,
In the basis matrix generation process, the first basis of the basis matrix is obtained by a method of simulating the transfer characteristic of the direct sound path, and a plurality of basis vectors orthogonal to each other based on the first basis are obtained, An utterance direction estimation method comprising generating a base matrix for decomposing a correlation matrix. In the speech direction estimation method according to claim 5 or 6,
In the base matrix generation process, a first base of the base matrix is obtained by an area emphasis filter, and a filter for emphasizing sounds in a different area from the first base is provided, and the correlation matrix is decomposed. A utterance direction estimation method, wherein a basis matrix is generated using the filter.   An apparatus program for causing a computer to function as the speech direction estimating apparatus according to claim 1.

Images