JP5235605B2 - Utterance direction estimation apparatus, method and program - Google Patents

Description

  The present invention relates to a technique for estimating the utterance direction of a speaker from an audio signal input to a microphone.

  A system for exchanging voice information such as a telephone or a voice conference terminal is generally called a voice communication system. In the video conference system, the video is added to the audio information and presented, so that the situation of the place is easily transmitted, but in the audio communication system, it is difficult to grasp the situation of the other party. One of the information on the other party's situation is utterance direction information. By receiving this information from the other party, it is possible to grasp the direction in which the speaker is speaking and to facilitate communication.

  Conventional techniques for estimating such speech direction information are disclosed in Non-Patent Documents 1 and 2 and the like, and a configuration example is shown in FIG. The speech direction estimation apparatus 10 in this configuration example estimates speech direction information as follows.

(i) The voice from the speaker 1 is picked up using M (M is an integer of 2 or more) microphones 11-1,. The collected analog signal is converted into a digital signal vx (t) = [x ₁ (t),..., X _M (t)] ^T by the AD converter 12. Here, t represents an index of discrete time.

(ii) The frequency domain transform unit 13 receives a set (frame) of the digital signals composed of a plurality of samples as an input, and performs frequency domain signal VX (ω, n) = [X ₁ (ω, n) by fast Fourier transform or the like. _{, ···, X M (ω,} n)] to convert to ^T. Here, ω represents a frequency index, and the total number of frequency indexes is Ω. N represents the index of the frame.

(iii) The fixed beamformer design unit 14 sets the fixed beamformer VG (ω, r, θ) = [G ₁ (ω, r, θ),..., G _M (for each speaker position and direction. ω, r, θ)] ^T is designed. G _i (ω, r, θ) is a coefficient to be multiplied by the frequency component x _i (ω, n) of the i-th microphone in order to emphasize / suppress the sound source at the speaker position r and the speech direction θ.
In designing, acoustic propagation characteristics between a sound source and a microphone VH (ω, r, θ) = [H ₁ (ω, r, θ),..., H _{M for} each predetermined speaker position and direction. (ω, r, θ)] ^T is obtained using simulation values or actual measurement values. Here, H _i (ω, r, θ) represents an acoustic propagation characteristic between the sound source at the speaker position r and the speech direction θ and the i-th microphone.
The fixed beamformer VG (ω, r, θ) is designed as a value that satisfies the expressions (1) and (2) representing the relationship with the acoustic propagation characteristics.
VH (ω, r _T , θ _T ) ^H · VG (ω, r _T , θ _T ) = 1 (1)
VH (ω, r _U , θ _U ) ^H · VG (ω, r _T , θ _T ) = 0 (2)
Expressions (1) and (2) emphasize the output power of the speaker position r _T and the speech direction θ _T and suppress the output power of the other speaker positions r _U and the speech direction θ _U. It shows that VG (ω, r, θ) is designed.

(iv) In the product-sum calculation unit 15, the frequency domain signal VX (ω, n) = [X ₁ (ω, n),..., X _M (ω, n)] ^T and the fixed beamformer VG (ω , r, θ) = [G ₁ (ω, r, θ),..., G _M (ω, r, θ)] ^T as an input, for each frequency ω, speaker position r, and speech direction θ. Multiplying the frequency component X _i (ω, n) corresponding to each microphone by the coefficient G _i (ω, r, θ) of the fixed beamformer, and adding the obtained M components, the output Y (ω, n, r, θ) is calculated. This calculation is synonymous with calculating Y (ω, n, r, θ) = VG (ω, r, θ) ^H · VX (ω, n).

(v) The power calculator 16 calculates and outputs power | Y (ω, n, r, θ) | ² from the output Y (ω, n, r, θ) from the product-sum calculator 15.

(vi) The frequency averaging processing unit 17 averages the power | Y (ω, n, r, θ) | ² output from the power calculation unit 16 by frequency, and calculates AY (n, r, θ). obtain. In this calculation, if F ₀ is defined as the frequency index used in the averaging process, and | F ₀ | is defined as the total number of frequency indexes,
Is equivalent to calculating Note that F ₀ satisfies Ω ≧ | F ₀ |.
(vii) The sound source direction selection unit 18 searches for the speaker position r and the speech direction θ where the power AY (n, r, θ) averaged by frequency for each frame is maximum, and the power AY (n , r, θ) is determined as the estimated speech direction θ _out (n).
Hiroshi Nakajima, “Extended Beamforming for Estimating Sound Source Direction”, Proceedings of the Acoustical Society of Japan, September 2005, p.619-620 Hiroshi Nakajima and 8 others, “Sound source directivity estimation using extended beamforming”, Acoustical Society of Japan Proceedings, September 2005, p.621-622

The following two points can be cited as problems of the prior art.
(i) To deal with utterances at any position and to estimate the direction of utterance with high accuracy, a large number of microphones are required and the position of the microphones must be devised. As the output power | Y (ω, n, r, θ) | ² of the fixed beamformer designed for each position and speech direction is different, the speech direction can be estimated with higher accuracy. However, assuming a sound source with a strong directivity in front of the mouth, such as a sound wave radiated from the mouth of the speaker, as shown in FIG. 16, it is necessary to collect sound so as to surround the speaker with a large number of microphones. Depending on the speaker position / speech direction, there is no difference in the output power of the fixed beamformer, and the speech direction estimation error increases (for example, in the experiment of Non-Patent Document 2, 64 microphones are used). Therefore, in order to reduce the error, a large number of microphones are required, the apparatus becomes large, and it is difficult to use it by attaching it to a portable apparatus such as a telephone or an audio conference terminal.

(ii) High reverberation direction estimation performance cannot be obtained in a reverberant environment where the reverberation time (the time from the direct wave arrival time to the 60 dB attenuation from the direct wave pickup power) is 250 msec or more. Below, since many strong reflected waves are mixed, it is difficult to design the acoustic propagation characteristics VH (ω, r, θ) with high accuracy. Therefore, ambiguity occurs in the output of the fixed beamformer, and the estimation accuracy deteriorates. For example, in an actual environment room that is not subjected to low reverberation processing, reverberation time is generally about 250 to 500 msec, so that accurate estimation is difficult.

  An object of the present invention is to estimate a speech direction, which does not require a large number of microphones so as to surround a speaker and can appropriately estimate a speech direction even in a reverberant environment with a reverberation time of 250 msec or more. To provide an apparatus, a method, and a program.

  In order to solve the above-described problem, according to the first aspect of the present invention, the speech direction estimating apparatus generates a correlation matrix representing a correlation between audio signals collected by each of a plurality of microphones constituting a microphone array. Then, from the eigenvector of this correlation matrix and at least the maximum eigenvalue of this correlation matrix, it is estimated whether the speaker speaks in front or left direction with respect to the microphone array.
  In order to solve the above-mentioned problem, according to the second aspect of the present invention, the speech direction estimating apparatus is a digital audio signal picked up by M (M is an integer of 2 or more) microphones constituting a microphone array. And a M × M correlation matrix representing a correlation between each frequency converted from the time domain to the frequency domain, and each digital audio signal converted to the frequency domain for each frequency. A correlation matrix calculation unit that obtains the result of comparing the evaluation value based on the eigenvector of the correlation matrix with a predetermined threshold, and the result of comparing the evaluation value based on at least the maximum eigenvalue of the correlation matrix with the predetermined threshold; And a front / side orientation determination unit that determines whether the speaker has spoken frontward or leftward / rightward with respect to the microphone array and outputs the result.
  In order to solve the above-described problem, according to a third aspect of the present invention, a speech direction estimating apparatus is a digital audio signal picked up by M (M is an integer of 2 or more) microphones constituting a microphone array. And a M × M correlation matrix representing a correlation between each frequency converted from the time domain to the frequency domain, and each digital audio signal converted to the frequency domain for each frequency. Obtained by normalizing the first eigenvalue and obtaining an eigenvector (hereinafter referred to as “first eigenvector”) corresponding to the maximum eigenvalue of the correlation matrix (hereinafter referred to as “first eigenvalue”). An eigenvalue decomposition unit that obtains a first normalized eigenvalue and a first eigenvalue that outputs M power elements by calculating power for each of the M elements constituting the first eigenvector One of the ratios of the two power elements corresponding to any two microphones among the M power elements, or the frequency average of the M power elements By comparing the ratio of two averaged power elements corresponding to two arbitrary microphones out of M averaged power elements with a predetermined threshold value, the speaker is placed in the microphone array. A left-right direction determination unit that determines whether the speech is directed to the left or the right, a first normalized eigenvalue, or a first averaged eigenvalue that is a frequency average of the first normalized eigenvalue and a predetermined threshold value; To determine whether the speaker speaks in front or side to the microphone array, and if it is determined to be front-facing, the determination result that the speaker has spoken in front is output. And And a front / horizontal direction determination unit that outputs the determination result of the left / right direction determination unit as it is.
  In order to solve the above problem, according to a fourth aspect of the present invention, an utterance direction estimation method uses a correlation matrix representing a correlation between audio signals collected by each of a plurality of microphones constituting a microphone array. Then, from the eigenvector of this correlation matrix and at least the maximum eigenvalue of this correlation matrix, it is estimated whether the speaker speaks in front or left direction with respect to the microphone array.
  In order to solve the above problems, according to a fifth aspect of the present invention, a speech direction estimation method includes a digital audio signal collected by M (M is an integer of 2 or more) microphones constituting a microphone array. And a M × M correlation matrix representing the correlation between each digital audio signal converted into the frequency domain for each frequency A correlation matrix calculation step for obtaining an evaluation value, a result obtained by comparing an evaluation value based on an eigenvector of the correlation matrix with a predetermined threshold value, and a result obtained by comparing an evaluation value based on at least the largest eigenvalue of the correlation matrix with a predetermined threshold value. Step to determine whether the speaker speaks in front of the microphone array or in the left-right direction and outputs the result. , Comprising a.
  In order to solve the above problems, according to a sixth aspect of the present invention, a digital audio signal picked up by M (M is an integer of 2 or more) microphones constituting a microphone array is composed of a plurality of samples. A frequency domain conversion step for converting from the time domain to the frequency domain in units of frames, and a correlation matrix calculation step for obtaining an M × M correlation matrix representing the correlation between the respective digital audio signals converted into the frequency domain for each frequency And an eigenvector (hereinafter referred to as “first eigenvector”) corresponding to the maximum eigenvalue of the correlation matrix (hereinafter referred to as “first eigenvalue”), and a first normalized eigenvalue obtained by normalizing the first eigenvalue as The obtained eigenvalue decomposition step and the first eigenvector that outputs the M power elements by calculating the power for each of the M elements constituting the first eigenvector. Toll power calculation step and any one of ratios of two power elements corresponding to arbitrary two microphones among M power elements, or frequency average of M power elements By comparing the ratio of the two averaged power elements corresponding to any two microphones out of the M averaged power elements with a predetermined threshold, the speaker can A left-right direction determination step for determining whether the utterance is directed to the left or the right, and a first normalized eigenvalue or a first averaged eigenvalue that is a frequency average of the first normalized eigenvalue and a predetermined threshold value. By comparing, it is determined whether the speaker has spoken frontward or sideways with respect to the microphone array, and if it is determined to be frontal, the determination result that the speaker has spoken frontward is output, A front / horizontal determination step that outputs the determination result in the left / right direction determination step as it is when it is determined to be in the horizontal direction.

  According to the speech direction estimating apparatus of the present invention, it is not necessary to arrange a large number of microphones so as to surround a speaker, and it is possible to appropriately estimate the speech direction even in a reverberant environment where the reverberation time is 250 msec or more. It becomes.

[First Embodiment]
FIG. 1 shows a functional configuration example of the speech direction estimating apparatus 100 of the present invention, and FIG. 2 shows a processing flow example thereof. The utterance direction estimation apparatus 100 estimates whether the utterance direction is leftward or rightward with respect to the microphone array.

  The utterance direction estimation apparatus 100 includes a microphone array 101 including M microphones (M is an integer of 2 or more) 101-1 to 101-M, an AD conversion unit 12, a frequency domain conversion unit 13, and a correlation matrix calculation unit. 102, an eigenvalue decomposition unit 103, a first eigenvector power calculation unit 104, a first frequency averaging processing unit 105, and a left / right direction determination unit 106. Among them, the AD conversion unit 12 and the frequency conversion unit 13 are the same as those used in the speech direction estimation apparatus 10 described in the background art.

  In the prior art, as shown in FIG. 16, it was necessary to arrange a large number of microphones so as to surround the speaker. However, in the present invention, M microphones 101-1 to 101-M can be provided. It is sufficient to arrange them closely. Although the number of microphones constituting such a microphone array 101 is not too large, according to the configuration of the present invention described below, it is possible to estimate the utterance direction with two or more microphones. The arrangement may be two-dimensional or three-dimensional. By adopting a configuration in which a small number of microphones are arranged densely in this way, it can be attached to a portable device such as a telephone or an audio conference terminal, and the direction of speech by surrounding speakers can be estimated It becomes. The speaker speaks around the microphone array 101. FIG. 3 shows an image of a speaker talking around a microphone array 101 composed of seven microphones as viewed from above, and the direction of the arrow is the direction of speech. FIG. 3 (a) shows a state where a speaker is speaking in a front direction at each position, and FIG. 3 (b) shows a state where a speaker is speaking in a horizontal direction.

The AD conversion unit 12 converts the analog voice signals uttered by the speaker 1 collected by the M microphones 101-1 to 101 -M into digital voice signals x ₁ (t),..., X _M (t (S1). Here, t represents an index of discrete time.

The frequency domain transform unit 13 receives as input a set (frame) of the above digital speech signals composed of a plurality of discrete time samples, and performs frequency domain digital speech signals X ₁ (ω, n),. It is converted into X _M (ω, n) and output (S2). Here, n represents a frame index, and ω represents a frequency index. The total number of frequency indexes is Ω.

The correlation matrix calculation unit 102 receives the digital audio signals X ₁ (ω, n),..., X _M (ω, n) in the frequency domain as inputs, and an M × M correlation matrix R representing the correlation between the signals. (ω, k) is sequentially generated and output for each frequency ω by equation (3) (S3).
R (ω, k) = E [VX (ω, n) ・ VX ^H (ω, n)] (3)
Here, VX (ω, n) = [X ₁ (ω, n),..., X _M (ω, n)] ^T
Note that H represents conjugate transposition, and E calculates VX (ω, n) · VX (ω, n) ^H for each frame, and then calculates an expected value for each L frame by averaging processing or the like. It is an operator. That is, the correlation matrix is sequentially output at a rate of once per L frame, and k represents an output index of the correlation matrix. L is preferably an integer greater than or equal to M.

The eigenvalue decomposition unit 103 receives the correlation matrix R (ω, k) as input, and first, M eigenvalues λ ₁ (ω, n),..., Λ _M (ω, n) so as to satisfy Equation (4). ) Eigenvalue matrix Λ (ω, k), which is a diagonal matrix with each square as a diagonal element, and M eigenvectors vv ₁ (ω, n),..., Vv _M (ω, k) Is decomposed into eigenvector matrix V (ω, k) by the eigenvalue decomposition method.
R (ω, k) = V (ω, k) ・ Λ (ω, k) ・ V ^H (ω, k) (4)
Where Λ (ω, k) = diag [λ ₁ ² (ω, k),..., Λ _M ² (ω, k)]
λ ₁ (ω, k) ≧ λ ₂ (ω, k) ≧ ・ ・ ・ ≧ λ _M (ω, k)
V (ω, k) = [vv ₁ (ω, k),..., Vv _M (ω, k)] ^T
vv _i (ω, k) = [v _{i, 1} (ω, k),..., v _{i, M} (ω, k)]
Note that diag [•] is an operator having the components in [•] as elements of a diagonal matrix.
Then, the first eigenvector vv ₁ (ω, k) corresponding to the first eigenvalue λ ₁ (ω, k) which is the maximum eigenvalue is output (S4).

The first eigenvector power calculation unit 104 receives the first eigenvector vv ₁ (ω, k) as an input, and v _1,1 (ω, k),... Constituting the first eigenvector vv ₁ (ω, k). For M elements of v _{1, M} (ω, k), the power is calculated by the equation (5), respectively, and M power elements pv _1,1 (ω, k),..., pv _{1, M} (ω, k) is output (S5).
pv _{1, i} (ω, k) = | v _{1, i} (ω, k) | (5)

The first frequency averaging processing unit 105 formulas _M power elements pv _1,1 (ω, k),..., Pv _{1, M} (ω, k) generated for each frequency ω, respectively. The average value is calculated according to (6) and M averaged power elements apv _1,1 (k),..., Apv _{1, M} (k) are output (S6).
F ₁ is an index of frequencies used for averaging, | F ₁ | is a total number of frequency indexes, and F ₁ is appropriately set so as to satisfy Ω ≧ | F ₁ |.

The left / right direction determination unit 106 receives M averaged power elements apv _1,1 (k),..., Apv _{1, M} (k) as input, and determines whether the left direction is spoken or the right direction is spoken. The result is output (S7). The left-right orientation determination is performed by using two averaged power elements apv _{1, α} (k), apv corresponding to two microphones 101-α and 101-β among the M microphones constituting the microphone array 101. This is done by taking a ratio of _{1, β} (k) and comparing it to a predetermined threshold value thr1. FIG. 4 illustrates an example of the determination image in the horizontal direction. In this example, the case where the speech is directed toward the midpoint between the microphone α and the microphone β shown in FIG. 4A is apv _{1, α} (k) / apv _{1, β} (k) = thr1, and this direction is set as follows. The reference speech direction in the left-right direction is determined based on the magnitude relationship between apv _{1, α} (k) / apv _{1, β} (k) and thr1. Specifically, since the rate of attenuation of apv _{1, β} (k) is larger than apv _{1, α} (k) as it goes to the left, apv _{1, α} (k) / apv _{1, β} (k )> Thr1 can be determined to be leftward (FIG. 4 (b)), and the more rightward is, the greater the rate at which apv _{1, α} (k) attenuates than apv _{1, β} (k). Therefore, when apv _{1, α} (k) / apv _{1, β} (k) <thr1, it can be determined to be rightward (FIG. 4 (c)). Note that the two microphones have the widest left-right spacing with respect to the speaker's position so that the average power elements apv _{1, α} (k) and apv _{1, β} (k) tend to differ. It is desirable to choose.

  A theoretical background capable of determining the left-right direction of speech with such a configuration will be described. FIG. 5 shows the propagation characteristics of the audio signal in the time domain. Propagation characteristics can be broadly divided into three types: direct wave, initial reverberation, and rear reverberation. In the time zone in which direct wave and initial reverberation are observed, the directivity with respect to the microphone array composed of a plurality of microphones. It is known that waves with In particular, in the initial reverberation time zone (after the arrival of the direct wave, the time from the direct wave pickup power to the attenuation of 10 dB), a strong reflected wave with directionality is mixed, but the power of this reflected wave depends on the direction of speech. Change. Specifically, since the direct wave power increases as the utterance direction is the front direction, the reflected wave power decreases, and the direct wave power decreases as it is in the horizontal direction. The power of will increase. The present invention uses such a property to estimate the speech direction.

This will be described below in accordance with the configuration of the present invention. FIG. 6 shows how the utterance state in front and side affects each eigenvalue λ _i (ω, k) of the correlation matrix R (ω, k). Here, a case where a microphone array is configured by three microphones is illustrated. When facing the front, many direct waves reach the microphone array, and the arrival rate of the reflected waves is relatively low. Therefore, as shown in FIG. Compared to the basis vector group to be expressed, it has a large power. At this time, the first eigenvalue λ ₁ (ω, k) is significantly larger than the second eigenvalue λ ₂ (ω, k) and the third eigenvalue λ ₃ (ω, k). On the other hand, in the case of the horizontal orientation, the direct waves that reach the microphone array are reduced, so that more reflected waves arrive accordingly. For this reason, as shown in FIG. 6B, the power of the basis vectors expressing the direct wave decreases, and the power of the basis vector group expressing the reflected wave increases. At this time, the first eigenvalue λ ₁ (ω, k) is smaller than that in the front direction, and conversely, the second eigenvalue λ ₂ (ω, k) and the third eigenvalue λ ₃ (ω, k) are in the front direction. Larger than the case. FIG. 7 shows an image of the difference that occurs in each eigenvalue between the case of facing forward and the case of facing sideways.

  From the above, it can be seen that the power of the basis vector expressing the direct wave is significantly reflected in the first eigenvalue. If so, the strength of each power element corresponding to the M microphones of the first eigenvector corresponding to the first eigenvalue is how strong the direct wave reaches each of the M microphones. It can be considered as a measure of The power at which direct waves reach each microphone varies depending on the direction of speech. Therefore, as described above, the ratio of the power elements of any two microphones in the reference utterance direction is set as a threshold (thr1), and the threshold and the power elements of the two microphones in a certain utterance direction It is possible to determine whether the utterance is uttered in the left direction or the utterance in the right direction with respect to the reference utterance direction from the magnitude relationship.

  As described above, according to the utterance direction estimation apparatus of the first embodiment, since a small number of microphones need only be densely arranged, a compact configuration can be achieved without enclosing a speaker with a large number of microphones. Become. In addition, since the reverberation is actively used, it is possible to appropriately estimate the utterance direction even in a reverberation environment where the reverberation time is 250 msec or more. In addition, since the eigenvalue decomposition process, which is the core of the process in the present invention, has a small amount of calculation, it is advantageous when it is incorporated in a device having a low CPU specification such as a portable terminal.

[Second Embodiment]
In the first embodiment, it is determined whether the utterance direction is leftward or rightward. However, the second embodiment further includes a front direction, and the utterance direction is frontal, leftward, or rightward. It is possible to determine whether or not.

FIG. 8 shows a functional configuration example of the speech direction estimating apparatus 200 of the present invention, and FIG. 9 shows a processing flow example thereof.
The utterance direction estimation apparatus 200 includes a microphone array 101 including M (M is an integer of 2 or more) microphones 101-1 to 101-M, an AD converter 12, a frequency domain converter 13, and a correlation matrix calculator. 102, an eigenvalue decomposition unit 201, a first eigenvector power calculation unit 104, a first frequency averaging processing unit 105, a left / right orientation determination unit 106, a second frequency averaging processing unit 202, and a front / side orientation determination unit 203. Among these, the components other than the eigenvalue decomposition unit 201, the second frequency averaging processing unit 202, and the front / side orientation determination unit 203 are the same as the components having the same names and symbols described in the first embodiment. Therefore, description of functions and processes is omitted.

The eigenvalue decomposition unit 201 performs the same processing as the eigenvalue decomposition unit 103 of the first embodiment, further normalizes the first eigenvalue λ ₁ (ω, k) by the equation (7), and obtains the first normalized eigenvalue nλ _1. (ω, k) is output (S8).

The second frequency averaging processing unit 202 calculates the average value of the first normalized eigenvalue nλ ₁ (ω, k) obtained for each frequency ω by the equation (8), and the first averaged eigenvalue aλ ₁ (k) is output (S9).

F ₂ is an index of frequencies used for averaging, | F ₂ | is the total number of frequency indexes, and F ₂ is appropriately set so as to satisfy Ω ≧ | F ₂ |.

The front / horizontal direction determination unit 203 receives the determination result from the left / right direction determination unit 106 and the first averaged eigenvalue aλ ₁ (k), and uses the first averaged eigenvalue aλ ₁ (k) as a predetermined threshold value thr2. If aλ ₁ (k) <thr2, the speaker determines that the speaker is facing sideways with respect to the microphone array, and otherwise determines that the speaker is facing frontward. If it is determined to be in the horizontal direction, the determination result in the left-right direction determination unit 106 is output as it is, and if not, the determination result that it is in the front direction is output (S10). FIG. 10 shows an example of the front / side orientation determination image. Here, thr2 may be arbitrarily set according to the environment and the position of the speaker. Since the first averaged eigenvalue aλ ₁ (k) is obtained for each frame group k, the determination result is also output for each frame group k.

  A theoretical background that can determine whether the speech direction is the front direction or the horizontal direction with such a configuration will be described. As described in the first embodiment, the power of the basis vector representing the direct wave is significantly reflected in the first eigenvalue. Specifically, in the case of the front direction, the power of the base vector expressing the direct wave shows a large value and the first eigenvalue also shows a large value. The first eigenvalue is also smaller than when facing the front. Therefore, by using the first eigenvalue as a front / side determination parameter, it is possible to appropriately determine that the first eigenvalue is front-facing if it is larger than a certain threshold value and that it is lateral if it is smaller.

  Thus, according to the utterance direction estimation device of the second embodiment, in addition to the effects of the configuration of the first embodiment, a section called front direction is further provided, and the utterance direction is front direction, left direction, or right direction. Therefore, communication with the other party via the network can be performed more smoothly.

  When the configuration of the utterance direction estimation device of each of the above embodiments is realized by a computer, the processing contents of the functions that each device should have are described by a program. The processing functions are realized on the computer by executing the program on the computer. In this case, at least a part of the processing content may be realized by hardware.

  Further, the various processes described above are not only executed in time series according to the description, but may be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. In addition, it can change suitably in the range which does not deviate from the meaning of this invention.

[Verification of effects]
In the sound collection environment shown in FIG. 11 (a), the effect of the present invention was verified under the conditions shown in FIG. 11 (b). The definition of the utterance direction is as shown in FIG.

FIG. 12 shows the relationship between each utterance direction and the first to seventh averaged eigenvalues aλ _i (k). Here, the second to seventh averaged eigenvalues are obtained by the same method as the first averaged eigenvalue. FIG. 12A shows the case where the reverberation time is 250 msec, and FIG. 12B shows the case where the reverberation time is 400 msec. In any reverberation time, the first averaged eigenvalue varies greatly depending on the speech direction, but the second to seventh averaged eigenvalues have a small difference depending on the speech direction. The first eigenvalue is the largest when it is 0 ° (frontward) and the smallest when it is ± 90 ° (laterally). From this verification result, it can be seen that the configuration of the present invention for determining the front / sideways orientation based on the first averaged eigenvalue shown in the second embodiment is valid and effective.

  FIG. 13 shows a comparison between the left and right utterance direction estimated by the configuration of the first embodiment and the actual utterance direction. FIG. 13A shows a case where the reverberation time is 250 msec, and FIG. 13B shows a case where the reverberation time is 400 msec. In any reverberation time, a correct answer rate of 75% or more was obtained. From this verification result, it can be seen that the configuration of the present invention for determining the left-right direction from the two power elements of the first eigenvector shown in the first embodiment is valid and effective.

[Service application example]
FIG. 14 is a structural example of a service in which the present invention is incorporated in an audio conference terminal. Assume that the conference hall A and the conference hall B are connected by a voice terminal through a network. Speech direction information is extracted from a voice signal picked up by a microphone attached to the voice conference terminal, and transmitted to the other party along with the voice information. By presenting the utterance direction information as visual information on the other party side, it is possible to convey the situation of the place that is difficult to convey only with the voice information.

The figure which shows the function structural example of the speech direction estimation apparatus of 1st Embodiment. The figure which shows the example of a processing flow of the speech direction estimation apparatus of 1st Embodiment. The image figure which shows the positional relationship of a microphone, a speaker, and a speech direction. The image figure about the method of determining left-right orientation. The figure which shows the propagation characteristic of an audio | voice signal in a time domain. The image figure which shows the correlation with an utterance direction, a base vector, and an eigenvalue / eigenvector. The image figure which shows the relationship between an utterance direction and an eigenvalue. The figure which shows the function structural example of the speech direction estimation apparatus of 2nd Embodiment. The figure which shows the example of a processing flow of the utterance direction estimation apparatus of 2nd Embodiment. The image figure about the method of judging front direction and sideways. The figure which shows the verification environment and conditions of an effect. The figure which shows the verification result about the relationship between an utterance direction and an eigenvalue. The figure which shows the verification result of left-right direction estimation. The figure which shows the service structural example which incorporated this invention in the audio conference terminal. The figure which shows the function structural example of the speech direction estimation apparatus by a prior art. The image figure which shows the relationship between the microphone and speaker by a prior art.

Claims (7)

A correlation matrix representing a correlation between audio signals picked up by each of a plurality of microphones constituting the microphone array is generated, and an utterer can perform the above processing from the eigenvector of the correlation matrix and at least the maximum eigenvalue of the correlation matrix. An utterance direction estimation device that estimates whether the utterance is directed to the microphone array from the front or the left and right .   A frequency domain conversion unit for converting a digital audio signal collected by M microphones (M is an integer of 2 or more) constituting a microphone array from a time domain to a frequency domain in units of frames composed of a plurality of samples;
  A correlation matrix calculator that obtains an M × M correlation matrix representing the correlation between the digital audio signals converted into the frequency domain for each frequency;
  The speaker uses the result of comparing the evaluation value based on the eigenvector of the correlation matrix with a predetermined threshold and the result of comparing the evaluation value based on at least the maximum eigenvalue of the correlation matrix with the predetermined threshold. A front / side determination unit that determines whether the microphone array is uttered in the front direction or the left / right direction and outputs the result;
An utterance direction estimation device comprising: A frequency domain conversion unit for converting a digital audio signal collected by M microphones (M is an integer of 2 or more) constituting a microphone array from a time domain to a frequency domain in units of frames composed of a plurality of samples;
A correlation matrix calculator that obtains an M × M correlation matrix representing the correlation between the digital audio signals converted into the frequency domain for each frequency;
Obtain an eigenvector (hereinafter referred to as “first eigenvector”) corresponding to the largest eigenvalue (hereinafter referred to as “first eigenvalue”) of the correlation matrix, and obtain a first normalized eigenvalue obtained by normalizing the first eigenvalue. An eigenvalue decomposition unit to obtain,
A first eigenvector power calculator that calculates power for each of M elements constituting the first eigenvector and outputs M power elements;
Of the M power elements, one of the ratios of two power elements corresponding to any two microphones, or M frequency averages of the M power elements. By comparing the ratio of two averaged power elements corresponding to any two microphones among the averaged power elements and a predetermined threshold value, the speaker can turn left with respect to the microphone array. A left-right direction determination unit that determines whether the utterance is spoken or rightward,
The first normalization eigenvalues, or by comparing the first averaging eigenvalues with a predetermined threshold value is a frequency average of the first normalized eigenvalue, speaker utterance frontally with respect to the microphone array If it is determined to be front-facing, the determination result that the utterance was directed to the front is output. If it is determined to be horizontal, the determination result from the left-right direction determination unit is output as it is. A front / side orientation determination unit,
An utterance direction estimation device comprising: A correlation matrix representing a correlation between audio signals picked up by each of a plurality of microphones constituting the microphone array is generated, and an utterer can perform the above processing from the eigenvector of the correlation matrix and at least the maximum eigenvalue of the correlation matrix. A speech direction estimation method for estimating whether the speech is directed to the microphone array from the front or the left and right .   A frequency domain conversion step of converting a digital audio signal collected by M microphones (M is an integer of 2 or more) constituting a microphone array from a time domain to a frequency domain in units of frames composed of a plurality of samples;
  A correlation matrix calculating step for obtaining an M × M correlation matrix representing the correlation between the digital audio signals converted into the frequency domain for each frequency;
  The speaker uses the result of comparing the evaluation value based on the eigenvector of the correlation matrix with a predetermined threshold and the result of comparing the evaluation value based on at least the maximum eigenvalue of the correlation matrix with the predetermined threshold. A front / side determination step for determining whether the microphone array is uttered in the front direction or the left / right direction and outputting the result;
An utterance direction estimation method comprising: A frequency domain conversion step of converting a digital audio signal collected by M microphones (M is an integer of 2 or more) constituting a microphone array from a time domain to a frequency domain in units of frames composed of a plurality of samples;
A correlation matrix calculating step for obtaining an M × M correlation matrix representing the correlation between the digital audio signals converted into the frequency domain for each frequency;
Obtain an eigenvector (hereinafter referred to as “first eigenvector”) corresponding to the largest eigenvalue (hereinafter referred to as “first eigenvalue”) of the correlation matrix, and obtain a first normalized eigenvalue obtained by normalizing the first eigenvalue. An eigenvalue decomposition step to obtain;
A first eigenvector power calculating step of calculating power for each of M elements constituting the first eigenvector and outputting M power elements;
Of the M power elements, one of the ratios of two power elements corresponding to any two microphones, or M frequency averages of the M power elements. By comparing the ratio of two averaged power elements corresponding to any two microphones among the averaged power elements and a predetermined threshold value, the speaker can turn left with respect to the microphone array. A left-right direction determination step for determining whether the utterance is spoken or the right direction,
The first normalization eigenvalues, or by comparing the first averaging eigenvalues with a predetermined threshold value is a frequency average of the first normalized eigenvalue, speaker utterance frontally with respect to the microphone array If it is determined to be front-facing, the determination result that the utterance was directed to the front is output, and if it is determined to be horizontal, the determination result in the left-right determination step is output as it is. Front / side orientation determination step,
An utterance direction estimation method comprising: Program for causing a computer to function as a device as claimed in any one of claims 1 to 3.