JP5660362B2 - Sound source localization apparatus and computer program - Google Patents

Description

  The present invention relates to a sound source localization technique in a real environment, and more particularly, to a sound source localization using a MUSIC (Multiple Signal Classification) method in a real environment, and a technique for detecting a sound generation section by a moving sound source based on the sound directionality.

  In voice communication between a person and a robot, the microphone attached to the robot is usually located at a distance (1 m or more). Therefore, for example, the signal-to-noise ratio (SNR) is lower than when the distance between the microphone and the mouth is several centimeters as in telephone speech. For this reason, the voices of others nearby and the noise of the environment become interference sounds, making it difficult for the robot to recognize the target speech. Therefore, sound source localization and sound source separation are important for robot applications.

  Various studies have been conducted on sound source localization in the past. However, most of them use only simulation data or lab data, and few evaluate real-world data in which the robot operates. There are few studies to evaluate 3D sound source localization. Talking and listening while looking at the face of the utterance partner is also an important behavior for improving dialogue interaction between humans and robots. For that purpose, three-dimensional sound source localization is also important.

  There exists a thing of patent document 1 as a prior art which assumed the real environment. The technique described in Patent Document 1 uses a known sound source localization technique called the MUSIC method with high resolution.

  In the invention described in Patent Document 1, a microphone array is used, and a current correlation matrix is calculated based on a received signal vector obtained by Fourier transform of a signal from the microphone array and a past correlation matrix. The correlation matrix obtained in this way is subjected to eigenvalue decomposition to obtain a maximum eigenvalue and a noise space that is an eigenvector corresponding to an eigenvalue other than the maximum eigenvalue. Furthermore, the direction of the sound source is estimated by the MUSIC method based on the phase difference of the output of each microphone, the noise space, and the maximum eigenvalue with one microphone as a reference in the microphone array.

JP 2008-175733 A

  The MUSIC method has a feature of high resolution, but there is a problem that the number of sound sources must be given when the MUSIC method is used. In the technique described in Patent Document 1, since it is assumed that there is one sound source, such a problem does not occur. However, the environment in which the robot actually operates is rarely such an environment, and there are always a plurality of sound sources, and the number is not constant. When the MUSIC method is used, if the number of sound sources is incorrectly predicted, sound source localization will also be incorrect, making it difficult for the robot to interact correctly with humans. In particular, if the number of sound sources is predicted too much, there is a problem that not only preferable interaction becomes difficult, but also the calculation cost increases.

  Furthermore, in the technique described in Patent Document 1, sound source localization is performed two-dimensionally. However, the actual operating environment of the robot is not two-dimensional but three-dimensional. For example, in a shopping street or the like, a speaker is placed at a relatively high position, and audio is often played from the speaker. Moreover, although the position of the speaker is constant, the volume may change. In such an environment, it is preferable to localize the sound source three-dimensionally, but the technique described in Patent Document 1 has a problem that it can be performed only two-dimensionally.

  In particular, in the case of a robot against a human being, the height of the human being varies, and in the case of an adult, the person speaks at a higher position than the robot, and the child often speaks at a lower position than the robot. From such a point, it is desirable to perform three-dimensional sound source localization. When a robot and a person interact with each other, it is necessary to point the robot's face in the direction of the opponent's face, but it is difficult to perform such a conversation unless three-dimensional sound source localization is performed.

  Furthermore, since humans move frequently, it is also necessary to track the sound source stably in real time.

  SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a sound source localization apparatus that can stably perform sound source localization using the MUSIC method. Here, sound source localization refers to continuously specifying the direction of the sound source.

  Another object of the present invention is to provide a sound source localization apparatus that can stably track a sound source using the MUSIC method.

  Still another object of the present invention is to provide a sound source localization apparatus that can stably track a sound source using the MUSIC method and accurately predict the generation and disappearance of the sound source in real time.

  According to the first aspect of the present invention, the sound source localization apparatus is based on each of a plurality of sound source signals obtained from the output of the microphone array and a positional relationship between the microphones included in the microphone array. MUSIC response calculating means for calculating MUSIC power for each predetermined time by a MUSIC algorithm for each of a plurality of orientations defined in a three-dimensional space centered on a point defined in relation to the position of the array; For each of a plurality of directions, a sound source for detecting a change in the direction of the sound source from the generation to the disappearance of the sound source based on the intensity of the MUSIC power value obtained as a time series by the MUSIC response calculation means and the amount of change Estimation means.

  Preferably, the sound source estimation means, for each of the plurality of azimuths, includes the MUSIC power calculated by the MUSIC response calculation means immediately before each of the azimuths adjacent to the processing target azimuth among the plurality of azimuths. And the latest MUSIC power value obtained for the azimuth to be processed is calculated, and a value satisfying a predetermined condition is output as the MUSIC Δ power for the azimuth. MUSICΔ power calculation means for performing, for each of a plurality of azimuths, the value of MUSIC power calculated by the MUSIC response calculation means for the azimuth and the MUSICΔ power output by the MUSICΔ power calculation means for the azimuth On the basis of whether or not a sound source has been generated, and in response to the detection of the sound source, Onset detection means for outputting information indicating the direction and the direction of the sound source, and in response to the detection of the generation of the sound source by the onset detection means, the sound source detected by the onset detection means MUSIC power value calculated by the MUSIC response calculating means for each of the azimuths adjacent to the direction of the sound source, and MUSIC Δ power output by the MUSIC Δ power calculating means for each of the directions adjacent to the direction of the sound source And tracking means for tracking until the sound source disappears.

  More preferably, the sound source localization apparatus further includes a smoothing unit for calculating and smoothing the moving average of the MUSIC power output by the MUSIC response calculating unit for each of a plurality of directions. Both the MUSIC Δ power calculation means and the tracking means receive the MUSIC power smoothed by the smoothing means as an input.

  More preferably, the onset detection means has a value of the MUSICΔ power calculated by the MUSICΔ power calculation means larger than the first onset threshold for each of the plurality of directions, and the MUSIC response calculation means. Based on the first determination means for determining whether or not the calculated MUSIC power value is greater than the second onset threshold, and the determination result by the first determination means, the generation of the sound source is performed. Second determining means for detecting.

  The second determination means identifies only a certain number of the azimuths determined by the first determination means as a detected sound source from the top of the MUSIC power calculated for the azimuth. The sound source limiting means for performing may be included.

  Preferably, the tracking unit responds to the detection of the generation of the sound source by the onset detection unit, and the direction of the sound source among the MUSIC responses output by the MUSIC response calculation unit every predetermined time after the detection. By tracing the maximum value of the MUSIC power value calculated for the azimuth adjacent to the signal, the means for tracking the movement of the sound source in which the occurrence is detected and the direction of the sound source tracked by the means for tracking are calculated. Sound source disappearance detecting means for detecting the disappearance of the sound source and stopping the tracking by the means for tracking when the MUSIC power and the MUSICΔ power satisfy the predetermined condition.

  More preferably, the sound source disappearance detecting unit is a first unit for determining whether or not the MUSICΔ power calculated for the direction of the sound source tracked by the unit for tracking is smaller than a first offset threshold value. Offset determination means and a second offset determination for determining whether or not the MUSIC power calculated for the direction is larger than a second offset constant that is larger than the MUSIC power at the time of generation of the sound source by a certain positive constant The result that either one of the determination results of the means and the first and second offset determination means is affirmative is the predetermined number of times (preferably a plurality of times) for the MUSIC response calculated every predetermined time by the MUSIC response calculation means. Sounds tracked by means of tracking when obtained continuously There were determined to have disappeared, and means for stopping the tracking.

  When the computer program according to the second aspect of the present invention is executed by a computer, it causes the computer to function as each means of any of the sound source localization apparatuses described above.

It is a figure which shows the state which attached the microphone stand 32 to the robot 30 which has the sound source localization process part 50 which concerns on one embodiment of this invention. 3 is a three-side view of a microphone base 32. FIG. 3 is a block diagram of a sound source localization processing unit 50. FIG. It is a flowchart which shows the control structure of the computer program for implement | achieving the function of the sound source estimation part 64. FIG. It is a flowchart which shows the control structure of the computer program which implement | achieves an onset detection process. It is a flowchart which shows the control structure of the computer program which implement | achieves tracking processing. It is a flowchart which shows the control structure of the computer program which implement | achieves an offset detection process. It is a graph which shows the result of the experiment conducted in order to measure the performance about the sound source localization process part 50 which concerns on embodiment. It is a graph which shows the result of the experiment conducted in order to measure the performance about the sound source localization process part 50 which concerns on embodiment. It is a graph which shows the result of the experiment conducted in order to measure the performance about the sound source localization process part 50 which concerns on embodiment. It is a block diagram of the computer which implement | achieves the sound source localization process part 50 which concerns on embodiment of this invention.

  In the following description of the embodiments of the present invention, the same reference numerals are assigned to the same components. Their functions are also the same. Therefore, detailed description thereof will not be repeated.

[Overview]
In the present embodiment, a microphone array is arranged near the head of the robot, a plurality of sound sources are localized in real time from signals obtained from these microphone arrays, and tracking thereof is performed. For this purpose, the sound source localization apparatus of the embodiment described below calculates the MUSIC spatial spectrum by the MUSIC algorithm with a fixed number of sound sources, and directly uses the obtained MUSIC spatial spectrum to move the number of sound sources and their positions. Adopt a mechanism that estimates automatically.

[Constitution]
FIG. 1 shows a state in which the microphone array is fitted to the chest of the robot 30. Specifically, a microphone base 32 for fitting a microphone around the neck of the robot 30 is created, and a plurality of microphones MC1 and the like are fixed to the microphone base 32, and then the microphone base 32 is placed around the neck of the robot 30. It is fixed.

  FIG. 2 shows a front view, a plan view, and a right side view of the microphone base 32. Referring to FIG. 2, only 14 microphones MC1 etc. are used. Nine of them are attached to the front part of the microphone base 32, and the remaining five are attached to the upper surface of the microphone base 32 so as to surround the neck of the robot 30. Of the 14 microphones, the output of the microphone MC1 at the center is used separately from others in the subsequent processing. In this embodiment, each microphone is an omnidirectional microphone.

FIG. 3 is a block diagram showing only the sound source localization processing unit 50 related to sound source localization from the robot shown in FIG. Referring to FIG. 3, sound source localization processing unit 50 receives 14 analog sound source signals from microphone array 52 including microphone MC1 and the like, performs analog / digital conversion, and outputs 14 digital sound source signals. D converter 54, a / D received fourteen digital sound signal output from the transducer 54, the block correlation matrix required by the MUSIC method and the eigenvalues and eigenvectors as one block have one 100ms The eigenvector calculation unit 60 for outputting each block, the MUSIC processing unit 62 that outputs the MUSIC space spectrum by the MUSIC method using the eigenvector output from the eigenvector calculation unit 60 for each block, and the MUSIC processing unit 62 output Based on the MUSIC spatial spectrum, the number of sound sources and their positions are dynamically estimated. A sound source that outputs a value representing the position (direction) (in this embodiment, two declination angles φ and θ in the three-dimensional polar coordinates; see “MUSIC response” in the appendix) in time series. An estimation unit 64 and a buffer 66 for accumulating the time series of the output of the sound source estimation unit 64 are included. In the present specification, the “MUSIC response” is obtained by averaging the MUSIC spatial spectrum obtained by the MUSIC algorithm using a predetermined formula. Please refer to “MUSIC Response” in the Appendix for details.

  In the present embodiment, the A / D converter 54 performs A / D conversion on the output of each microphone at a general 16 kHz / 16 bits.

  The eigenvector calculation unit 60 framing the 14 digital sound source signals output from the A / D converter 54 with a frame length of 4 milliseconds, and the output 14 from the framing processing unit 80. FFT (Fast Fourier Transform) is applied to the sound source signals framed in the channel, and a predetermined number of frequency regions (hereinafter, each frequency region is referred to as “bin”, and the number of frequency regions is referred to as “bin number”). .)), And a block processing unit 84 for blocking each bin value of each channel output from the FFT processing unit 82 every 4 milliseconds. And a correlation matrix whose element is the correlation between the bin values output from the block processing unit 84 (every 100 milliseconds). And a correlation matrix calculation unit 86 that calculates and outputs the correlation matrix output from the correlation matrix calculation unit 86 and an eigenvalue decomposition unit 88 that outputs the eigenvector 92 to the MUSIC processing unit 62. In the present embodiment, the frequency component of the sound source signal excludes a band of 1 kHz or less with a low spatial resolution and a band of 6 kHz or more where spatial aliasing may occur.

  Normally, 512 to 1024 points are used in FFT (corresponding to 32 to 64 milliseconds at a sampling rate of 16 kHz), but here one frame is set to 4 milliseconds (corresponding to 64 to 128 points in FFT). By reducing the frame length in this way, not only the amount of calculation of FFT is reduced, but also the amount of calculation in later calculation of correlation matrix, eigenvalue decomposition, and calculation of MUSIC response is reduced. As a result, sound source localization can be performed sufficiently in real time even if a relatively weak computer is used without degrading performance.

  The MUSIC processing unit 62 includes a position vector storage unit 100 for storing a position vector representing the position of each microphone included in the microphone array 52 using a predetermined coordinate system, and a microphone stored in the position vector storage unit 100. And a MUSIC spatial spectrum calculation unit 104 that calculates and outputs a MUSIC spatial spectrum by the MUSIC method on the assumption that the number of sound sources is fixed, using the position vector of Eq. And the eigenvector output from the eigenvalue decomposition unit 88. The fact that the eigenvalue of the correlation matrix obtained for each block is related to the number of sound sources is, for example, F.D. Asano et al., “Real-time sound source localization and generation system and its application in automatic speech recognition”, Eurospeech, 2001, Aalborg, Denmark, 2001, 1013-1016 (F. Asano, M. Goto, K. Itou, and H. Asoh, “Real-time sound source localization and separation system and its application on automatic speech recognition,” in Eurospeech 2001, Aalborg, Denmark, 2001, pp. 1013-1016) It is.

  In the present embodiment, not only the two-dimensional azimuth angle of each sound source but also the elevation angle is estimated. To that end, a three-dimensional version of the MUSIC algorithm (see appendix) was implemented. The set of azimuth and elevation is hereinafter referred to as sound source azimuth (DOA). This algorithm does not estimate the distance to the sound source. By estimating only the sound source azimuth, the processing time can be significantly reduced.

  The MUSIC processing unit 62 further calculates a MUSIC response based on the MUSIC spatial spectrum calculated by the MUSIC spatial spectrum calculation unit 104 according to the MUSIC method for each direction (described later) and outputs the MUSIC response. A calculation unit 106 is included.

  The “azimuth” here refers to each frame of the mesh defined in the three-dimensional space in order to search for the sound source position. In this embodiment, the mesh is divided into a ring shape in a range of an elevation angle of 5 degrees, and different numbers of search points are provided depending on the size of the elevation angle. The “search point” here refers to the center point of the mesh described above.

  The number of search points is selected so that the direction angle to an adjacent search point is 5 degrees in a ring with an elevation angle of 0 degrees. The number of search points is the maximum for a wheel with an elevation angle of 0 degrees, and decreases as the elevation angle increases. At this time, the distances (which may be considered as angles) between search points in one ring are equal to each other, and the distances (angles) are the distances (angles) between adjacent search points in a ring having an elevation angle of 0 degrees. And is chosen to be as close as possible.

  The sound source estimation unit 64 temporarily stores a predetermined number of MUSIC response peaks calculated by the MUSIC response calculation unit 106 in the FIFO format in a time series, and each block stored in the buffer 108. About the MUSIC response of each search point, the smoothing filter unit 110 for removing noise by calculating and smoothing the moving average, and the value of the MUSIC response of each search point of each block output by the smoothing filter unit 110 And a MUSIC Δ spectrogram calculating unit 112 for calculating a difference (MUSIC Δ power) between the MUSIC response at each search point and the MUSIC response of the previous block. Specifically, the MUSIC Δ spectrogram calculating unit 112 calculates the MUSIC Δ power as follows. That is, for each search point, the difference between the value of the MUSIC power of the current block and the MUSIC power of the previous block at all points adjacent to the search point is taken. The minimum value of the difference is set as the MUSIC Δ power at the search point. Here, as a time series, the MUSIC power calculated for each azimuth is called a MUSIC spectrogram, and the time series of values calculated for each azimuth is called a MUSIC Δ spectrogram.

  The sound source estimation unit 64 further uses the smoothed MUSIC response of each block output from the smoothing filter unit 110 to determine the direction in which the sound source has started generating sound (the sound source has started sound generation activity). This is referred to as onset.) And the process of detecting the azimuth and time at which sound generation from the sound source stopped (stopping sound generation by the sound source is referred to as offset) is performed. And an onset offset detection unit 114 for outputting an offset generation direction, a MUSICΔ power output from the MUSICΔ spectrogram calculation unit 112, and an onset detection output and an offset detection output from the onset offset detection unit 114 A tracking unit 118 for tracking the sound source from onset to offset, and MU The SIC Δ spectrogram calculation unit 112, the onset offset detection unit 114, and the tracking unit 118 store information on the position of each search point and the search point that is adjacent to each search point and is used for each process. And a search point storage unit 116.

  Prior to calculating the power of the audio signal from each sound source, inter-channel spectral binary masking processing is performed on the audio signal. This is a process of leaving the signal with the larger power between the two channels and setting the signal of the other channel to zero. By doing so, interference leakage between channels can be reduced. Further, using the audio signal from the microphone MC1 located in the center of the microphone array 52, a process of removing noise caused by surrounding music from all the audio signals is performed.

[Realization by computer]
The above-described sound source localization processing unit 50 is actually realized by the cooperation of hardware and software by computer hardware and a computer program executed by the computer hardware. Hereinafter, a control structure of a computer program for realizing the function of the sound source estimation unit 64 that is a feature of the present embodiment in the sound source localization processing unit 50 will be described.

  Referring to FIG. 4, the program for realizing the function of sound source estimation unit 64 becomes a processing target through step 130 for initialization after power-on of the robot and from MUSIC response calculation unit 106 via buffer 108. Step 132 for receiving the MUSIC response of the block, Step 134 for performing smoothing processing to remove noise from the MUSIC response by taking a moving average with respect to the MUSIC response of the block to be processed, and for all search points, Step 136 for calculating the MUSIC Δ power (MUSIC Δ spectrogram), Step 138 for detecting the generation (onset) of the sound source based on the MUSIC Δ power calculated for each direction in Step 136 and the MUSIC power, and the detection in Step 138 Tracked sound source And, and a step 140 of returning processing to step 132. Note that the identifiers of all search points, their orientations, and the identifiers of search points adjacent to the search points are stored in the search point storage unit 116 shown in FIG.

  As already described, in calculating the MUSIC Δ spectrum in step 136, for each search point, the difference between the MUSIC power of the current block and the MUSIC power of each previous block of the search points adjacent to the search point is calculated, By adopting the minimum value among them, the MUSIC Δ power is calculated for each search point for each block.

  Referring to FIG. 5, the onset detection process executed in step 138 of FIG. 4 includes steps 162, 164, 166 and 168 described below for each search point other than the search points to be tracked. Performing step 160.

  The process of step 160 determines whether or not the value of MUSICΔ power calculated in step 134 of FIG. 4 for the search point is greater than 1.0 dB, and if the value of MUSICΔ power is 1.0 dB or less, This is executed when it is determined in step 162 that the processing relating to the search point is finished and in step 162 that the MUSIC Δ power value of the search point is larger than 1.0 dB, and the MUSIC power value of the search point is more than 1.8 dB. If the MUSIC power value is 1.8 dB or less, it is determined in step 164 that the processing relating to this search point is terminated, and in step 164 it is determined that the MUSIC power value is greater than 1.8 dB. Step 166, which is executed occasionally, and temporarily stores the identifier of the search point as an onset candidate. Following-up 166, and stores the MUSIC power of the search point as a tracking onset MUSIC power of the sound source and the step 168 to terminate the processing for the search point. In step 166, 1 is added to a variable indicating the number of onset candidates prepared in advance.

  The onset detection process is further executed after it is determined in step 160 whether or not all search points are onset candidates, and it is determined whether or not the number of onset candidates is zero. Step 170 for ending the onset detection process for the current block is executed in response to the determination that the number of onset candidates is not zero in step 170. Among the onset candidates, the MUSIC power value is the largest. Step 172 for selecting up to 2 MUSIC power values in order of the MUSIC power as onsets in this block, and preparing a new tracking list for each of the up to 2 onsets selected in step 172 And on the search points that are onset, such as onset MUSIC power at each head element Store information and a step 174 to end the onset detection process. Here, all of the newly created tracking lists are targets for subsequent tracking. The head element of each tracking list is provided with an area for storing a tracking end flag, and its value is set to zero. The tracking end flag is a flag indicating whether or not tracking of the sound source corresponding to the tracking list has ended. If the tracking end flag is 0, it indicates that tracking for the list is in progress. If the value is 9, tracking for the list is completed (the generation of sound from the sound source has ended). )

  Referring to FIG. 6, the tracking process performed in step 140 of FIG. 4 includes a step 200 of executing steps 202-214 described below for each of the tracking lists.

  In step 200, the process executed for each tracking list determines whether the tracking end flag for the target tracking list is 0 or not. If the end flag is not 0, the process for the tracking list is ended 202. And when the tracking end flag is determined to be 0 in step 202, all the search points around the search point at the end of the tracking list (indicating the direction of the sound source detected in the previous block). Reading 204 the calculated MUSIC power from the storage device, and adding 206 a search point corresponding to the maximum of the read MUSIC powers to the end of the tracking list.

  The processing in step 200 is further continued from step 206 to step 208 in which the offset detection processing is executed for the target tracking list, and subsequent to step 208, whether or not the value set in the offset flag by the offset detection processing is 0. And step 210 for branching the flow of control according to the determination result. The offset flag is set to 9 when it is determined that sound generation has stopped for the sound source corresponding to this tracking list when it is 0, and is set to 0 at other times, that is, when it is determined that there is a continuous sound source. .

  The processing executed in step 200 is further executed when it is determined in step 210 that the value of the offset flag is not 0. As a result of the offset detection processing in step 208, the offset flag is set to a value other than 0 after 5 blocks. It is determined whether or not it has been set, and when it has not reached 5 blocks, nothing is done and step 212 ends the processing for the tracking list to be processed. In step 212, the offset flag is set to 0 for this tracking list for 5 consecutive blocks. And step 214 that is executed in response to the determination that the tracking list is not, sets the end flag of this tracking list (sets the value of the end flag to 9), and ends the processing for this tracking list.

  Referring to FIG. 7, in the offset detection process executed in step 208 in FIG. 6, the MUSIC Δpower of the search point last added to the target tracking list (step 206 in FIG. 6) is −1. This step is executed when it is determined whether the MUSICΔ power is smaller than -1.2 dB in step 230 for determining whether or not it is smaller than 2 dB and branching the control flow according to the determination result. And step 236 for setting the offset flag to 9 and ending the offset detection process.

In the offset detection process, when it is determined in step 230 that the value of the MUSICΔ power is −1.2 dB or more, a threshold value θ _H = onset MUSIC power + α (α> 0) for offset detection. Step 232 for calculating the threshold value θ _H for offset detection according to the equation, and the MUSIC power of the current block at the search point last added to the tracking list (step 206 in FIG. 6) is the threshold value described above. determines whether theta _H smaller than the step 234 if the determination result is to advance to step 236 to control if affirmative, when the determination at step 234 is negative, the offset flag for this tracking list 0 And step 238 for setting and ending the offset detection process.

  As will be described later, in the offset detection, the offset is forcibly set when the MUSIC power of the sound source being tracked is smaller than the power at the time of onset, and when it is smaller than the power at the time of onset + α. As a result, the effect of increasing the accuracy of tracking of the sound source can be obtained.

[Operation]
The sound source localization processing unit 50 according to the above embodiment operates as follows. It is assumed that the microphone array is attached to the robot 30 using a microphone base 32 as shown in FIGS.

  The microphone array 52 converts the sound from the sound source into 14 analog electric signals and supplies the analog electric signals to the A / D converter 54. The A / D converter 54 converts these signals into 16-bit digital signals at 16 kHz, and provides 14 digital signals to the framing processor 80.

  The framing processing unit 80 frames the digital sound source signals of these channels with a frame length of 4 milliseconds, and supplies the framing processing unit 80 to the FFT processing unit 82. The FFT processing unit 82 performs FFT on the digital sound source signal of each frame of each channel, converts the digital sound source signal into an output of each frequency component, and gives it to the blocking processing unit 84. During this time, the above-described inter-channel spectral binary masking process is performed on the audio signal, and the process of removing noise from surrounding music from all the audio signals using the audio signal from the microphone MC1 located in the center.

  The blocking processing unit 84 blocks the signal output every 4 milliseconds from the FFT processing unit 82 every 100 milliseconds, and gives it to the correlation matrix calculation unit 86. The correlation matrix calculation unit 86 calculates a correlation matrix for each channel for each of these blocks, and provides it to the eigenvalue decomposition unit 88. The eigenvalue decomposition unit 88 performs eigenvalue decomposition on the correlation matrix calculated by the correlation matrix calculation unit 86 and gives the result to the MUSIC spatial spectrum calculation unit 104.

  The processing after the MUSIC spatial spectrum calculation unit 104 is a three-dimensional processing of the normal MUSIC method. First, the MUSIC spatial spectrum calculation unit 104 determines that the number of sound sources is fixed based on the position vector stored in the position vector storage unit 100 and the eigenvector 92 output from the eigenvalue decomposition unit 88, and sets the MUSIC spatial spectrum to 100 milliseconds. Is calculated for each and provided to the MUSIC response calculation unit 106. The MUSIC response calculation unit 106 calculates a MUSIC response every 100 milliseconds based on the MUSIC spatial spectrum and stores it in the buffer 108.

  The buffer 108 accumulates a predetermined number of MUSIC responses output from the MUSIC response calculation unit 106 in time series in the FIFO format.

  The smoothing filter unit 110 reads the MUSIC response of each block stored in the buffer 108 (step 132 in FIG. 4), takes a moving average over a predetermined block for the MUSIC response of the block, and uses the smoothed MUSIC power. The MUSIC Δ spectrogram calculation unit 112 and the onset offset detection unit 114 are provided.

  When the MUSIC Δ spectrogram calculation unit 112 receives the block data (step 132 in FIG. 4), for each search point, the MUSIC power of the current block and the MUSIC power of the previous block of each search point adjacent to the search point are calculated. By calculating the difference and adopting the minimum value among them, the MUSICΔ power is calculated for each block for each search point (step 134 in FIG. 4) and provided to the tracking unit 118. The onset offset detection unit 114 is realized by the program having the above-described structure (FIGS. 5 and 7), and each block is based on the smoothed MUSIC power and MUSICΔ power output from the smoothing filter unit 110. If there are any onsets or offsets of the sound source at, they are detected and given to the tracking unit 118. At the time of onset detection, the direction (search point) is also detected and given to the tracking unit 118.

  Specifically, in onset detection, as indicated by steps 162 and 164 in FIG. 5, for each block, there are search points for which the MUSIC Δpower is greater than 1.0 dB and the MUSIC power is greater than 1.8 dB. Candidate onset. As shown by steps 170 to 174 in FIG. 5, when there are onset candidates for each block, up to two onsets are detected as onsets.

  In the detection of the offset, as indicated by steps 230-234 in FIG. 7, the last tracked search point is determined to be an offset when its MUSICΔ power is smaller than −1.2 dB. Even when the power becomes smaller than the onset MUSIC power + α of the sound source, the offset is forcibly determined.

  When the onset detection signal is given from the onset / offset detection unit 114, the tracking unit 118 starts tracking the sound source. Specifically, after detecting the onset, the tracking unit 118 tracks a search point having the maximum MUSIC power among search points adjacent to the search point for the sound source position (step 204 to step 206 in FIG. 6). Tracking is terminated when an offset is detected. However, in the present embodiment, after the offset occurs, tracking is continued up to 4 blocks, and tracking is finished only when a search point that satisfies the onset condition does not occur in the tracking direction even after 5 blocks have elapsed (see FIG. 6 steps 210, 212 and 214).

  The sound source direction tracked by the tracking unit 118 is stored in the buffer 66 for each block.

  With the above operation, the sound source localization processing unit 50 can continuously perform localization and tracking of a plurality of sound sources.

  FIG. 8 to FIG. 10 show the results of experiments performed to measure the performance of the sound source localization processing unit 50 according to the above embodiment. In this experiment, the following values are used to measure the performance of the sound source (Directions Of Arrival) localization. The first is the DOA accuracy, and the second is the DOA insertion rate. The DOA accuracy is a rate at which correct DOA is detected by the above-described apparatus. Higher DOA accuracy is preferable. The DOA insertion rate is an average value of the number of sound sources detected extra per block as compared with the correct number of DOAs. A lower DOA insertion rate is preferable.

  As the correct DOA, DOA obtained by using the correct number of sound sources from information indicating the activity of the sound source obtained from the sound source signal was used. While each sound source was active, the trajectory obtained from the predicted DOA position was approximated by piecewise linear approximation. Video was also used to check whether the sound source was moving.

  In the experiment, DOA prediction is performed by the sound source localization processing unit 50 mounted on the humanoid robot in two different environments (OFC, UCW, which will be described later), and the DOA accuracy and DOA insertion rate are obtained. It was. 8 to 10 show the results. These environments are environments that record audio for experiments, and are specifically as follows.

  That is, data recording by a microphone array was performed in two different environments. The first one is an office environment (OFC), and the internal noise of indoor air conditioners and robots are the main noise sources. The second environment is an outdoor shopping mall passage (UCW) where the experiment was conducted. The main noise source in UCW is pop-rock music flowing from speakers installed on the ceiling. The experimental data and images were recorded by placing the robot in various positions and various directions in the passage. 8 to 10, numbers added after “OFC” or “UCW” such as “OFC3” indicate the order of recording.

  In the OFC, four participants (four men) were used as sound sources. First, each participant spoke to the robot for about 10 seconds. During this time, the other participants were quiet. During the last 15 seconds of recording, four participants spoke at the same time. During recording, each participant wore a microphone connected to a separate audio capture device. Using the temporary sound at the start of recording, the recording of the speech of these participants and the recording of the audio signal from the microphone array 52 were manually synchronized.

  In UCW, two participants (both men) were used as sound sources for all recordings. In each experiment, in each case, each participant first spoke separately for about 10 seconds, and finally spoke at the same time. In UCW7 and UCW8, one participant moved, while another participant was stationary, and the two were speaking together for almost all the time. In UCW1-4 and UCW9, the robot was placed far away from the speaker on the ceiling. In UCW5-8, the robot was placed near the speaker (several meters). In UCW10-13, the robot was placed directly under the speaker.

  In all trials, the robot was pointed in various directions and the sound sources were placed in various positions to acquire data.

FIG. 8 shows DOA prediction performance (accuracy and insertion rate) in trials performed by changing some of the parameters related to sound source localization in the OFC and UWC environments. The changed parameters are the presence / absence of the smoothing filter unit 110, the presence / absence of the value α added to the threshold value θ _TH at the time of offset detection, the presence / absence of the restriction on the number of onsets per block, and before and after tracking. .

  The parameter values used in the calculation of the MUSIC spectrogram are as follows. That is, NFFT (FFT length) = 64, frequency range = 1-6 kHz, and the number of fixed sound sources in the MUSIC spatial spectrum calculation unit 104 = 2. When the value of NFFT is increased, the performance is somewhat improved. However, when NFFT = 64, the value can be used in real time even when a commercially available CPU having an operation clock frequency of about 2 GHz is used.

  The left side of FIG. 8 shows the average DOA accuracy for the sound source under each experimental condition and each of OFC and UCW, and the DOA insertion rate is shown in the center of FIG.

  From these graphs, it is understood that the DOA accuracy is highest when “no α” is set. However, in this case, the DOA insertion rate is also the worst. The DOA accuracy obtained for “no smoothing”, “no limit on number of sets”, and “before tracking” are very similar to each other. However, it can be clearly seen that the DOA insertion rate is decreasing “before tracking”. This indicates that smoothing and the limitation of the number of onsets per block are effective. Comparing “before tracking” and “after tracking”, UCW3-6 and UCW10-11 show a slight improvement in DOA accuracy, while UCW8-9 shows a very small improvement in DOA insertion rate. I understand.

  The results of DOA accuracy for environmental music are shown on the right side of FIG. According to this result, it is understood that the DOA accuracy is low in UCW1-4 and UCW7-9. This is probably because the robot was placed relatively far from the speaker. In UCW10-13, the DOA accuracy is close to 100%. This is because, under these experimental conditions, the robot was placed directly under the speaker, and the background music was detected as a sound source with a clear directivity. On the other hand, in the UCW5-6 where the robot is placed close to the speaker, the DOA accuracy is an intermediate value.

  Examining the performance of each sound source, as shown in FIG. 9, the DOA accuracy of the second and fourth sound sources of OFC2, the first sound source of UCW9, and the second sound source of UCW12 is low. In these environments, since the sound from the sound source was behind the robot, it seems that both the power and directivity were low in the sound from the front of the robot.

  According to the sound source localization processing unit 50 according to the above embodiment, sound source localization is performed using raw data of a MUSIC response. As is clear from the experimental results, the number of sound sources can be predicted and tracked with high accuracy by such processing. Further, by adding α to the value of the MUSIC power at the time of onset as a threshold for detecting the offset, it is considered that the offset has been detected even if the MUSIC power is slightly higher than the MUSIC power at the time of onset. As a result, the frequency of post-insertion of DOA can be reduced. As a result, sound source localization by the MUSIC method can be performed stably and accurately. When the source of background noise is in the immediate vicinity of the sound source localization processing unit 50, the accuracy is lowered. However, in actual robot implementation, whether the sound source is a conversation partner using not only speech but also images. Can be predicted, and it can be expected to prevent a decrease in accuracy.

Furthermore, since the three-dimensional MUSIC method is used in the present embodiment, it is possible to estimate the sound source azimuth including not only the azimuth angle but also the elevation angle within a certain range. Therefore, even in an environment where voice is received from various directions in a real environment, a robot or the like can correctly locate a sound source and perform an appropriate operation. Even when the robot interacts with a human, it can be expected to perform an appropriate operation while looking at the face of the other party, and the interaction between the robot and the human can be made smoother.
[Realization by computer]
The sound source localization processing unit 50 according to this embodiment can be realized by computer hardware, a program executed by the computer hardware, and data stored in the computer hardware. FIG. 11 shows the internal configuration of the computer system 330.

  Referring to FIG. 11, this computer system 330 includes a computer 340 having a memory port 370 and a DVD (Digital Versatile Disc) drive 364 as a removable memory attachment / detachment unit, a keyboard 346, a mouse 348, and a monitor 342. Including.

  In addition to the memory port 370 and the DVD drive 364, the computer 340 is connected to the CPU (Central Processing Unit) 356, the bus 366 to which the CPU 356, the memory port 370 and the DVD drive 364 are connected, and the bus 366. Are connected to a bus 366, a random access memory (RAM) 360 that temporarily stores program instructions, system programs, work data, and the like, and a bus 366. A hard disk 362 that is a non-volatile storage device storing a large amount of data, an input / output interface (I / F) 368 connected to the bus 366 and receiving the output from the A / D converter 54, and a local area network ( LAN) And a radio network I / F372.

  A computer program for causing the computer system 330 to function as the sound source localization processing unit 50 is stored in the DVD 390 or the removable memory 392 inserted into the DVD drive 364 or the memory port 370 and further transferred to the hard disk 362. Alternatively, the program may be transmitted to the computer 340 through a network (not shown) and stored in the hard disk 362. The program is loaded into the RAM 360 when executed. The program may be loaded into the RAM 360 directly from the DVD 390, from the removable memory 392, or via a network.

  This program includes a plurality of instructions for causing the computer 340 to operate as the sound source localization processing unit 50 of this embodiment. Some of the basic functions necessary to perform this operation may be provided by the OS when an operating system (OS) is running on the computer 340. These functions may be provided by third party programs or modules of various toolkits installed on the computer 340. Therefore, this program does not necessarily include all functions necessary for realizing the system and method of this embodiment. This program includes only an instruction for executing the operation as the sound source localization processing unit 50 by calling an appropriate function or “tool” in a controlled manner so as to obtain a desired result. Just go out. The operation of computer system 330 is well known and will not be repeated here.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

[Appendix: MUSIC method]
The Fourier transform Xm (k, t) of M microphone inputs is modeled as shown in Equation (1).

ただし、ベクトルｓ（ｋ、ｔ）はＮ個の音源のスペクトルＳ_ｎ（ｋ，ｔ）から成る：ｓ（ｋ、ｔ）＝［Ｓ_１（ｋ，ｔ），…，Ｓ_Ｎ（ｋ、ｔ）］^Ｔ。ｋとｔはそれぞれ周波数と時間フレームのインデックスを示す。ベクトルｎ（ｋ、ｔ）は背景雑音を示す。行列Ａ_ｋは変換関数行列であり、その（ｍ、ｎ）要素はｎ番目の音源から、ｍ番目のマイクロホンへの直接パスの変換関数である。Ａ_ｋのｎ列目のベクトルをｎ番目の音源の位置ベクトル（Ｓｔｅｅｒｉｎｇ Ｖｅｃｔｏｒ）と呼ぶ。

However, the vector s (k, t) consists of N sound source spectra S _n (k, t): s (k, t) = [S ₁ (k, t),..., S _N (k, t) )] ^T. k and t indicate frequency and time frame indexes, respectively. Vector n (k, t) indicates background noise. The matrix A _k is a conversion function matrix, and its (m, n) element is a conversion function of a direct path from the nth sound source to the mth microphone. The n-th column vectors of A _k is referred to as a position vector of the n-th sound source (Steering Vector).

First, a spatial correlation matrix R _k defined by Equation (2) is obtained, and E _k composed of an eigenvalue diagonal matrix Λ _k and an eigenvector is obtained by eigenvalue decomposition of R _k shown in Equation (3).

固有ベクトルはＥ_ｋ＝［Ｅ_ｋｓ｜Ｅ_ｋｎ］のように分割出来る。Ｅ_ｋｓとＥ_ｋｎとはそれぞれ支配的なＮ個の固有値に対応する固有ベクトルと、それ以外の固有ベクトルとを示す。

The eigenvector can be divided as E _k = [E _ks | E _kn ]. E _ks and E _kn indicate eigenvectors corresponding to the dominant N eigenvalues and other eigenvectors, respectively.

  The MUSIC spatial spectrum is obtained by equations (4) and (5). r is a distance, and θ and φ are an azimuth angle and an elevation angle, respectively. Equation (5) is a normalized position vector at the scanned point (r, θ, φ).

空間スペクトル（本明細書では「ＭＵＳＩＣ応答」と呼ぶ。）は、ＭＵＳＩＣ空間スペクトルを式（６）のように平均化したものである。

The spatial spectrum (referred to herein as a “MUSIC response”) is an averaged MUSIC spatial spectrum as shown in Equation (6).

式（６）においてｋ_Ｌ及びｋ_Ｈは、それぞれ周波数帯域の下位と上位の境界のインデックスであり、Ｋ＝ｋ_Ｈ−ｋ_Ｌ＋１。音源の方位は、ＭＵＳＩＣ応答のピークから求められる。

In Expression (6), k _L and k _H are indices of the lower and upper boundaries of the frequency band, respectively, and K = k _H −k _L +1. The direction of the sound source is obtained from the peak of the MUSIC response.

30 robot 32 microphone base 50 sound source localization processing unit 52 microphone array 60 eigenvector calculation unit 62 MUSIC processing unit 64 sound source estimation unit 86 correlation matrix calculation unit 88 eigenvalue decomposition unit 104 MUSIC spatial spectrum calculation unit 106 MUSIC response calculation unit 110 smoothing filter unit 112 MUSIC Δ Spectrogram Calculation Unit 114 Onset Offset Detection Unit 118 Tracking Unit

Claims (5)

Based on each of the sound source signals of a plurality of channels obtained from the output of the microphone array and the positional relationship between the microphones included in the microphone array, the point determined in relation to the position of the microphone array is the center. MUSIC response calculation means for calculating MUSIC power for each predetermined time by a MUSIC algorithm for each of a plurality of orientations respectively corresponding to a plurality of mesh frames defined in a three-dimensional space;
For detecting a change in the direction of the sound source from the generation to the disappearance of the sound source based on the intensity of the MUSIC power value obtained as a time series by the MUSIC response calculation means and the amount of change for each of the plurality of directions. and the sound source estimation means a including sound source localization apparatus,
The sound source estimation means includes
For each of the plurality of azimuths, the MUSIC response calculation means immediately before each of the azimuths corresponding to the frame adjacent to the frame corresponding to the processing target azimuth among the plurality of azimuths. The difference between the MUSIC power value and the latest MUSIC power value obtained for the azimuth to be processed is calculated, and the smallest value among the differences is output as the MUSIC Δ power for the azimuth. MUSICΔ power calculation means for
The MUSICΔ power value calculated by the MUSICΔ power calculation means is larger than the first onset threshold value for each of the plurality of directions that are not the direction of the sound source, and the MUSIC response calculation means. First determination means for determining whether or not the value of the MUSIC power calculated by the above is larger than a second onset threshold value ;
In response to the determination results by the first determination means being all positive, it is determined that a sound source has occurred in the processing direction, and information indicating the generation of the sound source and information indicating the direction are output. Second determination means,
In response to the occurrence of the sound source is detected by the second judging means, the orientation of the sound source generating is detected by the second judging means, corresponding to the azimuth of the sound source by the MUSIC response calculator means The MUSIC power value calculated for each azimuth corresponding to the frame adjacent to the frame and the MUSICΔ power output by the MUSICΔ power calculation means for each azimuth corresponding to the frame adjacent to the frame corresponding to the direction of the sound source. based on the power, look including a tracking means for tracking up to disappearance of the sound source,
The tracking means includes
Responding to the direction of the sound source among the MUSIC responses output by the MUSIC response calculating means every predetermined time after the detection in response to the occurrence of the sound source detected by the second determination means. Means for tracking the movement of the sound source in which the occurrence has been detected by following the maximum value of the MUSIC power value calculated for the orientation corresponding to each of the frames adjacent to the frame ;
First offset determination means for determining whether or not the MUSICΔ power calculated for the orientation of the sound source tracked by the tracking means is smaller than a first offset threshold;
Second offset determination means for determining whether or not the MUSIC power calculated for the azimuth is smaller than a second offset constant that is larger than the MUSIC power at the time of generation of the sound source by a certain positive constant;
When the result that either one of the determination results of the first and second offset determination means is affirmative is continuously obtained a predetermined number of times for the MUSIC response calculated every predetermined time by the MUSIC response calculation means to, determine a sound source being tracked by said means for tracking is extinguished, including a means for stopping the tracking, sound source localization device. And a smoothing means for calculating and smoothing the moving average of the MUSIC power output by the MUSIC response calculating means for each of the plurality of directions.
2. The sound source localization apparatus according to claim 1, wherein both of the MUSICΔ power calculating unit and the tracking unit receive the MUSIC power smoothed by the smoothing unit as an input. The second determination means includes
Of the azimuths for which the determination result by the first determination unit is affirmative, only a certain number limited from the top of the MUSIC power calculated for the azimuth is identified as a detected sound source limiting unit. The sound source localization apparatus according to claim 1 , comprising: The sound source localization apparatus according to claim 1 , wherein the predetermined number of times is a plurality of times. A computer program that, when executed by a computer, causes the computer to function as each means according to any one of claims 1 to 4 .

Images