JP6467736B2 - Sound source position estimating apparatus, sound source position estimating method, and sound source position estimating program - Google Patents

Description

  The present invention relates to a sound source localization technique in a real environment, and more particularly to a sound source position estimation technique using sound directivity by a plurality of sensor arrays in a real environment.

  2. Description of the Related Art Conventionally, there has been a proposal for an imaging device that can efficiently acquire a clear image with a small number of imaging means by detecting the sound source direction (for example, Patent Document 1).

  This Patent Document 1 discloses the following technique. That is, the system is provided with two sound source direction detection units, each of which includes a plurality of microphones and detects the sound source direction based on the sound pressure level of the audio signal of each microphone. The sound source position estimation unit geometrically estimates the sound source position in the imaging target room based on the sound source direction detected by the sound source direction detection unit. The imaging unit is controlled so as to shoot at the estimated sound source position. The captured video data is subjected to image recognition processing by an image recognition unit. The image recognizing unit controls the zoom function of the imaging unit so that the subject (sound source) is displayed large.

  With such a configuration, the position of an object (subject) including a human being as a sound source can be grasped, so that the subject can be clearly photographed by a relatively small number of imaging means, and the imaging apparatus The overall system cost can be suppressed.

  However, such a system focuses on taking an image according to the utterance, and is not intended to estimate and record who is speaking when and where.

  For this purpose, there have also been reported examples of collecting data on actual science classes by installing multiple microphone arrays and multiple kinetic sensors in an elementary science room (for example, non-patented) Reference 1).

  On the other hand, there has also been proposed a method for estimating a sound source position using reflected sound from only sound and space information using a plurality of arrays (see Patent Document 2).

Japanese Patent Application Laid-Open No. 2005-151042 JP 2014-98568 A Specification

Ishi, C., Even, J., Hagita, N. (2013). “Using multiple microphone arrays and reflections for 3D localization of sound sources,” IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS 2013), pp. 3937-3942, Nov., 2013.

  However, in the technologies disclosed in Non-Patent Document 1 and Patent Document 2, the correspondence between the speech section and the person is not performed.

  If a dialogue behavior recognition platform that estimates who is speaking when and where is realized, observation of data when people are talking and collaborating while moving from time to time, such as in a classroom or a meeting Is expected to be easier.

  The present invention has been made to solve the above-described problems, and its purpose is to estimate and record who is speaking when and where in a predetermined space. A sound source position estimating apparatus is provided.

  Another object of the present invention is to provide a sound source position estimation apparatus that can estimate the direction of the face of a talking person in a predetermined space.

  In the present invention, the direction of the sound source is estimated using a plurality of microphone arrays, and the position of the person is estimated using information on the estimation of the position of the person. Position estimation).

  According to one aspect of the present invention, there is provided a sound source position estimating device, a plurality of sound sensor arrays, human position estimating means for estimating the position of a person in a predetermined space, and each sound sensor in the sound sensor array A storage device for storing the arrangement information and the position information of the person, and a positional relationship between each of the signals of the plurality of channels from the plurality of sound sensor arrays and each of the sound sensors included in the sound sensor array A set of sound source localization means for executing processing for specifying the direction in which sound arrives at a plurality of sound sensor arrays, and the direction of arrival of sound respectively identified by different sound sensor arrays among the plurality of sound sensor arrays And speech section estimation means for estimating the person who is speaking based on the position information of the person.

  Preferably, the speech section estimation means estimates a sound source candidate position having a minimum sum of distances to a sound sensor array used for specifying a sound arrival direction among sound source candidate positions as a sound source position.

  Preferably, the speech section estimation means estimates a person who is speaking in response to the estimated sound source position and person position being equal to or less than the second threshold value.

  Preferably, the speech section estimation means estimates the direction of the face of the person who is speaking according to the position of the person estimated to be the person who is speaking and the corresponding sound source position.

  Preferably, the apparatus further includes sound source separation means for separating the voice of the person who is speaking, estimated by the voice section estimation means, and recording the speech content and the speaker in association with each other.

According to another aspect of the present invention, there is provided a sound source position estimating method for estimating a speaker in a predetermined space based on signals from a plurality of sound sensor arrays and estimated human positions, Based on the step of estimating the position of the person in the predetermined space from the measurement data of the plurality of sound sources, and the positional relationship between each of the sound source signals of the plurality of channels from the plurality of sound sensor arrays and each sound sensor included in the sound sensor array Identifying a direction in which sound arrives in a plurality of sound sensor arrays, and a set of directions in which sound is identified respectively by different sound sensor arrays out of the plurality of sound sensor arrays and person position information And estimating the person who is speaking, the step of estimating the person speaking is that the shortest distance between the extension lines in the direction of arrival is a first threshold value for each set of directions in which sound arrives. In Depending on Rukoto comprises estimating a candidate position of the sound source exists on a straight line corresponding to the shortest distance.

According to still another aspect of the present invention, a computer having an arithmetic device and a storage device is used to estimate a speaker in a predetermined space based on signals from a plurality of sound sensor arrays and estimated human positions. A sound source position estimation program, in which the arithmetic device estimates the position of a person in a predetermined space based on measurement data from the position sensor, and the arithmetic device includes a plurality of sound sensor arrays. A step of identifying a direction in which sound arrives at the plurality of sound sensor arrays based on each of the sound source signals of the plurality of channels and a positional relationship between the sound sensors included in the sound sensor array; of sound sensor array, on the basis of the position information of the direction of the set and people arriving sound identified respectively different sound sensor array, in the step of estimating the human in speech Thus, for each set of sound arrival directions, the sound source candidate position is located on a straight line corresponding to the shortest distance in response to the shortest distance between the extension lines of the arrival directions being equal to or less than the first threshold value. And causing the computer to execute steps including the step of presuming that it exists .

  According to the present invention, it is possible to improve the accuracy of a system for detecting voice activity by combining sound source position estimation with a plurality of arrays and human position information.

  In addition, according to the present invention, it is possible to estimate the direction of the speaker's face when speaking, providing a clue as to what context in the space was spoken, and enabling more advanced interactive behavior recognition It becomes.

It is a conceptual diagram for demonstrating the structure of the interactive action recognition system 1000 containing the sound source position estimation apparatus of this Embodiment. It is a figure which shows the environment at the time of implementing experiment. It is a block diagram which shows the outline | summary of a structure of the sound source position estimation apparatus 2000 shown in FIG. It is a flowchart for demonstrating the flow of a process when the sound source position estimation apparatus 2000 is implement | achieved by the computer. It is a block diagram which shows the structure of the sound source direction estimation part 1040. FIG. It is a conceptual diagram which shows the procedure which calculates | requires the shortest distance. FIG. 2 is a block diagram showing a hardware configuration of a computer system 2000 for executing a computer program. It is a figure which shows arrangement | positioning of the microphone in a microphone array. It is a figure which shows the position of a microphone array, and the positional information of the evaluated person. It is a figure which shows the result of an utterance area detection rate when two speakers utter independently. It is a figure which describes the analysis result regarding estimation of the direction of a face. It is a figure which describes the analysis result regarding estimation of the direction of a face. It is a figure which shows the statistical value of the estimation result of a face direction. It is a figure which shows the estimation result of the direction of a face when two persons speak simultaneously.

  Hereinafter, the configuration of a sound source position estimation apparatus according to an embodiment of the present invention will be described with reference to the drawings. In the following embodiments, components and processing steps given the same reference numerals are the same or equivalent, and the description thereof will not be repeated unless necessary.

  In the following description, as a sound sensor, a so-called microphone, more specifically an electret condenser microphone will be described as an example, but other sound sensors may be used as long as they can detect sound as an electric signal. Also good.

  In a real environment, a plurality of sounds generated at different locations are observed in a mixed manner, so in the sound source position estimation device of the present embodiment, as described below, in order to localize and separate a plurality of sound sources, Link multiple microphone arrays.

Here, “sound source localization” means to continuously specify the direction of the sound source, and “sound source position estimation” is based on the direction of the sound source specified by the sound source localization in a predetermined space. It means estimating the position of a three-dimensional sound source.
[System configuration]
FIG. 1 is a conceptual diagram for explaining a configuration of a dialogue action recognition system 1000 including a sound source position estimation apparatus according to the present embodiment.

  Referring to FIG. 1, in interactive action recognition system 1000, in a predetermined space, for example, a conference room, microphone arrays 1052.1 and 1052.2 are installed on the ceiling, and a position closer to the floor surface in the conference room, For example, microphone arrays 1052.3 and 1052.4 are installed on the table. Although not particularly limited, for example, the direction connecting the microphone arrays 1052.1 and 1052.2 and the direction connecting the microphone arrays 1052.3 and 1052.4 are arranged to be orthogonal to each other.

  Note that the number of microphone arrays is not limited to four as described above, and generally there is no particular limitation as long as the number is plural.

  Assume that there are, for example, a speaker 10.1 speaking in a standing state and speakers 10.2 and 10.3 speaking in a sitting position in the conference room.

  Further, in the conference room, laser range finders (LRF) 1010.1, 1010.2, and 1010.3 (hereinafter collectively referred to as “LRF”) for detecting the position of a person at the three corners, respectively. , Referred to as LRF1010). The laser range finder is an example of a detection device for estimating the position of a person in the meeting room, and may be another sensor as long as it can detect the position of the person in the meeting room. Also, is not limited to three.

The sound source position estimating apparatus 2000 estimates the human position based on the data from the LRF 1010, and also refers to a microphone array 1052 (when collectively referring to a plurality of microphone arrays such as the microphone arrays 1052.1 to 1052.4). The position of the sound source acquired by the microphone array 1052) is collected over time, each voice utterance section is identified, and a speaker for each voice utterance period is specified.
[System installation environment]
FIG. 2 is a diagram illustrating an environment when an experiment described later is performed.

As shown in FIG. 2, in the experiment, a meeting space in a laboratory where a plurality of microphone arrays were installed was used. Two 16-channel microphone arrays were installed on the desk and two 8-channel microphone arrays were installed on the ceiling.
[Configuration for sound source position estimation]
FIG. 3 is a block diagram showing an outline of the configuration of the sound source position estimation apparatus 2000 shown in FIG.

  FIG. 4 is a flowchart for explaining a processing flow when the sound source position estimating apparatus 2000 is realized by a computer, as will be described later.

  3 and 4, first, based on signals from a plurality of microphone arrays 1052.1 to 1052.4, three-dimensional spatial sound source direction estimation units 1040.1 to 1040.4 (in the case of generic names) , Referred to as a sound source direction estimation unit 1040), respectively, performs sound source direction estimation (estimation of azimuth and elevation angle) in a three-dimensional space (S102). In many sound source localization studies, only the azimuth is estimated, but when there are a large number of people, such as in a conference room or classroom, the probability that multiple sound sources exist in the same direction increases, and the estimation of the elevation angle is also important. It becomes.

  The sound source direction estimation unit 1040 is a system based on the MUSIC method that estimates a sound source direction in a three-dimensional space with a spatial resolution of 5 degrees and a time resolution of 100 ms by real-time processing.

  A higher resolution is desirable for the sound source direction detection, but the processing time is increased for the search in the three-dimensional space, and real-time processing becomes difficult for a general CPU. Therefore, in the present embodiment, as described above, first, i) 3 degrees (search range: −6 to 6 degrees) and ii) 2 degrees (in a hierarchical manner with respect to the direction detected with a resolution of 5 degrees. Search range: −4 to 4 degrees), iii) Final sound source azimuth is estimated while sequentially increasing the resolution, such as 1 degree (search range: −3 to 3 degrees).

  The number of directions searched every 100 ms is proportional to the number of directions detected at the same time, but a CPU (Central Processing Unit) with a clock frequency of 2.6 GHz can sufficiently operate in real-time processing.

  As described above, the human position estimation unit 1070 uses two-dimensional LRF 1010 and uses two-dimensional human position estimation (S104).

  In addition, about the method of person position estimation using LRF, the following literature has an indication, for example.

Known Document 1: DF Glas et al., “Laser tracking of human body motion using adaptive shape modeling,” Proceedings of 2007 IEEE / RSJ International Conference on Intelligent Robots and Systems, pp. 602-608, 2007.
The speech section estimation unit 1080 determines whether or not the person is speaking based on the sound source direction and person position information, as will be described later (S106). Based on the room space information and the array position information stored in the nonvolatile storage device 2080, the sound source direction obtained from each microphone array and the person position information obtained from the person position estimation unit are superimposed. “Room space information” includes, for example, information on the installation position of the microphone array in a predetermined space such as a conference room. Note that, as described in Patent Document 2, when the reflected sound is also used, the “room space information” may include information on the positions of the walls and the ceiling of the predetermined space.

  It is not uncommon for the microphone array 1050 to overlap the direction of a person and the direction of a noise source such as an air conditioner or an air conditioner, and it is necessary to reduce false detections. Therefore, only a case where a plurality of directions overlap is regarded as a sound source candidate, and if the overlapping position of the sound source directions overlaps with the position of a person, the probability that the person is speaking is high.

  Finally, the sound source separation unit 1090 performs sound source separation for each detected sound source section by directing the beam in the detected direction using the microphone array closest to the sound source, and outputs the sound from the sound source. Then, the voice from the speaker is recorded in the nonvolatile storage device 2080 (S108).

Subsequently, if it is determined that the end of the process is instructed, the process is terminated. If the end of the process is not instructed, the process returns to step S102 to perform the process in the next time block.
[Sound source direction estimation using the MUSIC method]
FIG. 5 is a block diagram illustrating a configuration of the sound source direction estimation unit 1040. The configuration of the sound source direction estimation units 1040.1 to 1040.4 is basically the same.

  As an example, a MUSIC (Multiple Signal Classification) method will be described as an example of a method for estimating the direction of a sound source in order to estimate the position of the sound source. However, other methods may be used as long as the direction of the sound source can be estimated.

The outline of the MUSIC method will be described. First, a multi-channel spectrum X (k, t) is obtained for each frame by fast Fourier transform, and a spatial correlation matrix R _k between channels in a spectral region is obtained for each block. The eigenvalue decomposition of the directional component and the omnidirectional component subspace is decomposed, the eigenvector E _k ⁿ corresponding to the omnidirectional subspace, and the direction vector a _k prepared in advance according to the target search space. Is used to obtain a (narrowband) MUSIC spatial spectrum P (k) for each frequency bin, and a MUSIC spatial spectrum for each frequency bin within a specific frequency band is integrated to obtain a wideband MUSIC spatial spectrum.

  In the following, the broadband MUSIC spatial spectrum is simply referred to as “MUSIC spatial spectrum”, and the time series of the MUSIC spatial spectrum is referred to as “MUSIC spectrogram”.

  In sound source localization, the direction of the sound source is obtained by searching for the peak of the MUSIC spatial spectrum.

  In the following, a case where there is one microphone array will be described as an example, but the number of microphone arrays may be larger.

  Referring to FIG. 5, sound source power spectrum acquisition section 1050 includes microphones 1052.1 to 1052. An A / D converter 1054 that receives p analog sound source signals from the microphone array MC1 including p (p: natural number), performs analog / digital conversion, and outputs p digital sound source signals, respectively, and A / Receiving p digital sound source signals respectively output from the D converter 1054, the correlation matrix and its eigenvalues and eigenvectors required by the MUSIC method are set for each block with a predetermined time, for example, 100 milliseconds as one block. An eigenvector calculation unit 61 for outputting, and a MUSIC processing unit 62 that uses the eigenvector output from the eigenvector calculation unit 61 for each block and outputs a MUSIC space spectrum by the MUSIC method. The sound source direction estimation unit 1060 estimates the direction of the sound source (in the present embodiment, two deflection angles φ and θ in the three-dimensional polar coordinates) based on the MUSIC spatial spectrum output from the MUSIC processing unit 62. . In the present specification, the “MUSIC response” is obtained by averaging the MUSIC spatial spectrum obtained by the MUSIC algorithm using a predetermined formula.

  Although not particularly limited, in this embodiment, the A / D converter 1054 A / D converts the output of each microphone at a general 16 kHz / 16 bits.

  In addition, the eigenvector calculation unit 61 framing the p digital sound source signals output from the A / D converter 1054 based on the signal from the microphone array MC1 with a frame length of, for example, 4 milliseconds. Unit 80 and p-channel framed sound source signals output from framing processing unit 80 are each subjected to FFT (Fast Fourier Transform), and a predetermined number of frequency regions (hereinafter, each frequency region is referred to as a “bin”). The FFT processing unit 82 that converts the frequency domain number into a number of bins and outputs it, and the bin value of each channel that is output from the FFT processing unit every 4 milliseconds is expressed as 100. A blocking processing unit 84 for blocking every millisecond, and each bin output from the blocking processing unit 84 A correlation matrix calculation unit 86 that calculates and outputs a correlation matrix having a correlation between the two as a factor every predetermined time (every 100 milliseconds), a correlation matrix output from the correlation matrix calculation unit 86 is subjected to eigenvalue decomposition, and an eigenvector 92 Is output to the MUSIC processing unit 62.

  Normally, 512 to 1024 points are used in FFT (corresponding to 32 to 64 milliseconds at a sampling rate of 16 kHz), but here one frame is set to 4 milliseconds (corresponding to 64 to 128 points in FFT). By reducing the frame length in this way, not only the amount of calculation of FFT is reduced, but also the amount of calculation in later calculation of correlation matrix, eigenvalue decomposition, and calculation of MUSIC response is reduced. As a result, sound source localization can be performed sufficiently in real time even if a relatively weak computer is used without degrading performance.

  The MUSIC processing unit 62 includes a microphone arrangement storage unit 100 for storing a position vector representing the position of each microphone included in the microphone array MC1 using a predetermined coordinate system, and a microphone stored in the microphone arrangement storage unit 100. And a MUSIC spatial spectrum calculation unit 104 that calculates and outputs a MUSIC spatial spectrum by the MUSIC method on the assumption that the number of sound sources is fixed, using the position vector of Eq. And the eigenvector output from the eigenvalue decomposition unit 88.

  The fact that the eigenvalue of the correlation matrix obtained for each block is related to the number of sound sources is also described in the following documents, for example, and is already known.

Known Document 2: F.R. Asano et al., “Real-time sound source localization and generation system and its application in automatic speech recognition”, Eurospech, 2001, Aalborg, Denmark, 2001, 1013-1016 (F. Asano, M. Goto, K. Itou, and H. Asoh, “Real-time sound source localization and separation system and its application on automatic speech recognition,” in Eurospeech 2001, Aalborg, Denmark, 2001, pp. 1013-1016)
In the present embodiment, not only the two-dimensional azimuth angle of each sound source but also the elevation angle is estimated. Therefore, as the MUSIC algorithm, an algorithm that can calculate in three dimensions is implemented. The set of azimuth and elevation is hereinafter referred to as sound source azimuth (DOA). The algorithm executed by the MUSIC processing unit 62 does not estimate the distance to the sound source. By estimating only the sound source azimuth, the processing time can be significantly reduced.

  The MUSIC processing unit 62 further includes a MUSIC response calculation unit 106 for calculating and outputting a value called a MUSIC response for each direction according to the MUSIC method based on the MUSIC spatial spectrum calculated by the MUSIC spatial spectrum calculation unit 104. .

  The sound source direction estimation unit 1060 includes a buffer 108 for temporarily accumulating a predetermined number of MUSIC response peaks calculated by the MUSIC response calculation unit 106 in time series in the FIFO format. Further, the sound source direction estimation processing unit 110 estimates the direction of the sound source (the two declination angles φ and θ described above) for the MUSIC response of each search point of each block accumulated in the buffer 108.

Here, in the MUSIC method, in the estimation of the narrow band MUSIC space spectrum, it is necessary to give the number of sound sources (NOS) having directivity emitted at that time, but in the following explanation, a fixed number is given and the MUSIC space is given. In the following description, it is assumed that only peaks that exceed a specific threshold on the spectrum are regarded as directional sound sources.
(MUSIC method)
Hereinafter, the MUSIC method for calculating the above-described three-dimensional orientation will be briefly summarized.

  For example, the Fourier transform Xm (k, t) of M microphone inputs is modeled as in equation (M1).

However, the vector s (k, t) consists of N sound source spectra S _n (k, t) (n = 1,..., N).

That is, s (k, t) = [S ₁ (k, t),..., S _N (k, t)] ^T. Here, k and t indicate frequency and time frame indexes, respectively. Vector n (k, t) indicates background noise. The matrix A _k is a conversion function matrix, and its (m, n) element is a conversion function of a direct path from the nth sound source to the mth microphone. The n-th column vectors of A _k is referred to as a position vector of the n-th sound source (Steering Vector).

First, a spatial correlation matrix R _k defined by Expression (M2) is obtained, and E _k composed of a diagonal matrix Λ _k of eigenvalues and eigenvectors is obtained by eigenvalue decomposition of Rk shown in Expression (M3).

The eigenvector can be divided as E _k = [E _ks | E _kn ]. E _ks and E _kn indicate eigenvectors corresponding to the dominant N eigenvalues and other eigenvectors, respectively.

  The MUSIC spatial spectrum is obtained by equations (M4) and (M5). r is a distance, and θ and φ are an azimuth angle and an elevation angle, respectively. Equation (M5) is a normalized position vector at the scanned point (r, θ, φ).

The MUSIC response (corresponding to power) is obtained by averaging the MUSIC spatial spectrum as shown in Equation (M6).

In Expression (M6), k _L and k _H are indices of the lower and upper boundaries of the frequency band, respectively, and K = k _H −k _L +1. The direction of the sound arriving at the microphone array can be obtained by searching for the peak of the MUSIC response.

  As described above, the MUSIC method can be used as an algorithm for estimating the direction of arrival of sound, while other methods such as the steered response power method can be used.

  For example, the steered response power method is disclosed in the following document.

Known Document 3: M. Brandstein and H. Silverman, “A robust method for speech signal time-delay estimation in reverberant rooms,” in IEEE Conference on Acoustics, Speech, and Signal Processing, ICASSP 1997, 1997, pp. 375-378 .
Known Document 4: A. Badali, J.-M. Valin, F. Michaud, and P. Aarabi, “Evaluating realtime audio localization algorithms for artificial audition on mobile robots,” in Proceedings of IEEE / RSJ International Conference on Intelligent Robots and Systems, IROS 2009, 2009, pp. 2033-2038.
[Processing of Speech Section Estimating Unit 1080]
Next, processing for estimating the position of the sound source candidate by the sound section estimation unit 1080 based on the sound source direction obtained from the microphone array will be described below.

A plurality of directions detected from a plurality of microphone arrays are evaluated for each pair. In order to determine whether two directions (dir ₁ , dir ₂ ) intersect in a three-dimensional space, first, the shortest distance (dist _dir ) is calculated by the following equation.

Here, v ₁ and v ₂ are vectors parallel to each direction, and p ₁ and p ₂ are the positions of the arrays.

  FIG. 6 is a conceptual diagram showing a procedure for obtaining such a shortest distance.

As shown in FIG. 6, the parameter display linear l ₁ as parallel to the vector v ₁ point p ₁ by the parameter t, x = p ₁ + expressed in tv _1, a straight line parallel to the point p ₂ and as vector v ₂ Let l ₂ be expressed as x = p ₂ + uv ₂ by the parameter display by the parameter u.

Consider a plane α: (n · x) + d = 0 parallel to the straight line l ₂ and including the straight line l ₁ .

Since the normal of the plane α is perpendicular to the two straight lines l ₁ and l ₂ , the normal vector n is n = (v ₁ × v ₂ ) / | v ₁ × v ₂ as the outer product of the two straight line direction vectors. |

Also, the plane alpha, because it contains a point p ₁ on the straight line l _1, d = - a (n · p _1).

  Accordingly, the plane α is expressed by the following formula.

If the distance between the straight line l ₂ and the plane α is h, the distance PQ between the point P on the straight line l ₁ and the point Q on the straight line l ₂ is always greater than or equal to the distance h. In other words, the minimum value of PQ, that is, dist _dir is expressed by the above-described equation (1) as the distance between the point p ₂ and the plane α.

That is, h = dist _dir (dir ₁ , dir ₂ ) holds.

Here, if the shortest distance dist _dir is smaller than a predetermined threshold (dist _dir-th ), the speech section estimation unit 1080 considers the two directions to intersect.

Although not particularly limited, in the experiment described later, dist _{dir-th is set} to 20 cm.

For the direction pair determined to have intersected directions, the speech segment estimation unit 1080 estimates the position of the sound _source (pos _source ) using the following equation.

Here, pos _n indicates a coordinate point where a straight line with respect to the shortest distance intersects with a straight line describing a sound source direction from each array.

  Next, the speech section estimation unit 1080 evaluates the overlap with the human position for the sound source position candidates obtained by evaluating all the directional pairs by the above-described processing.

  In the overlap of the sound source position and the human position due to the overlapping of the sound source directions, since the human position detection is two-dimensional, the two-dimensional distance (distance in the xy plane in FIG. 1) is evaluated. That is, the speech section estimation unit 1080 calculates the two-dimensional distance between each detected human position and each sound source position candidate, and if the two-dimensional distance is smaller than the threshold as shown in the following formula (4), the person Is considered speaking.

Here, although not particularly limited, in an experiment described later, a threshold (dist _pos-th ) is set to 30 cm for evaluation of the position error.

  In human position estimation, height information cannot be obtained in two dimensions, but since sound source position estimation is obtained in three dimensions, it is possible to limit in consideration of the position of the person's mouth. Assuming that the position of the mouth is sitting and standing, only when the height of the sound source position is within a predetermined range, for example, a range of z = 80 to 170 cm, the speech section estimation unit 1080 considers that the person has a high probability of speaking.

  Since the human position is estimated every 33 to 66 ms and the sound source direction is estimated every 100 ms, the speech segment estimation unit 1080 detects the speech segment with a time resolution of 100 ms.

  Furthermore, if the speech section estimation unit 1080 is sandwiched between blocks determined to have voice activity in a section of 300 ms (3 blocks) or less, the section is merged. Further, the speech segment estimation unit 1080 detects a speech segment detected by adding a pre-roll period and an after-roll period to 200 ms before and after such a merged speech segment. To do.

  Furthermore, in consideration of the fact that the human mouth is located in front of the intersection of the median sagittal plane and the midline coronal plane of the human body, the speech section estimation unit 1080 will be described later in this embodiment. Also, the direction of the face is estimated.

  Among the sound source position candidates whose distance between the sound source position and the human position is smaller than the threshold (distpos-th), the sound source position best candidate having the smallest total distance from the plurality of microphone arrays whose sound source directions are estimated is determined. That is, when a sound source position candidate is specified by sound source direction estimation from two microphone arrays, a “distance sum” between the sound source position candidate and each microphone array is obtained. When there are multiple sound source candidate positions for the same person position, the sound source candidate position corresponding to the sum of the smallest distances among these “sum of distances” (ie, estimated by a microphone array closer to the person position) The selected sound source candidate position) is selected as the best candidate, and the direction of the vector connecting the person position and the best sound source position is set as the “face direction” of the person in the utterance section.

Finally, the sound source separation unit 1090 performs sound source separation by directing the beam in the detected direction using the microphone array closest to the sound source for each detected utterance section, and outputs the sound from the sound source. For example, the voice from the speaker is recorded in the nonvolatile storage device 2080 in association with the speech section.
[Realization by computer]
The processing of the sound source direction estimation unit 1040, the speech segment estimation unit 1080, and the sound source separation unit 1090 of the sound source position estimation device 2000 is actually performed by computer hardware and a computer program executed by the computer hardware. Realized by collaboration with software. The operation of the computer program for realizing these functions will be briefly described below.

  FIG. 7 is a block diagram showing the hardware configuration of a computer system 2000 for executing such a computer program.

  As shown in FIG. 7, in addition to the disk drive 2030 and the memory drive 2020, the computer main body 2010 constituting the computer system 2000 includes a CPU (Central Processing Unit) 2040 and a ROM (Read Only memory) 2060 and RAM (Random Access Memory) 2070, a non-volatile rewritable storage device such as a hard disk 2080, a communication interface 2090 for performing communication via a network, and a microphone array MC1. And an audio input interface 2092 for exchanging signals with MC2. The disc 2030 is loaded with an optical disc such as a CD-ROM 2200. A memory card 2210 is attached to the memory drive 2020.

  When the processing programs of the sound source direction estimation unit 1040, the speech segment estimation unit 1080, and the sound source separation unit 1090 of the sound source position estimation apparatus 2000 operate, a database that stores information that is the basis of the operation is stored in the hard disk 2080. It will be described as a thing.

  In FIG. 7, the CD-ROM 2200 is assumed as a medium capable of recording information such as a program installed in the computer main body. However, other media such as a DVD-ROM (Digital Versatile Disc) is used. Alternatively, a memory card or a USB memory may be used. In that case, the computer main body 2200 is provided with a drive device capable of reading these media.

  The main part of the sound source position estimation apparatus 2000 is configured by computer hardware and software executed by the CPU 2040. Generally, such software is stored and distributed in a storage medium such as a CD-ROM 2200, read from the storage medium by a disk drive 2030 or the like, and temporarily stored in the hard disk 2080. Alternatively, when the device is connected to the network 310, it is temporarily copied from the server on the network to the hard disk 2080. Then, the data is further read from the hard disk 2080 to the RAM 2070 in the memory and executed by the CPU 2040. In the case of network connection, the program may be directly loaded into the RAM and executed without being stored in the hard disk 2080.

  The program for functioning as the sound source position estimation apparatus 2000 does not necessarily include an operating system (OS) that causes the computer main body 2010 to execute functions such as an information processing apparatus. The program only needs to include an instruction portion that calls an appropriate function (module) in a controlled manner and obtains a desired result. How the computer system 20 operates is well known and will not be described in detail.

  Further, the computer that executes the program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

  Further, the CPU 2040 may be a single processor or a plurality of processors. That is, it may be a single core processor or a multi-core processor.

Note that a database that stores information serving as a basis for the operation of the program of the sound source position estimation apparatus 2000 may be stored in an external storage device connected via the interface 2090. For example, when connected to an external server via a network, a database that stores information serving as a basis of operation may be stored in a storage device such as a hard disk (not shown) in the external server. In this case, the computer 2000 operates as a client machine, and exchanges such database data with an external server via a network.
[Experimental result]
(1) Data Collection As described above, the experiment was conducted in a meeting space in the laboratory where a plurality of microphone arrays were installed as shown in FIG.

  Two 16-channel microphone arrays were installed on the desk and two 8-channel microphone arrays were installed on the ceiling.

  FIG. 8 is a diagram showing the arrangement of microphones in the microphone array.

  As shown in the plan view of FIG. 8 (a) and the side view of FIG. 8 (b), an array frame was prepared so that the 16-channel array was arranged on a hemisphere with a diameter of 30 cm. The 8-channel array has a shape in which microphones are evenly arranged on a 15 cm circle.

  FIG. 9 is a diagram showing the position of the microphone array and the position information of the evaluated person.

  In FIG. 9, the central rectangle represents a table.

  The height of the array on the table is z = 730 mm, and the height of the array on the ceiling is z = 2690 mm. At 10 locations (P1 to P10) around the table, data uttered under sitting and standing conditions were collected.

Two speakers (one male and one female) face each direction in four directions (front: F, left: L, back: B, right: R). ".
(Evaluation result: When speaking alone)
First, FIG. 10 is a diagram showing the results of the speech segment detection rate (precision and recall) when two speakers (female F1 and male M1) speak alone.

  Here, “precision” is determined to be an utterance interval including correctness (actually an utterance interval, and an interval that has been preceded as an utterance interval, and actually a non-utterance interval) Yes, it indicates how much the correct utterance section is included in the utterance section, and “recall” means that the utterance section is correctly determined (actually in the utterance section) In other words, it indicates how many sections are correct utterance sections in a section that is judged as an utterance section and a section that is actually a non-utterance section and is judged as a non-utterance section.

  In the speech section detection, the direction in which the direction is opposite to all the arrays at each position (for example, backward B of position P1, rightward R of position P2, etc.) is expected to be low in accuracy of sound source direction estimation. The results when these conditions are excluded are also described.

  In FIG. 10, the result using all data is shown as “all data”, and the result of excluding the direction against the table is shown as “excluding outside direction”.

  First, the result ("all data") for all data was 96% for speaker F1 (female) and 83% for speaker M1 (male). In contrast to this result, in the result of excluding the case of disobeying the table (“excluding outside direction”), a high detection rate of 97% or more was obtained for all speakers.

  In addition, there was almost no difference in precision and recall values, indicating that there were few insertion errors. This is considered to be effective as a candidate in which a sound source exists only at a position where a plurality of sound sources overlap.

  Next, FIG. 11 and FIG. 12 are diagrams illustrating analysis results relating to face orientation estimation.

  In FIG. 11 and FIG. 12, the white circles indicate the position of the microphone array. The sound source direction estimation result and the detected direction (black arrow) are shown.

  Lines pointing away from people correspond to noise sources such as ceiling air conditioners and air conditioners.

  From the examples of FIGS. 11 and 12, it can be seen that the position where the sound source directions detected by the plurality of arrays intersect with each other is shifted in the direction facing the face from the center point of the human position depending on the face direction.

  It can also be seen that at least four directions can be distinguished from the direction of the face estimated as the direction from the center point of the person position toward the position where the sound source directions intersect.

  FIG. 13 is a diagram illustrating statistical values of face direction estimation results.

From the results shown in FIG. 13, the average value of the face direction estimation error is close to 0 degrees under any condition, and the estimation varies around the correct direction. Regarding the variation, in the case of all data ("all data"), the standard deviation is around 30 degrees, and in the case of excluding the conditions that are not in the array ("excluding outside direction"), it is around 20 degrees. From this result, it can be confirmed that at least front, back, left and right can be identified during utterance.
(Evaluation result: When multiple people speak at the same time)
Next, the results when two people speak at the same time will be described.

  The positions when two people utter the same sentence at the same time are (P10; P1), (P10; P2), (P9; P2), (P1; P3), (P2; P4), (P4, P5) Six combinations were evaluated. The direction of the face was not specified, and it was instructed to speak in a natural direction as if speaking toward each other assuming a meeting place.

  In the detection of the utterance interval, although the number of conditions is small, a detection rate of 98% was obtained, and it was shown that the utterance interval can be detected with high accuracy even when two people speak simultaneously.

  FIG. 14 is a diagram showing the estimation result of the face orientation when two people speak simultaneously.

  As shown in FIG. 14, in the face direction, in the example of FIG. 14 (a), the speakers are sitting side by side at about 1 meter, but even when speaking toward each other, the speech is directed to the face. I can lead to not. On the other hand, FIG. 14B shows a state in which there are speakers on two adjacent sides of the table, and it can be seen that the speakers are speaking in the directions of each other.

  It was also confirmed that it was possible to detect the utterance section without any problem even when 4 people uttered around the meeting table at the same time. In addition, we collected data to utter while walking around the table, and confirmed that the utterance section and face orientation work correctly when moving.

  As described above, according to the interactive action recognition system 1000 of the present embodiment, it is possible to improve the accuracy of a system that detects voice activity by combining sound source position estimation with a plurality of arrays and human position information. is there.

  Further, according to the dialog action recognition system 1000, it is possible to estimate the direction of the speaker's face when speaking, which is a clue of what context in the space is spoken, and more advanced dialog action. Recognition is possible.

  In the above description, the dialogue action recognition system 1000 is described as a system for observing data when a plurality of people sometimes perform conversations and collaborative work while changing their seats, such as in a classroom or a meeting. did. However, for example, assuming a meeting scene, it is possible to specify the person who spoke during the meeting and the content of that person's utterance. In this case, if the content of the utterance is converted into a text sentence by voice recognition technology, it can be applied to a system that automatically creates minutes.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  61 eigenvector calculation unit, 62 MUSIC processing unit, 86 correlation matrix calculation unit, 88 eigenvalue decomposition unit, 106 MUSIC response calculation unit, 110 sound source direction estimation processing unit, 1040 sound source direction estimation unit, 1050 sound source power spectrum acquisition unit, 1060 sound source direction Estimator, 1080 Voice segment estimator, 1070 Person position estimator, 1090 Sound source separator, MC1, MC2 Microphone array.

Claims (7)

A plurality of sound sensor arrays;
Human position estimating means for estimating the position of a person in a predetermined space;
A storage device for storing information on arrangement of each sound sensor in the sound sensor array and position information of a person;
Based on each of the signals of the plurality of channels from the plurality of sound sensor arrays and the positional relationship between the sound sensors included in the sound sensor array, the direction in which the sound arrives at the plurality of sound sensor arrays is specified. Sound source localization means for executing processing for,
A speech section estimation means for estimating a person who is speaking based on a set of directions of arrival of the sounds specified by different sound sensor arrays among the plurality of sound sensor arrays and the position information of the person And
The speech section estimation means, on a straight line corresponding to the shortest distance, according to the shortest distance between the extension lines of the arrival direction being equal to or less than a first threshold value for each set of directions in which the sound arrives wherein you estimate a candidate location of the sound source is present, the sound source position estimation apparatus. The speech section estimation means estimates a candidate position of a sound source having a minimum sum of distances to the sound sensor array used for specifying the direction of arrival of the sound among the candidate positions of the sound source as a sound source position. The sound source position estimation apparatus according to claim 1 . The sound source position estimation unit according to claim 2 , wherein the speech section estimation unit estimates a person who is speaking in response to the estimated sound source position and the person position being equal to or less than a second threshold value. apparatus. The sound source according to claim 3 , wherein the speech section estimation unit estimates a face direction of a person who is speaking according to the position of the person estimated to be a person who is speaking and the corresponding sound source position. Position estimation device. Separating the audio for the person in the estimated speech by the speech interval estimation means further comprises sound source separation means for recording in association with speech content and a speaker, one of the claims 1-4 1 The sound source position estimation apparatus according to the item. A sound source position estimation method for estimating a speaker in a predetermined space based on signals from a plurality of sound sensor arrays and estimated human positions,
Estimating a position of a person in the predetermined space from measurement data from a position sensor;
Based on each of the sound source signals of a plurality of channels from the plurality of sound sensor arrays and the positional relationship between the sound sensors included in the sound sensor array, the direction in which the sound comes to the plurality of sound sensor arrays is specified. And steps to
Estimating a person who is speaking based on a set of directions of arrival of the sounds respectively identified by different sound sensor arrays out of the plurality of sound sensor arrays, and the position information of the person,
The step of estimating the person who is speaking is configured to set the shortest distance for each set of directions in which the sound arrives according to a shortest distance between extension lines of the arrival directions being equal to or less than a first threshold value. A sound source position estimation method including a step of estimating that a candidate position of the sound source exists on a corresponding straight line . A sound source position estimation program for estimating a speaker in a predetermined space based on signals from a plurality of sound sensor arrays and estimated human positions in a computer having an arithmetic device and a storage device, The sound source position estimation program is
The arithmetic device estimating the position of the person in the predetermined space from the measurement data from the position sensor;
The arithmetic unit is configured to output sound to the plurality of sound sensor arrays based on each of a plurality of sound source signals from the plurality of sound sensor arrays and a positional relationship between the sound sensors included in the sound sensor array. Identifying the direction of arrival;
The computing device estimates a person who is speaking based on a set of directions of arrival of the sounds respectively identified by different sound sensor arrays from among the plurality of sound sensor arrays and the position information of the person. And the sound source on a straight line corresponding to the shortest distance in response to the shortest distance between the extension lines of the arrival directions being equal to or less than a first threshold value for each set of the sound arrival directions. A sound source position estimation program for causing a computer to execute a step including a step of estimating that there is a candidate position .