JP5724125B2 - Sound source localization device - Google Patents

Description

  The present invention relates to a sound source localization technique, and more particularly, to a sound source localization technique for tracking a voice uttered by a human with high accuracy in an environment where humans and noise are mixed.

  In voice communication between a person and a robot, the microphone attached to the robot is usually located at a distance (1 m or more). Therefore, for example, the signal-to-noise ratio (SNR) is lower than when the distance between the microphone and the mouth is several centimeters as in telephone speech. For this reason, the voices of others nearby and the noise of the environment become interference sounds, making it difficult for the robot to recognize the target speech. Therefore, sound source localization and sound source separation are important for robot applications.

  Various studies have been conducted on sound source localization in the past. However, most of them use only simulation data or lab data, and few evaluate real-world data in which the robot operates. There are few studies to evaluate 3D sound source localization. Talking and listening while grasping the position of the utterance partner is also an important behavior for improving the interaction between humans and robots, and for that purpose the localization of the moving sound source is also important.

  There exists a thing of patent document 1 as a prior art which assumed the real environment. The technique described in Patent Document 1 uses a known sound source localization technique called the MUSIC method with high resolution.

  In the invention described in Patent Document 1, a microphone array is used, and a current correlation matrix is calculated based on a received signal vector obtained by Fourier transform of a signal from the microphone array and a past correlation matrix. The correlation matrix obtained in this way is subjected to eigenvalue decomposition to obtain a maximum eigenvalue and a noise space that is an eigenvector corresponding to an eigenvalue other than the maximum eigenvalue. Furthermore, the direction of the sound source is estimated by the MUSIC method based on the phase difference of the output of each microphone, the noise space, and the maximum eigenvalue with one microphone as a reference in the microphone array.

JP 2008-175733 A

  However, the method described in Patent Document 1 seems to have room for further improvement. For example, when humans and other noise sources are mixed, it is necessary to accurately separate speech and noise generated by humans. If the accuracy of such sound source separation does not increase, the accuracy of processing such as speech recognition or speaker identification cannot be increased. This is particularly a problem when there is a sound source that moves like a human being, or when a sound source localization is provided on a movable source such as a robot. Furthermore, if the type of sound can be determined prior to speech recognition and speaker identification, it is more preferable because the burden of subsequent processing can be reduced.

  Therefore, an object of the present invention is to provide a sound source localization apparatus that can perform sound source localization and determination of attributes of those sound sources.

  A sound source localization apparatus according to a first aspect of the present invention includes a human position detection unit that detects a human position with a laser range finder, each of a plurality of sound source signals obtained from an output of a microphone array, and a microphone array. Predetermined directions for each of a plurality of directions defined in a space centered on a point determined in relation to the position of the microphone array based on the positional relationship between the microphones and the output of the human position detecting means. Sound source localization means for calculating MUSIC power every time and detecting the peak of the MUSIC power as a sound source position every predetermined time, and sound from the sound source position detected by the sound source localization means from the output signal of the microphone array Sound source separation means for separating signals, and sound source for determining attributes of the audio signal separated by the sound source separation means And a sex determination means.

  The human position detected by the laser range finder is used as information for sound source localization. The sound source localization accuracy can be increased as compared with the case where only the audio signal is used. If the accuracy of sound source localization can be increased, the attributes from the separated sound sources can be estimated stably and with high accuracy. As a result, it is possible to provide a sound source localization and sound attribute estimation device that can perform sound source localization and determination of attributes of these sound sources.

  Preferably, the sound source attribute determination means includes a plurality of individual acoustic models that are statistical models of acoustic features of a plurality of individual sounds, and sound from a noise source that is a sound source other than a human and has a known attribute. Receiving a plurality of noise acoustic models, which are statistical models of the characteristic features, the output of the human position detection means, and the output of the sound source localization means, and when there is a person in the sound source direction, a plurality of individual acoustic models and a plurality of A noise acoustic model is selected, and when there is no person in the sound source direction, a plurality of noise acoustic models are selected, and the acoustic model selected by the acoustic model selection means is separated by the sound source separation means. And statistical estimation means for estimating the attributes of the voice signal by a probabilistic method.

  When estimating the attribute of the audio signal, depending on the position of the person detected by the laser range finder, if there is a possibility that the sound source of the audio signal is a person, an individual acoustic model and a noise acoustic model are used. When the human position is not detected by the laser range finder, only the noise acoustic model is used. Therefore, it is possible to reduce the amount of calculation when estimating the attribute of the sound source, increase the processing speed, and increase the accuracy of attribute determination.

  More preferably, the sound source localization means is configured to perform a predetermined time for each of the plurality of directions based on each of the sound source signals of a plurality of channels obtained from the output of the microphone array and the positional relationship between the microphones included in the microphone array. MUSIC power is calculated for each, and the approximate position estimation means for estimating the position and direction where the peak at which the MUSIC power exceeds the threshold exists as the approximate position of the sound source, and the position and direction estimated by the approximate position estimation means Among them, detailed detection means for detecting the sound source position by detecting the peak of the MUSIC power in more detail around the position and direction where the person is detected by the human position detection means.

  After performing a rough sound source localization based on the sound source information obtained from the audio signal, the sound source localization can be performed more finely around the position where the person is detected. The accuracy of sound source localization centering on the human position can be increased, and an increase in the amount of calculation for this can be suppressed.

It is a schematic diagram which shows the fundamental structure of the process of the sound source separation and sound kind determination apparatus which concerns on one embodiment of this invention. FIG. 2 is a block diagram showing a schematic functional configuration of a sound source separation and sound type determination device shown in FIG. 1. It is a block diagram which shows the functional structure of the sound source localization process part shown in FIG. It is a block diagram which shows the functional structure of the sound source localization part shown in FIG. It is a block diagram which shows the functional structure of the sound source separation process part shown in FIG. It is a block diagram which shows the functional structure of the sound source kind identification process part shown in FIG. It is a block diagram which shows the functional structure of the sound source attribute determination part shown in FIG. It is a schematic diagram which shows schematic structure of the attribute candidate list | wrist output from the sound source attribute determination part shown in FIG. It is a flowchart which shows the outline of the control structure of the computer program which performs a process of sound source attribute determination. 10 is a flowchart showing a control structure of an ID exchange check routine that is recursively called in the program shown in FIG. 9. It is a figure which shows the external appearance of the computer system for implement | achieving the sound source separation and sound kind determination apparatus which concerns on embodiment of this invention. It is a block diagram of the hardware constitutions of the computer system shown in FIG.

  In the following description and drawings, the same parts are denoted by the same reference numerals. Therefore, detailed description thereof will not be repeated.

  In the following embodiments, the distance to an object, called a laser range finder (LRF), is measured to determine whether or not a person is within the measurement range, and the measured person is tracked. Use technology. Such a technique is widely used in the field of mobile robots that need to move while measuring the surrounding environment. A technique for identifying an object by matching the detected object with a previously registered object using the output of the LRF has been developed. Such a technique is described in Reference Document 1 below, for example. Furthermore, the technique which estimates not only a person's position but the direction is also developed (reference document 2). Note that the device for detecting the position of a person is not limited to the LRF. You may use an image processing technique with respect to the image image | photographed with the camera etc.

[Reference 1]
Shunsuke Sakaba, Tetsuo Serizawa, Kotaro Ohba, Kazuyoshi Wada, "Study on path planning using knowledge of distributed object shape and LRF" (8th Society of Instrument and Control Engineers System Integration Division Lecture Meeting (S12007), 2007 December 7, 2012, Society of Instrument and Control Engineers.

[Reference 2]
Norihiro Miyashita, Glas Dylan, Hiroshi Ishiguro, Norihiro Hamada, “Human Tracking Method Using Adaptive Human Shape Model with Laser Distance Meter”,
The 25th Annual Conference of the Robotics Society of Japan, 1I13, 2007.

  In this way, the development of the position and orientation of an object using LRF, matching with a known object, and the like are in progress. However, conventionally, no consideration has been given to combining LRF with sound source localization. In the present embodiment, the accuracy of sound source separation, sound source tracking, and sound source type determination is improved by applying human tracking and human identification technology using LRF to sound source tracking and sound source type identification.

[Constitution]
FIG. 1 conceptually shows the principle of the configuration of one embodiment of the present invention. A sound source separation and sound type determination device, which is an example of a sound source localization device according to the present invention, includes an LRF not shown in FIG. The combination of the human position measuring device for determining the sound source and the technique disclosed in Patent Document 1 for sound source localization performs sound source type determination and sound source localization. As can be seen from the fact that FIG. 1 is shown in the form of a flowchart, the present embodiment is implemented by computer hardware including a CPU (central processing unit) and computer hardware, thereby generating a sound source. This is realized by a computer program that performs type determination and sound source localization. Of course, the present invention is not limited to such a combination. For example, it goes without saying that the same effect can be obtained by a device that implements the same algorithm as that of the program by hardware, or a device that implements the program in hardware.

  Referring to FIG. 1, a program for controlling the operation of the sound source localization apparatus according to this embodiment uses an LRF, an infrared sensor (not shown), or a thermal sensor (not shown) to determine whether or not a person is present in the environment. Detecting and ending the operation of the device when it is determined that there is no person, and when the determination of step 30 is affirmative (when it is determined that there is a person) It is determined whether or not the process (end process) has been performed by the user, and if so, step 32 for stopping the operation of the apparatus, and when the determination of step 32 is negative, Based on the output of the human position determination device (not shown), sound source localization is performed, and the result of the process in step 34 is used to identify the sound type of each sound source. Separating voices of non-human and non-human voices, and step 38 of sequentially building a dialogue voice database by tracking the voices of the people separated in step 36 and storing them with appropriate labels. Including. When the process of step 38 is completed, the control returns to step 32, and thereafter the processes of steps 32 to 38 are repeatedly executed until an end instruction is given by the user.

  Referring to FIG. 2, a sound source localization system including this device includes a microphone array group 52 including a plurality of microphone arrays, an LRF group 56 including a plurality of LRFs, and features related to humans who may be in the surroundings in advance. The identifier is stored, and information indicating which person is present at which position (position information and human identifier. These are collectively referred to as human position information hereinafter) is output using the output of the LRF group 56. The person position measuring device 58, the synchronization time server 54 for controlling the synchronization of each part of the system, the output of the microphone array group 52, the synchronization control signal output from the synchronization time server 54, and the person It is connected so as to receive the human position information output from the position measuring device 58, and the sound source localization is performed based on the audio signal output from the microphone array group 52. It includes a sound source localization apparatus 50 for separating the sound source, and outputs the identified that type more for each sound source.

  The sound source localization device 50 receives audio signals output from each microphone array from the microphone array group 52, receives human position information from the human position measurement device 58, performs sound source localization processing, and generates sound that is considered to have been obtained from the sound source. A sound source localization processing unit 60 that outputs information indicating a direction (in many cases, this is a plurality corresponding to the number of sound sources), information indicating the directions of a plurality of sounds obtained from the sound source localization processing unit 60, and a microphone A sound source separation processing unit 70 that receives the sound signal obtained from the array group 52 and separates and outputs the sound signal 74 from the sound source in the direction obtained from the sound source localization processing unit 60 from other sound signals; Information about the direction and position of a plurality of sound signals output from the sound signal localization processing unit 60, and the audio signal 74 output from the unit 70, the human position information output from the human position measuring device 58 Used to identify the sound types from the sound sources, and a sound source localization processing unit 72 for outputting.

  Referring to FIG. 3, a sound source localization processing unit 60 shown in FIG. 2 includes a position vector DB (database) 80 that stores position vectors for specifying a plurality of directions to be searched for sound source localization, and a microphone array group. 52 is provided corresponding to each microphone array in the microphone array group 52 and stores a position vector in the search direction from the position vector DB 80. The position vector of the corresponding microphone array is received from the position DB 82, and the person position information is received from the person position measuring device 58 shown in FIG. 1, and the sound source using the person position information is added to the sound source localization method by the known MUSIC method. To determine the sound source direction with high accuracy by localization and output each .., 88, and the position information of each microphone array obtained from the array position DB 82 and the position vector of each direction stored in the position vector DB 80. A relative position vector generation unit 108 that generates and outputs a relative position vector of each direction, sound source direction information output by each of the plurality of sound source localization units 84,..., 86, and 88, and output from the relative position vector generation unit 108 Detailed search that uses the relative position vector and the human position information given from the human position measuring device 58 to search in detail the position where the peak of the MUSIC spectrum is likely to exist and to output a signal indicating the peak position Part 110.

The plurality of sound source localization parts 84, ..., 86, ..., 88 all have the same structure. Therefore, the structure of the sound source localization unit 84 will be described below as a representative.
Referring to FIG. 4, sound source localization unit 84 receives analog sound source signals corresponding to the number of microphones (for example, 14) included in the array from the corresponding microphone array, performs analog / digital (A / D) conversion, and the same number. An A / D converter 100 that outputs a digital sound source signal and a plurality of digital audio signals output from the A / D converter 100, frames the audio signal at predetermined time intervals, and calculates a MUSIC response for each frame The eigenvector calculation unit 102 that calculates and outputs a correlation matrix related to the microphone output necessary for the output, its maximum eigenvalue, and a noise space that is an eigenvector corresponding to an eigenvalue other than the maximum eigenvalue, and from the eigenvector calculation unit 102 every predetermined time Each of which is determined by the position vector obtained from the position vector DB 80 By comparing the MUSIC processing unit 104 that outputs the MUSIC response calculated by the MUSIC method with respect to the direction and the MUSIC response output by the MUSIC processing unit 104 with a threshold value, it is possible to determine the direction in which the MUSIC sound source is likely to exist. That is, it includes a peak detection unit 106 that estimates the peak direction and outputs information indicating the direction of the sound source.

  In the present embodiment, A / D converter 100 converts an analog signal, which is an output of each microphone, into a digital signal at a general 16 kHz / 16 bit.

  The eigenvector calculation unit 102 includes a framing processing unit 120 for framing a plurality of digital sound source signals output from the A / D converter 100 with a predetermined frame length and a predetermined shift length, and the framing processing unit 120 FFT (Fast Fourier Transform) is applied to each framed sound source signal to be output, and a predetermined number of frequency regions (hereinafter, each frequency region is referred to as a “bin”, and the number of frequency regions is defined as “the number of bins”. The correlation between the FFT processing unit 122 that converts and outputs the bin value of each channel that is output from the FFT processing unit 122 at a time interval according to the shift length in the framing processing unit 120. A correlation matrix calculation unit 124 that calculates and outputs a correlation matrix having elements as elements at predetermined time intervals, and a correlation matrix calculation unit 1 The eigenvalue decomposition unit 126 outputs eigenvalue decomposition of the correlation matrix output from 24 and outputs an output 112 composed of the maximum eigenlocation and noise space to the MUSIC processing unit 104. In the present embodiment, the frequency component of the sound source signal excludes a band of 1 kHz or less with a low spatial resolution and a band of 6 kHz or more where spatial aliasing may occur.

  The MUSIC processing unit 104 receives a position vector representing the position of each microphone included in the corresponding microphone array from the position vector DB 80, and uses the eigenvector and noise space output from the eigenvalue decomposition unit 126 to fix the number of sound sources. A MUSIC spatial spectrum calculation unit 140 that calculates and outputs a MUSIC spatial spectrum by the MUSIC method, and a value called a MUSIC response according to the MUSIC method based on the MUSIC spatial spectrum calculated by the MUSIC spatial spectrum calculation unit 140 is a position vector. And a MUSIC response calculation unit 142 for calculating and outputting each azimuth corresponding to the.

  The “azimuth” here refers to each frame of the mesh defined in the three-dimensional space in order to search for the sound source position. In this embodiment, the mesh is divided into a ring shape in a range of an elevation angle of 5 degrees, and different numbers of search points are provided depending on the size of the elevation angle. The “search point” here refers to the center point of the mesh described above.

  The number of search points is selected so that the direction angle to an adjacent search point is 5 degrees in a ring with an elevation angle of 0 degrees. The number of search points is the maximum for a wheel with an elevation angle of 0 degrees, and decreases as the elevation angle increases. At this time, the distances (which may be considered as angles) between search points in one ring are equal to each other, and the distances (angles) are the distances (angles) between adjacent search points in a ring having an elevation angle of 0 degrees. And is chosen to be as close as possible.

  Referring to FIG. 5, sound source separation processing unit 70 is provided corresponding to the microphone array, and is close to the sound source direction based on information indicating the direction and position of one sound source output from sound source localization processing unit 60. A plurality of adaptive beamformers 160 for separating and outputting the sound signal of the sound source by enhancing the signal from the target direction and suppressing the interference sound from the other direction with respect to the output from the microphone array. , 162,..., 164. The outputs of the plurality of adaptive beamformers 160, ..., 162, ..., 164 are provided to the sound source type identification processing unit 72 as audio signals 74 from the separated sound sources.

  Referring to FIG. 6, the sound source type identification processing unit 72 includes a plurality of individual GMMs 180 each of which is an individual acoustic model, and a plurality of noises each of which is an acoustic model corresponding to a specific type of noise. Based on the GMM 182 and the direction and position of the sound source received from the sound source localization processing unit 60, it is determined whether or not the sound source is a human. Depending on the determination result, either the plurality of individual GMMs 180 or the plurality of noise GMMs 182 A plurality of sound source attribute determining units 190,..., 192,. The sound source attribute determination units 190, ..., 192, ..., 194 all have the same configuration. Therefore, the configuration of the sound source attribute determination unit 190 will be described below as a representative of these.

  Referring to FIG. 7, sound source attribute determination unit 190 compares information on the position of the person given from person position measurement device 58 with information on the direction and position of the sound source given from sound source localization processing unit 60, A comparison unit 210 that outputs a signal indicating whether the sound source is from a person based on whether the sound sources match, an input connected to a plurality of individual GMMs 180, and an input connected to a plurality of noise GMMs 182 And a selection unit 212 that selects both when there is a person in the direction of the sound source based on the output of the comparison unit 210, and selects and outputs only the noise GMM 182 when there is no person.

Source attribute determination unit 190 further from the sound source of the audio signal that corresponds to the input to the comparison unit 210 a separate sound sources, and extracted acoustic features, such as MFCC for each frame, the output as a sequence of feature vectors And a GMM group selected by the selection unit 212 (a plurality of noise GMMs 182 or a plurality of individual GMMs 180 and a plurality of noise GMMs 182) for the feature vector sequence extracted by the feature extraction unit 214 and the feature extraction unit 214 And a sound source attribute estimation unit 216 that estimates audio attributes and outputs estimation results. The output of the sound source attribute estimation unit 216 is a candidate list including a candidate human identifier and its likelihood if the sound source is a human, and if the sound source is other than a human, the candidate noise specifying information and its likelihood. A candidate list consisting of

  Thus, when there is a person in the direction of the sound source, the attribute of the sound source is estimated using a plurality of individual GMM 180 and noise GMM 182, and when it is considered that there is no person, only the noise GMM 182 is used for attribute estimation. When there is no person, the model is narrowed down to the noise GMM 182 only, so that the processing amount is reduced, the processing time is shortened, and the estimation accuracy is also increased.

  The detection ID / likelihood list 230 output from the sound source attribute estimation unit 216 includes likelihood. Therefore, the attribute of the sound source may be changed during the process. Therefore, in the tracking performed in step 38 shown in FIG. 1, it is necessary to always monitor whether or not the order of candidates has changed on the detection ID / likelihood list 230.

Referring to FIG. 8A, detection ID / likelihood list 230 shown in FIG. 7 is a candidate of estimation results of a plurality of attributes, and a plurality of candidate entries arranged according to the respective likelihoods. Including. Each candidate entry in the detection ID / likelihood list 230 includes a candidate identifier CIDn (n indicates an order on the detection ID / likelihood list 230) and a probability that a voice is generated by the candidate. Likelihood CPProb _n (where n is the same as n of CIDn). A plurality of such candidates are arranged in the detection ID / likelihood list 230.

  Note that, for the processing of step 38 shown in FIG. 1, this apparatus stores the time series of the detection ID / likelihood list 230 as a tracking history of the sound source attribute. Since each voice of the frame and these histories are associated with each other, as a result, it is possible to maintain a dialogue database with each speaker's label and speaker position.

With reference to FIG. 8B, for the processing of step 38 shown in the figure, this apparatus includes a candidate list 240 for work obtained by copying the above-described detection ID / likelihood list 230 and the top of the above history. A working history list 242 that is a copy of the candidate list is used. Each candidate in the history list 242 has the same configuration as each candidate in the candidate list 240. Here, the immediately preceding candidate is represented by the identifier HIDn (n is the rank in the history list 242), and the likelihood is represented by HProbe _n (n is the same as n of HIDn). Using the candidate list 240 and the history list 242, the attribute exchange is checked in step 38 of FIG. The method will be described with reference to FIG. 9 and FIG.

  FIG. 9 shows an outline of a control structure of a program for realizing step 38 in FIG. 1 in a flowchart format. FIG. 9 shows a case where each step is performed on a single sound source for the sake of clarity, but in actuality, these processes are performed on all of the sound sources.

  Referring to FIG. 9, this program copies the candidate list at the end of the history (same as detection ID / likelihood list 230 output from sound source attribute estimation unit 216 in FIG. 7 in the previous cycle) to history list 242. The step 250, the step 252 for copying the detection ID / likelihood list 230 output from the sound source attribute estimation unit 216 in the current cycle to the candidate list 240, the candidate list 240 and the history list 242 as arguments are shown in FIG. And a step 254 of calling an ID exchange check routine indicating the control structure. As will be described later, the ID exchange check routine is a program that recursively calls itself, and when the control finally returns to step 254, the list after the exchange if there is an ID exchange is a history list. If not, the history list 242 passed as an argument is returned as it is as a return value. The contents of this routine will be described later with reference to FIG.

  If there is a blank part in the history list 242 which is a return value from the ID exchange check routine in step 254, this program further displays the corresponding element (candidate identifier and likelihood) of the detected ID / likelihood list 230. Step 256 for copying, step 258 for adding the history list 242 thus finally obtained to the end of the history, and adding each voice data to the dialogue database together with the ID and position information of the speaker, and the process is terminated. Step 260.

  Referring to FIG. 10, the ID exchange check routine called in step 254 of FIG. 9 has the following control structure. That is, this program determines whether or not the number of elements in the candidate list 240 is 1, and when the determination result is step 280 for branching the control flow, and when the determination in step 280 is negative, the program further starts the candidate list 240. It is determined whether or not the candidate identifier CID1 and the identifier HID1 of the first candidate in the history list 242 match, and the determination in step 282 and step 280 or step 282 is affirmed according to the determination result. And a step 306 which copies the candidate list 240 to the history list 242 and terminates execution of this routine and returns control to the calling routine. When this routine returns control to the caller routine, the history list 242 is returned as a return value.

  The program further provides new likelihoods NProb1 and NProb1 according to the following equations based on the likelihoods CProb1 and CProbe2 of the identifiers CID1 and CID2 of the first and second candidates in the candidate list 240 when the determination in step 282 is negative: Step 284 is included to recalculate Nprob2.

ただしｗは０＜ｗ＜１を満たす任意の値である。

However, w is an arbitrary value satisfying 0 <w <1.

  In step 284, the program further determines whether the newly calculated likelihood NProb1 is greater than the likelihood NProb2 and branches the control flow according to the determination result, and the determination in step 286 is positive. Is executed, and the identifier CID1 of the first candidate in the candidate list 240 is substituted for the identifier HID1 of the first candidate in the history list 242 and the likelihood NProb1 recalculated for the first candidate in the candidate list 240 is Subsequent to step 288 for substituting the likelihood HProbe1 for the first candidate in the history list 242 and step 288, the identifier CID2 for the second candidate in the candidate list 240 is substituted for the identifier HID2 for the second candidate in the history list 242. The likelihood NProb2 recalculated for the second candidate in the candidate list 240 is used. Substituted in the second candidate likelihood HProb2 list 242 and a step 290 which returns control to the calling routine.

  The program further determines whether or not the newly calculated likelihood NProb2 is greater than the likelihood HProbe2 when the determination at step 286 is negative, and branches the control flow to step 288 when the determination is negative. This is executed when the determinations in step 292 and step 292 are affirmative, and the identifier CID2 of the second candidate in the candidate list 240 is substituted for the identifier HID1 of the first candidate in the history list 242, and the second candidate in the candidate list 240 Subsequent to step 294, substituting the likelihood NProbe2 recalculated for the candidate into the likelihood HProbe1 of the first candidate in the history list 242, the identifier CID1 of the first candidate in the candidate list 240 is assigned to the history list 242. Substituting the identifier HID2 of the second candidate for the first candidate in the candidate list 240 And a step 296 that assigns a likelihood NProb1 recalculated to the second candidate likelihood HProb2 history list 242.

  In step 296, the program further recursively calls itself as a new argument (history list 242) from the history list 242 excluding the entry of the first candidate HID1. And a step 300 of adding an entry of the first candidate HID1 to the head of the return value history list 242 by processing and returning the control to the calling source routine using the history list 242 as a return value.

[Operation]
The sound source separation and sound type determination apparatus described above operates as follows. Prior to this operation, the person position measuring device 58 shown in FIG. 2 stores information necessary for identifying the person to be measured based on the output of the LRF group 56 and the identifier of each person. Shall. Further, the position vector DB 80 shown in FIG. 3 stores in advance position vectors that specify each point (orientation) of the spatial grid for the sound source separation and sound type determination device to calculate the MUSIC response. The array position DB 82 stores the positions of the microphone arrays constituting the microphone array group 52. As the individual GMMs 180, acoustic models created in advance for each person to be measured are prepared. As the noise GMM 182, an acoustic model relating to noise that is collected in advance and whose attributes are known in advance is prepared.

  When the sound source separation and sound type determination device starts operating, the LRF group 56 outputs information about a person existing in the vicinity with reference to FIG. 1 and FIG. Based on the output from the LRF group 56, the human position measuring device 58 outputs the positions of the persons existing around and the identifiers of these persons to the sound source localization processing unit 60 and the sound source type identification processing unit 72. When no person is detected (NO in step 30 in FIG. 1), the sound source separation and sound type determination device ends the operation. If a person is detected and no operation for instructing the apparatus to end the operation is performed (NO in step 32), a sound source localization process is executed in step 34.

  The sound source localization process is performed as follows. Referring to FIG. 2, each microphone array of microphone array group 52 converts sound into an electrical signal that is an analog electrical signal at each position by a plurality of microphones, and provides it to sound source localization processing unit 60.

  With reference to FIGS. 3 and 4, the following processing is executed in each of sound source localization units 84 of sound source localization processing unit 60. Referring to FIG. 4 in particular, A / D converter 100 converts a plurality of audio signals provided from a corresponding microphone array into a digital audio signal of a plurality of channels, and provides it to framing processing unit 120 of eigenvector calculation unit 102. . The framing processor 120 framing the digital audio of the plurality of channels with a predetermined frame length and a predetermined shift length, and provides the frame to the FFT processor 122. The FFT processing unit 122 performs FFT on each of the given digital audio signals of a plurality of channels for each frame, converts it to the frequency domain, and provides it to the correlation matrix calculation unit 124. Correlation matrix calculation section 124 calculates a correlation matrix having the correlation between the bin values output from FFT processing section 122 as elements, and provides the correlation matrix to eigenvalue decomposition section 126. The eigenvalue decomposition unit 126 obtains the maximum eigenvalue of the correlation matrix and a noise space that is an eigenvector corresponding to an eigenvalue other than the maximum eigenvalue, and gives the output 112 to the MUSIC space spectrum calculation unit 140.

  The MUSIC space spectrum calculation unit 140 receives a position vector representing the position of the microphone in the microphone array corresponding to the sound source localization unit 84 from the position vector DB 80, uses the eigenvector and noise space received from the eigenvalue decomposition unit 126, and uses the MUSIC method To calculate and output the MUSIC spatial spectrum. At this time, the MUSIC spatial spectrum calculation unit 140 calculates the MUSIC spatial spectrum assuming that the number of sound sources is fixed. The calculated MUSIC spatial spectrum is given to the MUSIC response calculation unit 142.

  The MUSIC response calculation unit 142 calculates the MUSIC response for each direction according to the position vector according to the MUSIC method based on the given MUSIC spatial spectrum, and outputs the MUSIC response to the peak detection unit 106.

  The peak detection unit 106 compares the value of the MUSIC response for each azimuth output from the MUSIC response calculation unit 142 with a threshold value, determines a peak position candidate of the MUSIC response as a sound source position, and indicates information indicating the direction. It gives to the detailed search part 110 (FIG. 3).

  Referring to FIG. 3, relative position vector generation unit 108 sets each microphone based on each position vector stored in position vector DB 80 and the position of the microphone array in microphone array group 52 stored in array position DB 82. A relative position vector with respect to the array is calculated and provided to the detailed search unit 110. The detailed search unit 110 uses the person position and ID given from the person position measuring device 58 (FIG. 2) and each relative position vector given from the relative position vector generation unit 108, and outputs them from the sound source localization unit 84. Based on the sound source azimuth information, the position where the value of the MUSIC response is high in a predetermined range centering on the intersection of the half line passing through the sound source position and starting from the position of the microphone array is searched for and the position is indicated. The signal is output to the sound source separation processing unit 70. The above corresponds to the processing of step 34 in FIG.

  Referring to FIG. 5, the plurality of adaptive beamformers 160,..., 162,... 164 of the sound source separation processing unit 70 each use information on the direction and position of the sound source output from the corresponding detailed search unit 110. The sound signal of the sound source is separated from the sound signal output from the microphone array and provided to the sound source type identification processing unit 72.

  Referring to FIG. 6, the sound source attribute determination units 190,..., 192,..., 194 of the sound source type identification processing unit 72 are respectively information indicating a person's position and ID given from the human position measuring device 58, and Based on the information indicating the direction and position of the sound source given from the sound source localization processing unit 60, the attribute of the sound source is determined as follows and the result is output.

  Referring to FIG. 7, for example, the comparison unit 210 of the sound source attribute determination unit 190 compares the position of the person with the direction and position of the sound source, and if they match, a plurality of individual GMMs 180 and a plurality of noise GMMs 182 are compared. Otherwise, only the plurality of noise GMMs 182 are selected and supplied to the sound source attribute estimation unit 216, respectively. On the other hand, the feature extraction unit 214 extracts a predetermined feature amount from an audio signal from a sound source to be processed, and gives a sequence of feature amount vectors for each frame to the sound source attribute estimation unit 216.

  The sound source attribute estimation unit 216 uses only the plurality of noise GMMs 182 selected by the selection unit 212 or a plurality of individual GMMs 180 and noise GMMs 182, and the feature vector sequence from the feature extraction unit 214 is each individual or each noise. The likelihood generated from the source is calculated, and a candidate list 240, which is a candidate list consisting of a predetermined number of higher ranks, is created and output. The detection ID / likelihood list 230 has the same configuration as that of the candidate list 240 shown in FIG. 8A, in which the individual who generated the sound source or the noise source candidates are arranged in order of likelihood. It includes the noise source identifier (CID) and its likelihood. The sound source attribute estimation unit 216 outputs the detection ID / likelihood list 230 at intervals corresponding to the frame shift time. The above corresponds to the processing of step 36 in FIG.

  Referring to FIG. 1, the tracking process in step 38 is performed as follows. Here, it is assumed that the history of the likelihood list of each sound source output by the sound source attribute estimation unit 216 is already stored, and the corresponding speech data is also stored in the dialogue database.

  Referring to FIG. 9, when the detection ID / likelihood list 230 is output from the sound source attribute estimation unit 216, the sound source separation and sound type determination device, at step 250, the list at the end of the history already stored. Is copied to the history list 242. In subsequent step 252, the detection ID / likelihood list 230 output by the sound source attribute estimation unit 216 is copied to the candidate list 240.

  In the following step 254, an ID exchange check routine whose control structure is shown in FIG. 10 is called.

  Referring to FIG. 10, the ID exchange check routine is executed as follows. Here, two cases will be described in order. In order to make the explanation easy to understand, it is assumed that the number of elements of the detection ID / likelihood list 230 output by the sound source attribute estimation unit 216 is three. First, the case where the identifier of the top candidate in the detection ID / likelihood list 230 is the same as the identifier of the top candidate in the list at the end of the history will be described. Next, a case will be described in which the identifiers of the first and second candidates in the detection ID / likelihood list 230 are obtained by replacing the identifiers of the first and second candidates in the list at the end of the history.

<When the new and old first and second candidates are the same>
First, in step 280, it is determined whether or not the number of elements in the candidate list 240 (number of candidate entries) is one. If the determination result is affirmative, control proceeds to step 306 where the candidate list 240 is copied to the history list 242 and returns to the caller routine. Here, since it is assumed that the number of elements in the detection ID / likelihood list 230 is 3, the determination in step 280 is negative and control proceeds to step 282.

  In step 282, it is determined whether the identifier CID1 of the first candidate in the candidate list 240 is equal to the identifier HID1 of the first candidate in the history list 242. If the determination result is negative, control proceeds to step 284. Here, since the determination result is affirmative from the assumption, control proceeds to step 306, the candidate list 240 is copied to the history list 242, and control returns to the caller routine.

  Referring to FIG. 9, in step 256, the corresponding element of candidate list 240 is copied to the blank portion of history list 242. Here, since candidate list 240 has already been copied to history list 242, nothing is processed in step 256.

  In the subsequent step 258, the history list 242 obtained as a result of the steps 254 and 256 is added to the end of the history. In step 260, the voice data at this time is recorded in the dialogue database together with the identifier at the head of the history list 242, and the control returns to the next process (step 32 in FIG. 1).

<When the first and second candidates of the old and new are replaced>
Referring to FIG. 10, the determination result in step 280 is NO. The determination result in subsequent step 282 is also NO. Control proceeds to step 284 where the likelihoods of CID1 and CID2 are recalculated according to the above equation, resulting in NProbe1 and NProbe2.

  In step 286, it is determined whether NProb1 is greater than NProb2. If the determination is affirmative, control proceeds to step 288, otherwise control proceeds to step 292. In step 292, it is further determined whether NProbe2 is greater than the second likelihood HProbe2 in the history list 242, and the flow of control branches according to the result.

  The operation will be described below in three cases.

(1) Affirmation of Step 286 In this case, the processing of Step 288 substitutes the identifier CID1 of the first candidate in the new candidate list 240 into the identifier HID1 of the first candidate in the history list 242, and the candidate list 240 The first candidate likelihood NProb1 is substituted into the first candidate likelihood HProbe1 of the history list 242. Further, the second candidate identifier CID2 of the new candidate list 240 is substituted into the second candidate identifier HID2 of the history list 242 by the processing of step 292, and the likelihood NProbe2 of the second candidate of the candidate list 240 is Substituted into the likelihood HPProb2 of the second candidate in the history list 242. That is, instead of the first and second candidates in the history list 242, the first and second candidates in the candidate list 240 are substituted into the first and second candidates in the history list 242, respectively. Thereafter, control returns to step 256 in FIG.

  Here, the third information in the history list 242 contains information on the previous third candidate. Therefore, nothing is processed in step 256. In subsequent step 258, the history list 242 is added to the end of the history, and in step 260 data is added to the interaction database. In short, in the case of (1), the first and second candidates are the same as in the previous time, and are not interchanged.

(2) The determination in step 286 is affirmative and the determination in step 292 is affirmative. In this case, in step 294, the identifier CID2 of the second candidate in the new candidate list 240 is the identifier HID1 of the first candidate in the history list 242. And the likelihood NProbe2 of the second candidate in the candidate list 240 is assigned to the likelihood HProbe1 of the first candidate in the history list 242. Further, in step 296, the identifier CID1 of the first candidate of the new candidate list 240 is substituted for the identifier HID2 of the second candidate of the history list 242, and the likelihood NProb1 of the first candidate of the candidate list 240 is the history list. 242 is substituted into the likelihood HPProb2 of the second candidate. In short, the immediately preceding first and second candidates are interchanged.

  In step 298, the caller itself is further called with the argument obtained by removing the first element from the candidate list 240 and the history list 242. This process will be described later. Here, as a result of the process of step 298, the candidate list 240 and the history list 242 that have become new arguments are used, and as a result of likelihood recalculation, the first candidate and the second candidate have to be switched. In this case, it is pointed out that the history list 242 so changed is returned as a return value, and when such replacement is not necessary, the history list 242 without candidate replacement is returned as a return value. . However, the likelihood may be corrected by the result of step 284.

  After the processing in step 298, in step 300, a new history list 242 is generated by adding the top candidate entry removed in step 298 to the top of the history list 242 obtained as a return value in step 298. . Using this history list 242 as a return value, the execution of this routine is terminated, and the routine returns to the caller routine (step 256 in FIG. 9).

  The following processing is the same as in the case of (1) above.

(3) The determination in step 286 is affirmative and the determination in step 292 is negative. In this case, the same processing as in (1) above is executed.

<Recursive processing>
The processing by this program when a recursive call is made at step 298 in FIG. 10 will be described. For easy understanding, the routine of FIG. 9 is “main routine”, the routine of FIG. 10 called from the main routine is “child routine”, the routine of FIG. 10 called from the child routine is “grandchild routine”, and the grandchild The routine of FIG. 10 called from the routine will be referred to as a “great-grandchild routine”. According to the above description, in the grandchild routine, the number of elements in the new candidate list 240 and history list 242 are both 2. For ease of explanation, the identifier and likelihood of each entry in the candidate list 240 and history list 242 passed as arguments are indicated in the same way as in the parent routine.

  In this example, the determination result in step 280 is negative. Thereafter, the same processing as the execution of the child routine is performed on the candidate list 240 and the history list 242 with two elements passed as arguments, and a new history list 242 (the number of elements is obtained by the processing) is obtained. The execution of this routine is terminated with 2) as a return value, and control returns to the child routine. However, the processing in step 298 in FIG. 10 becomes a problem. That is, when the process of this step is performed, the grandchild routine calls this routine again. However, since the candidate list 240 and the history list 242 passed to this routine as arguments in the grandchild routine are both lists excluding the top element, the number of those elements is all one.

  Therefore, in the great-grandchild routine, the determination result in step 280 is affirmative, and the history list 242 having only that element is returned to the grandchild routine as a return value.

  Therefore, the return value of step 298 of the grandchild routine is the history list 242 itself that the grandchild routine passes to the great-grandchild routine as an argument. In step 300, the candidate entry removed in step 298 is added to the top of this list and returned to the child routine as a return value. As a result, in step 298 of the child routine, the history list 242 having the number of elements of 2 is obtained as a return value, and the elements removed in step 298 of the child routine are added to the head of the history list 242 and added to the three items. A history list 242 having entries is returned as a return value to the main routine.

  Through the above processing, as a result of step 254 in FIG. 9, a history list 242 in which the possibility of alternation of candidates is adjusted based on the result of recalculating the likelihood of each candidate is obtained.

  Note that the description here is based on the assumption that the number of entries in the original history list 242 is 3 for easy understanding. However, when the number of entries is 4 or more, a similar result can be obtained by recursive processing.

[Realization by computer]
The sound source separation and sound type determination device according to this embodiment is realized by computer hardware, a program executed by the computer hardware, and data stored in the computer hardware. FIG. 11 shows the external appearance of the computer system 530, and FIG. 12 shows the internal configuration of the computer system 530.

  Referring to FIG. 11, the computer system 530 includes a computer 540 having a memory port 552 and a DVD (Digital Versatile Disc) drive 550, a keyboard 546, a mouse 548, and a monitor 542.

  Referring to FIG. 12, in addition to the memory port 552 and the DVD drive 550, the computer 540 boots up a CPU (Central Processing Unit) 556, a bus 566 connected to the CPU 556, the memory port 552, and the DVD drive 550, and A read only memory (ROM) 558 that stores programs and the like, and a random access memory (RAM) 560 that is connected to the bus 566 and stores program instructions, system programs, work data, and the like. The computer 540 further includes a network interface card (NIC) 574 that provides a connection to the local area network (LAN) to the computer 540 and an A / D conversion function that receives input from the microphone array and converts it into a digital audio signal. And a sound board 568. The CPU 556 performs communication with the synchronization time server 54 and the human position measuring device 58 illustrated in FIG. 2 by network communication using the bus 566 and the NIC 574.

  A computer program for causing the computer system 530 to operate as a sound source separation and sound type determination device is stored in the DVD 562 or the semiconductor memory 564 inserted into the DVD drive 550 or the memory port 552 and further transferred to the hard disk 554. . Alternatively, the program may be transmitted to the computer 540 through a network (not shown) and stored in the hard disk 554. The program is loaded into the RAM 560 when executed. The program may be loaded into the RAM 560 directly from the DVD 562, from the semiconductor memory 564, or via a network.

  This program includes a plurality of instructions that cause the computer 540 to operate as the sound source separation and sound type determination device of this embodiment. Some of the basic functions required to perform this operation are provided by operating system (OS) or third party programs running on the computer 540 or various toolkit modules installed on the computer 540. Therefore, this program does not necessarily include all functions necessary to realize the system and method of this embodiment. This program includes only instructions for executing the above-described operation as the sound source separation and sound type determination device by calling an appropriate function or “tool” in a controlled manner so as to obtain a desired result. Should be included. The operation of computer system 530 is well known and will not be repeated here.

  As described above, according to the present embodiment, not only the sound from the microphone array but also information related to the human position detected by the LRF is used for sound source localization and sound source attribute estimation. Compared to the case of only sound, the accuracy of sound source localization can be increased, and an increase in processing amount at that time can also be suppressed. Also in the case of sound source attribute estimation, since the individual-specific GMM is used only when there is a possibility that there is a person, the sound source attributes can be accurately performed while suppressing an increase in the amount of calculation.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are included. Including.

50 sound source localization device 52 microphone array group 54 synchronization time server 56 LRF group 58 human position measurement device 60 sound source localization processing unit 70 sound source separation processing unit 72 sound source type identification processing unit 80 position vector DB
82 Array position DB
84, 86, 88 Sound source localization unit 102 Eigenvector calculation unit 104 MUSIC processing unit 106 Peak detection unit 108 Relative position vector generation unit 110 Detailed search unit 160, 162, 164 Adaptive beamformer 180 Individual GMM
182 Noise GMM
190, 192, 194 Sound source attribute determination unit 210 Comparison unit 212 Selection unit 214 Feature extraction unit 216 Sound source attribute estimation unit 230 Detection ID / likelihood list 240 Candidate list 242 History list

Claims (2)

And the human position detection means for detecting the position of a person,
Receiving each of the sound source signals of a plurality of channels obtained from the output signal of the microphone array, and the positional relationship between each microphone included in the microphone array, a center point defined with respect to the position before Symbol microphone array MUSIC power is calculated for each of a plurality of directions defined in the space to be determined from the sound source signals of the plurality of channels every predetermined time, and a predetermined number of directions giving a peak equal to or greater than the threshold value of the MUSIC power Sound source localization means for detecting at predetermined time intervals as
Sound source separation means for separating an audio signal from a sound source position detected by the sound source localization means from an output signal of the microphone array;
Look including a sound source type determination means for determining a source type of the separated audio signal by the sound source separation means,
The sound source type determination means includes
A plurality of individual acoustic models that are statistical models of acoustic features of at least MFCC of a plurality of individual voices;
A plurality of noise acoustic models that are non-human sound sources and are statistical models of the acoustic features from noise sources of known sound source types;
In response to the output of the person position detecting means and the output of the sound source localization means, when there is a person in the sound source direction, the plurality of individual acoustic models and the plurality of noise acoustic models are selected, and the person in the sound source direction is selected. Acoustic model selection means for selecting the plurality of noise acoustic models when not present;
Using the acoustic model selected by the acoustic model selection unit, the likelihood that gives the sequence of the acoustic feature amount of the audio signal separated by the sound source separation unit is calculated, and corresponds to the acoustic model having the highest likelihood A sound source localization apparatus including an estimation unit that estimates a sound source type as a sound source type of the audio signal . Further, the sound source localization of people direction estimated by means, for each of the direction towards a person is detected by the person position detection means, about the direction, details from the time search of the sound source position by the sound source localization means The sound source localization according to claim 1 , further comprising: detailed detection means for detecting a more detailed sound source position by calculating the MUSIC power in each direction by changing the direction and detecting a peak of the MUSIC power. apparatus.

Images