JP4595364B2 - Information processing apparatus and method, program, and recording medium - Google Patents

Description

  The present invention relates to an information processing apparatus and method, a program, and a recording medium, and in particular, based on a speaker orientation recognized from an image obtained by photographing, accurately extracts voice data of each speaker, For example, the present invention relates to an information processing apparatus and method, a program, and a recording medium that can accurately identify a speaker.

  In recent years, there are electronic conference systems that automatically record the utterances of a plurality of speakers participating in a conference as minutes. When the utterance recorded by the electronic conference system is reproduced, it is desired that the person who is listening can identify the speaker (speaker) as well as the reproduced sound has little noise.

  Therefore, for example, a microphone (microphone) is installed in advance for each speaker so that each speaker can be identified during playback, and the microphone assigned to each speaker is registered in the electronic conference system. Thus, there has been proposed a method for identifying a speaker from a microphone to which a voice is input.

  In addition, for example, it has been proposed to identify the speaker from the direction of the sound source by fixing the microphone directivity to each speaker and registering the speaker's orientation in the electronic conference system. Yes.

  By the way, in Patent Document 1, a sound source direction is estimated based on speech data input from a microphone array, and a camera and a highly directional sound collection microphone are directed to the sound source direction. An imaging system that acquires a person's image and sound together is disclosed.

  Patent Document 2 discloses a robot that recognizes and tracks a speaker.

JP-A-6-351015 JP 2002-366191 A

  In some conventional electronic conferencing systems as described above, in order to identify a speaker from an utterance input to a microphone, it is assumed that the correspondence between the microphone and its directivity and each speaker is known. Therefore, there is a problem that speaker identification cannot be performed when the relationship between them is unknown.

  That is, the user needs to register the relationship between the microphone and the speaker or the microphone directivity and the speaker in advance in the electronic conference system.

  In addition, depending on the technique described in Patent Document 1, a microphone array for collecting voice that is a source for estimating the direction of the speaker and a voice of each speaker after estimating the direction Two types of microphones, a highly directional sound collecting microphone for collecting sound, are required.

  Furthermore, according to the technique described in Patent Document 2, the purpose is to track a specific speaker by controlling a microphone and a camera, and identification of each of a plurality of speakers is not considered.

  In addition, there is a method of performing speaker identification using a standard pattern (model) of speech of each speaker registered in advance and a speech feature amount of speech data of speech in a conference or the like. However, in this method, because the environment for obtaining the voice used to create the standard pattern to be registered in advance is different from the environment for obtaining the voice for extracting the voice feature amount in a conference, etc. The accuracy of person identification sometimes deteriorated.

  The present invention has been made in view of such a situation, and based on the speaker orientation recognized from the image obtained by shooting, the voice data of each speaker is accurately extracted, for example, This makes it possible to accurately identify the speaker.

The information processing apparatus of the present invention images one or more speakers, and extracts voice data of each speaker based on an imaging unit that outputs an image and a positional relationship between the speakers in an image output by the imaging unit. An extraction range calculation unit that calculates an extraction range that is a range of azimuth; a voice data extraction unit that extracts voice data within the extraction range calculated by the extraction range calculation unit out of voice data of an utterance by a speaker ; The storage unit stores a standard pattern for speaker identification based on the voice data extracted by the voice data extraction unit, the voice data newly extracted by the voice data extraction unit, and the storage unit. Speaker identification means for performing speaker identification using the standard pattern, and sound source direction estimation for estimating the sound source direction based on the voice data newly extracted by the voice data extraction means Further comprising the step, and a speaker to present the sound source direction estimated by the sound source direction estimation means, a determining means for determining whether the speaker identified by the speaker identification using speaker identification means match It is characterized by.

  The information processing apparatus according to the present invention may further include a face detection unit that detects a speaker's face from an image output by the imaging unit, and the extraction range calculation unit includes a face position detected by the face detection unit. The extraction range can be calculated based on the relationship.

The information processing method of the present invention captures one or more speakers, outputs an image, and voice data of each speaker based on the positional relationship of the speakers in the image output by the processing of the imaging step. An extraction range calculation step for calculating an extraction range that is a range of azimuths for extracting voices, and a voice for extracting voice data within the extraction range calculated by the processing of the extraction range calculation step from the voice data of the utterance by the speaker A data extraction step , a storage step for storing a speaker identification standard pattern based on the voice data extracted by the voice data extraction step for each speaker, and a new extraction by the voice data extraction step processing A speaker identification step for performing speaker identification using the voice data and the standard pattern stored in the processing of the storage step; and a voice data extraction step. A sound source azimuth estimation step for estimating the sound source azimuth based on the voice data newly extracted by the processing of the step, a speaker existing in the sound source azimuth estimated by the processing of the sound source azimuth estimation step, and a speaker identification step And a determination step of determining whether or not the speaker identified by the speaker identification by the process is matched .

The program of the present invention picks up one or more speakers and outputs the image, and extracts voice data of each speaker based on the positional relationship of the speakers in the image output by the processing of the imaging step. An extraction range calculation step for calculating an extraction range that is a range of azimuths to be extracted, and voice data extraction for extracting voice data within the extraction range calculated by the processing of the extraction range calculation step from the speech data of the utterance by the speaker The standard pattern for speaker identification based on the voice data extracted by the step of the voice data extraction step is newly extracted by the storage control step for controlling the storage for each speaker and the voice data extraction step processing. A speaker identification step for performing speaker identification using the recorded voice data and the standard pattern stored in the process of the storage control step; Based on the voice data newly extracted by the data extraction step processing, the sound source direction estimation step for estimating the sound source direction, the speakers existing in the sound source direction estimated by the processing of the sound source direction estimation step, and the speakers And a determination step of determining whether or not the speaker identified by the speaker identification by the identification step processing matches .

The program recorded on the recording medium of the present invention captures one or more speakers, outputs an image, and based on the speaker's positional relationship in the image output by the processing of the imaging step. Voice within the extraction range calculated by the extraction range calculation step of the extraction range calculation step for calculating the extraction range that is the range of the direction in which each voice data is extracted and the speech range of the speaker's utterance A voice data extraction step for extracting data; a storage control step for controlling storage for each speaker of a standard pattern for speaker identification based on the voice data extracted by the processing of the voice data extraction step; and a voice data extraction step. Speaker identification is performed using the voice data newly extracted by the process of step S4 and the standard pattern stored by the process of the storage control step. Based on the voice data newly extracted by the person identification step and the voice data extraction step, and the speech existing in the voice source direction estimated by the voice source direction estimation step and the voice source direction estimation step And a determination step of determining whether or not the speaker and the speaker identified by the speaker identification by the speaker identification step process coincide with each other.

In the information processing apparatus and method, the program, and the recording medium of the present invention, one or more speakers are captured and an image is output. Further, an extraction range for extracting the voice data of each speaker is calculated based on the positional relationship of the speakers in the image. Then, the voice data within the extraction range is extracted from the voice data of the utterance by the speaker, and a standard pattern for speaker identification based on the extracted voice data is stored for each speaker and newly extracted. Speaker identification is performed using the stored voice pattern and the stored standard pattern, and the sound source direction is estimated based on the newly extracted voice data, and the speaker existing in the estimated sound source direction, It is determined whether the speaker identified by speaker identification matches .

  According to the present invention, it is possible to accurately extract voice data of each speaker based on the speaker orientation recognized from an image obtained by photographing.

  Further, according to the present invention, a speaker can be identified with high accuracy.

  BEST MODE FOR CARRYING OUT THE INVENTION The best mode of the present invention will be described below. Correspondences between constituent features described in the claims and specific examples in the embodiments of the present invention are exemplified as follows. This description is to confirm that specific examples supporting the invention described in the claims are described in the embodiments of the invention. Therefore, even if there are specific examples that are described in the embodiment of the invention but are not described here as corresponding to the configuration requirements, the specific examples are not included in the configuration. It does not mean that it does not correspond to a requirement. On the contrary, even if a specific example is described here as corresponding to a configuration requirement, this means that the specific example does not correspond to a configuration requirement other than the configuration requirement. not.

  Further, this description does not mean that all the inventions corresponding to the specific examples described in the embodiments of the invention are described in the claims. In other words, this description is an invention corresponding to the specific example described in the embodiment of the invention, and the existence of an invention not described in the claims of this application, that is, it may be divided in the future. It does not deny the existence of an invention which appears by amendment and is added.

The information processing apparatus according to claim 1, picks up an image of one or more speakers, outputs an image (for example, the camera module 2 in FIG. 2), and the speaker's image in the image output by the image pickup unit. Based on the positional relationship, an extraction range calculation means (for example, a beam width calculation unit 122 in FIG. 4) that calculates an extraction range that is a range of directions in which each speaker's voice data is extracted; of voice data, the voice data extracting means (e.g., beam forming process unit 143 of FIG. 4) for extracting audio data in the extraction range that is calculated by the extraction range calculating means, extracted by the voice data extracting means A storage unit (for example, speaker information DB 132 in FIG. 4) for storing each speaker for a standard pattern for speaker identification based on the recorded voice data, and the voice data extraction unit. Speaker identification means (for example, voice identification processing unit 151 in FIG. 4) for performing speaker identification using newly extracted voice data and a standard pattern stored in the storage means, and the voice data extraction means The sound source direction estimating means for estimating the sound source direction (for example, the sound source localization detecting unit 142 in FIG. 4) based on the voice data newly extracted by the voice, and the story existing in the sound source direction estimated by the sound source direction estimating means Determining means (for example, identification information adding unit 152 in FIG. 4) for determining whether or not the speaker and the speaker identified by speaker identification by the speaker identifying means match. .

The information processing apparatus according to claim 2 can further include a face detection unit (for example, the face detection unit 121 in FIG. 4) that detects the face of the speaker from an image output by the imaging unit. The range calculation means can calculate the extraction range based on the positional relationship of the faces detected by the face detection means.

An information processing method according to a third aspect of the present invention relates to an imaging step of imaging one or more speakers and outputting an image (for example, the processing of step S1 in FIG. 6), and an image output by the processing of the imaging step. An extraction range calculation step (for example, the process of step S4 in FIG. 6) for calculating an extraction range that is a range of azimuths for extracting voice data of each speaker based on the positional relationship between the speakers, and the speaker The voice data extraction step (for example, the process in step S45 in FIG. 7 or the step S74 in FIG. 8) extracts the voice data in the extraction range calculated by the processing in the extraction range calculation step from the voice data of the utterance by and processing), storing stearyl the standard pattern for speaker identification based on the audio data extracted by the processing of the voice data extracting step is stored in each of the speaker (For example, the process of step S47 of FIG. 7), the voice data newly extracted by the process of the voice data extraction step, and the standard pattern stored by the process of the storage step A speaker identification step (for example, the process of step S76 in FIG. 8) and a sound source direction estimating step (for example, estimating a sound source direction based on the voice data newly extracted by the voice data extracting step) 8 in step S72 of FIG. 8), a speaker existing in the sound source direction estimated by the processing of the sound source direction estimation step, and a speaker identified by speaker identification by the processing of the speaker identification step And a determination step (for example, the process of step S77 in FIG. 8) for determining whether or not the two match .

In the program according to claim 4 and the program recorded in the recording medium according to claim 5 , an embodiment (however, an example) corresponding to each step is the information processing method according to claim 3. It is the same.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram showing an embodiment of an example of use of a tabletop speaker identification device 1 to which the present invention is applied.

  FIG. 1 shows a state in which three speakers are holding a conference in a conference room, living room, or the like, surrounding a table. Note that the number of speakers is not limited to three, and may be any number of one or more.

  The table-top type speaker identification device 1 placed at substantially the center of the table identifies a speaker and records the speaker and the utterance content in association with each other as a minutes. The appearance of the table top type speaker identification device 1 is shown in an enlarged manner in FIG.

  As shown in FIG. 2, the table top type speaker identification device 1 has a cylindrical shape, and includes a camera module 2, a microphone array (microphone array) 3 provided on the upper surface of the camera module 2, and a camera module. 2 includes an information processing unit 4 disposed under 2 and a display unit 5 provided on the surface of the information processing unit 4.

  The camera module 2 is a module in which one camera 2A is set directly below a hyperboloidal mirror (a hemisphere above the camera module 2), for example, which can image the entire circumference of 360 degrees. The all-around image data captured by the camera module 2 is supplied to the information processing unit 4.

  Note that the camera module 2 may be, for example, one that mechanically rotates the imaging direction of the camera and images the entire periphery, or is configured from a plurality of cameras that image each direction. Also good.

  The microphone array 3 is composed of, for example, four microphones 3-1, 3-2, 3-3, 3-4 such as condenser microphones. The sound collected by the microphones 3-1 to 3-4 is supplied to the information processing unit 4 through a cable (not shown). Of course, the microphone array 3 may be composed of a plurality of microphones other than four.

  The information processing unit 4 detects the orientation of the speakers surrounding the tabletop speaker identification device 1 based on the all-around image data supplied from the camera module 2, and based on the positional relationship between the speakers, A beam forming width, which is a width of an azimuth in which voice data is beam-formed, is calculated for each speaker.

  In addition, the information processing unit 4 extracts each speaker's utterance from the voice input to the microphone array 3 using the calculated beamforming width, performs speaker identification, and determines the identification result and the content of the utterance. Record in association.

  As will be described in detail later, according to beamforming, utterances by each speaker are extracted with high accuracy (assuming less noise), so that compared to collecting utterances without performing beamforming. Thus, a standard pattern for identifying a more suitable speaker can be obtained, and speaker identification after obtaining the standard pattern can be performed with high accuracy.

  The display unit 5 includes an LCD (Liquid Crystal Display) or the like, and displays, for example, the image data of the speaker imaged by the camera module 2, the result of speaker identification by the information processing unit 4, and the like.

  FIG. 3 is a block diagram illustrating a hardware configuration example of the tabletop speaker identification device 1 of FIG.

  A CPU (Central Processing Unit) 51 executes various processes according to a program stored in a ROM (Read Only Memory) 52 or a program loaded from a storage unit 58 to a RAM (Random Access Memory) 53. The RAM 53 also appropriately stores data necessary for the CPU 51 to execute various processes.

  The CPU 51, ROM 52, and RAM 53 are connected to each other via a bus 54. An input / output interface 55 is also connected to the bus 54.

  The input / output interface 55 includes a camera module 2, an input unit 56 including a microphone array 3 and various buttons (not shown), an output unit 57 including a display unit 5 and a speaker (not shown), a hard disk, and the like. And a communication unit 59 including a modem, a LAN (Local Area Network) adapter, and the like.

  The storage unit 58 is controlled by the CPU 51, stores data such as a program supplied via the communication unit 59, and supplies the stored data to the RAM 53, the output unit 57, the communication unit 59, and the like as necessary. To do.

  The communication unit 59 communicates with other devices (not shown) via a network. In addition to communication using a modem or the like, the communication unit 59 is not limited to various types such as USB (Universal Serial Bus), IEEE (Institute of Electrical and Electronic Engineers) 1394, or SCSI (Small Computer System Interface). It has a function to perform communication processing conforming to the standard.

  A drive 60 is also connected to the input / output interface 55, and a magnetic disk 61 (including a flexible disk), an optical disk 62 (including a compact disk-read only memory (CD-ROM), a DVD (digital versatile disk)), optical A magnetic disk 63 (including MD (Mini-Disk) (trademark)), a semiconductor memory 64, or the like is appropriately mounted, and a program read from the disk is installed in the storage unit 58 as necessary.

  FIG. 4 is a diagram illustrating a functional configuration example of the information processing unit 4 in FIG. 2.

  At least a part of the configuration in FIG. 4 is realized, for example, by executing a predetermined program by the CPU 51 in FIG. 3.

  The information processing apparatus 4 includes an image data processing unit 111 including a face detection unit 121 and a beam width calculation unit 122, a registration unit 112 including a DB (Data Base) management unit 131 and a speaker information DB 132, a microphone array. An audio data processing unit 113 including an I / F (Interface) 141, a sound source localization detection unit 142, and a beamforming processing unit 143; an audio identification processing unit 151; an identification information adding unit 152; an output control unit 153; A speaker identification unit 114 configured from the minutes DB 154 is realized.

  The face detection unit 121 of the image data processing unit 111 detects face image data in a region corresponding to the speaker's face in the all-around image data supplied from the camera module 2, and determines the orientation of the face. Detect as direction. The face detection unit 121 associates the detected speaker orientation and face image data of each speaker with a speaker ID (Identification) for identifying the speaker, and the speaker orientation associated with the speaker ID. And the face image data are supplied to the beam width calculator 122.

  The face detection unit 121 detects, for example, the center of gravity of the area corresponding to the eyes and nose and the center of the skin color area in the face image data, and the orientation toward the center of gravity is determined from the camera module 2 by the speaker. Detected as a bearing.

  The beam width calculation unit 122 calculates the beamforming width for each speaker based on the positional relationship of the speakers obtained from the speaker orientation supplied from the face detection unit 121, and calculates the speaker orientation and the calculated beamforming. The width is associated with the speaker ID and supplied to the sound source localization detection unit 142. Further, the beamforming width calculation unit 122 supplies the face image data supplied from the face detection unit 121 to the DB management unit 131. Calculation of the beam forming width by the beam width calculation unit 122 will be described later with reference to FIG.

  The DB management unit 131 of the registration unit 112 stores the face image data supplied from the beam width calculation unit 122 and the standard pattern of each speaker's voice supplied from the speaker identification unit 114 in the speaker information DB 132. . The face image data and the standard pattern registered in the speaker information DB 132 are provided to the voice identification processing unit 151 as necessary.

  The microphone array I / F 141 of the audio data processing unit 113 performs A / D (Analog / Digital) conversion on the analog audio signal supplied from the microphone array 3, and converts the obtained digital audio data into the sound source localization detection unit 142 and the beam. This is supplied to the forming processing unit 143.

  The sound source localization detection unit 142 estimates the sound source azimuth from the audio data supplied from the microphone array I / F 141, and the speaker located in the azimuth closest to the estimated sound source azimuth (estimated azimuth) from the beam width calculation unit 122. Detection is based on the supplied speaker orientation.

  Here, in the sound source localization detection unit 142, for example, a time difference (phase difference) when the correlation of the sound data collected by each of the microphones 3-1 to 3-4 is maximized is obtained, and the time difference is calculated as the time difference. Based on this, the sound source direction of the sound source (speaker) of the utterance is estimated. The sound source localization detection unit 142 also analyzes the sound data from each of the microphones 3-1 to 3-4, for example, to estimate the sound source direction of each of a plurality of sound data spoken simultaneously.

  Further, the sound source localization detection unit 142 selects the beam forming width of the detected speaker from among the beam forming widths of all the speakers supplied from the beam width calculation unit 122, and selects the selected beam forming width. This is supplied to the beamforming processing unit 143 together with the speaker orientation of the speaker.

  Furthermore, the sound source localization detection unit 142 uses the detected speaker ID (speaker ID associated with the speaker orientation) of the detected speaker to the voice identification processing unit 151 of the speaker identification unit 114 and the identification information adding unit 152. It also supplies to.

  The beamforming processing unit 143 performs beamforming of audio data supplied from the microphone array I / F 141 using the speaker orientation and the beamforming width supplied from the sound source localization detection unit 142.

  In other words, the beam forming processing unit 143 is supplied with audio data from the microphone array I / F 141 from the direction of the range represented by the beam forming width with the speaker orientation supplied from the sound source localization detecting unit 142 as a reference. By emphasizing the voice data from all directions, the voice data from the direction of the range is extracted.

  Therefore, the voice data extracted here is the one in which only the voice data of one speaker is emphasized (the voice data of other speakers, the voice data of reflected sound, etc. are removed (suppressed). Therefore, it is possible to obtain a standard pattern that enables high-accuracy speaker identification or the like, or a voice for speaker identification.

  The beam forming processing unit 143 supplies the voice identification processing unit 151 with the voice data extracted by the beam forming.

  In addition, when a plurality of sound source azimuths are estimated by the sound source localization detection unit 142, the beam forming processing unit 143 receives supply of respective speaker azimuths and beam forming widths corresponding to the plurality of sound source azimuths. Voice data is extracted from the voice data supplied from the microphone array I / F 141 for each of a plurality of speaker orientations and beamforming widths.

  The voice identification processing unit 151 of the speaker identification unit 114 performs voice data supplied from the beamforming processing unit 143 (by beamforming performed on the speaker orientation and beamforming width supplied from the sound source localization detection unit 142). Speech feature values are extracted by performing discrete Fourier transform (DFT) on the obtained speech data), and used for speaker identification based on the speech feature values. For example, HMM (Hidden A standard pattern (model) such as Makov Model is generated.

  In addition, the voice identification processing unit 151 adds a speaker ID (speech used for beam forming to obtain voice data used to generate the standard pattern) supplied to the standard pattern from the sound source localization detection unit 142. The person ID and the speaker ID associated with the beam forming width are associated with each other, and the data is supplied to the DB management unit 131. The standard pattern associated with the speaker ID supplied to the DB management unit 131 is stored in the speaker information DB 132.

  When performing speaker identification of the voice input to the microphone array 3, the voice identification processing unit 151 is registered in the speaker feature DB 132 and the voice feature amount extracted from the voice data supplied from the beamforming processing unit 143. For example, speaker identification is performed based on the HMM method. The speaker ID associated with the standard pattern (model) with the highest likelihood of observing the speech data from the beamforming processing unit 143 obtained by speaker identification based on the HMM method is the speaker identification result. Are supplied to the identification information adding unit 152 together with the audio data supplied from the beam forming processing unit 143.

  The identification information adding unit 152 determines whether or not the speaker ID supplied from the sound source localization detection unit 142 matches the speaker ID supplied from the voice identification processing unit 151. When the identification information adding unit 152 determines that the speaker IDs match, that is, the speaker obtained as a result of speaker identification in the voice identification processing unit 151 and the estimation of the sound source estimated by the sound source localization detection unit 142 If it is determined that the speakers in the azimuth closest to the azimuth match, the face image data associated with the speaker ID is read from the speaker information DB 132 via the DB management unit 131, and the voice identification processing unit 151 It is supplied to the output control unit 153 together with the supplied audio data.

  When the speaker obtained as a result of the speaker identification in the speech identification processing unit 151 matches the speaker in the direction closest to the estimated direction of the sound source estimated by the sound source localization detection unit 142, it is supplied to the output control unit 153. As a result, more accurate speaker identification results can be obtained.

  The output control unit 153 causes the display unit 5 to display an image obtained from the face image data supplied from the identification information adding unit 152. Therefore, the face image of the identified speaker is displayed on the display unit 5. Note that the face image as a result of speaker identification is not only displayed on the display unit 5, but the voice of the identified speaker may be output from the speaker by the output control unit 153.

  Further, the output control unit 153 stores the audio data and face image data supplied from the identification information adding unit 152 in the minutes DB 154 in association with the speaker ID or the like.

  Here, beam forming by the information processing unit 4 having the above configuration will be described.

  Delay sum beam forming (beam forming by the delay sum method) performed in the information processing unit 4 makes audio data from each of the microphones 3-1 to 3-4 in phase based on a time delay corresponding to the distance between the microphones. Only audio data from a predetermined direction is emphasized by matching its phase.

  Therefore, according to the delay sum beamforming, voice data from other azimuth other than a predetermined azimuth and noise, for example, irrelevant to the distance between the mics are removed.

  By the way, as the beam forming width at the time of beam forming is narrower, the voice data of one speaker can be separated from the voice data of another speaker. However, if the beam forming width is narrow, the information amount of the audio data obtained by the microphone array 3 is reduced, and the audio data extracted by the beam forming becomes a muffled sound.

  On the other hand, if the beam forming width is simply widened, it becomes difficult to accurately extract the voice data of a single speaker from the voice data from the microphone array 3. That is, when the beamforming width is simply widened, when voice data of a certain speaker is extracted, voice data of a speaker adjacent to the speaker is also extracted.

  Therefore, the information processing unit 4 can extract the voice data of each speaker with high accuracy so that it can be distinguished from the voice data of other speakers. The calculation is performed based on this. As a result, an optimum beamforming width can be obtained for each speaker.

  Calculation of the beam forming width based on the positional relationship between speakers will be described with reference to FIGS. 5A and 5B.

  FIG. 5A shows an image of the entire periphery imaged by the camera module 2 in a state where four speakers X-1 to X + 2 surround the tabletop speaker identification device 1 as shown in FIG. 5B. An example of the development is shown.

  In FIG. 5A, the 360-degree peripheral image data around the camera module 2 is rectangular image data with the horizontal direction being the circumferential direction (angular direction) and the vertical direction being the radial direction. It is shown as Therefore, if the azimuth at the left end of the entire surrounding image data in FIG. 5 is, for example, 0 degrees, the azimuth at the right end is 360 degrees.

In FIG. 5A, the direction (speaker direction) where the speaker X + i is located is represented by D _{X + i} (i = −1, 0, 1, 2).

4 calculates the beamforming width B _X of the speaker X, the direction D _X of the speaker _X and the speaker X-1 adjacent to the speaker X in the counterclockwise direction. The difference (angle) | D _X-1 −D _X | of the direction D _X-1 is calculated, and the direction D of the speaker _X and the direction of the speaker X + 1 adjacent to the speaker X in the clockwise direction are calculated. D _{X + 1} of the difference between orientation | D _X + 1 -D _X | is calculated.

The beam width calculation unit 122 then determines the difference in orientation between the speakers X and X-1 | D _X-1 −D _X | and the difference in orientation between the speakers X and X + 1 | D _{X + 1} −. The smaller of D _X | is defined as the beam forming width B _X of the speaker _X (the hatched portion in FIG. 5B).

Similarly, the beam width calculation unit 122 determines the difference in orientation between the speakers X + 1 and X | D _X −D _{X + 1} | and the difference in orientation between the speakers X + 1 and X + 2 | D _{X + 2} −D. The smaller of _{X + 1} | is set as the beam forming width B _{X + 1} of the speaker X + 1.

Further, the beam width calculation unit 122 determines the difference in orientation between the speakers X + 2 and X + 1 | D _{X + 1} −D _{X + 2} | and the difference in orientation between the speakers X + 2 and X−1 | D _X−. The smaller of ₁₋ D _{X + 2} | is the beam forming width B _{X + 2 of} speaker X + 2, and the difference in orientation between speaker X−1 and speaker X + 2 | D _{X + 2} −D _X−1 | And the smaller difference between the orientations of the speakers X-1 and X | D _X -D _X-1 | is defined as the beam forming width B _X-1 of the speaker X-1.

  Note that the smaller of the heading difference between the speaker and another speaker adjacent in the counterclockwise direction and the heading difference between the speaker and another speaker adjacent in the clockwise direction is the speaker's By setting the beam forming width, the beam forming width of the speaker is set up to the center of the direction of the other speaker in the direction closest to the direction of the speaker. Therefore, for each speaker, beam forming is performed up to the center of the direction of the other speaker in the closest direction.

  As described above, the beam width calculation unit 122 determines the beamforming width of a predetermined speaker as the difference between the direction of the predetermined speaker and the direction of another speaker adjacent in the counterclockwise direction, Is the smaller of the differences between the direction of the other speaker and the direction of another speaker adjacent in the clockwise direction. As a result, the voice data of a predetermined speaker is separated from the voice data of other speakers and beam-formed with an optimum beam forming width.

In addition, for the speaker X, for example, an angle in a range from (D _X-1 + D _X ) / 2 to (D _{X + 1} + D _X ) / 2 can be set as the beam forming width. The same applies to the other speakers X-1, X + 1, and X + 2.

  Thus, the beam forming width is obtained based on the positional relationship between the speakers, and the beam forming width is used to perform beam forming suitable for each speaker. It is possible to extract speech data from which other speakers' utterances and noise of the microphone itself are removed.

  Next, the operation of the tabletop speaker identification device 1 having the above configuration will be described.

  First, with reference to the flowchart of FIG. 6, the process of the table top type speaker identification device 1 that obtains the speaker orientation and the beamforming width and registers the face image data in the speaker information DB 132 will be described.

  For example, when the table top type speaker identification device 1 is turned on and a predetermined button is operated, in step S1, the camera module 2 images the entire periphery surrounding the table top type speaker identification device 1, All-around image data is acquired and supplied to the face detection unit 121.

  In step S <b> 2, the face detection unit 121 extracts face image data corresponding to the speaker's face from the entire surrounding image data supplied from the camera module 2, and sets a speaker ID unique to each of the extracted face image data. Give.

  In step S <b> 3, for each speaker, the face detection unit 121 detects the speaker orientation, which is the orientation of the speaker position in the real world with the camera module 2 as the center, from the speaker's face image data. The detected speaker orientation is associated with the speaker ID together with the face image data, and is supplied to the beam width calculation unit 122.

  In step S <b> 4, the beam width calculation unit 122 calculates the beamforming width based on the speaker orientation associated with the speaker ID supplied from the face detection unit 121. That is, the beam width calculation unit 122 pays attention to a certain speaker ID, and the speaker orientation associated with the speaker ID of interest and the two speakers adjacent to the speaker with the speaker ID. Recognize the speaker orientation associated with the speaker's speaker ID, each of the two speakers' one or the other speaker orientation, and the speaker orientation of the speaker with the speaker ID of interest And the smaller difference is used as the beamforming width of the speaker with the speaker ID of interest. Similarly, the beam width calculation unit 122 calculates beam forming widths for all speakers (speaker IDs).

  In step S5, the beam width calculation unit 122 associates the speaker orientation associated with the speaker ID with the beam forming width of the speaker with the speaker ID, and sets each speaker ID to the speaker. The speaker orientation and beamforming width associated with the ID are supplied to the sound source localization detection unit 142.

  In step S <b> 6, the beam width calculation unit 122 supplies face image data associated with each speaker ID to the DB management unit 131. In the DB management unit 131, face image data associated with the speaker ID is registered in the speaker information DB 132, and the process ends.

  Note that the beam width calculation unit 122 determines the beam forming width of the speaker, for example, a predetermined default when the number of speakers is one, that is, when the speaker ID supplied from the face detection unit 121 is one. The beam forming width.

  As described above, the speaker orientation, the face image data, and the beamforming width are obtained based on the all-around image data acquired by the camera module 2, and the speaker orientation and beam associated with the speaker ID are obtained. The forming width is supplied to the sound source localization detection unit 142. Also, face image data associated with the speaker ID is registered in the speaker information DB 132.

  Next, processing for registering a standard pattern of speaker voice data in the speaker information DB 132 will be described with reference to the flowchart of FIG.

  This process is performed after the process described with reference to FIG. Therefore, the sound source localization detection unit 142 is supplied with the speaker orientation and the beamforming width associated with the speaker ID of each speaker from the beam width calculation unit 122.

  In step S41, the sound source localization detection unit 142 receives the speaker orientation and the beamforming width associated with the speaker ID supplied from the beam width calculation unit 122, and stores them in a built-in memory (not shown). To do.

  After storing the speaker orientation and the beamforming width in the memory, the sound source localization detection unit 142 proceeds to step S42, and determines whether or not audio data is supplied from the microphone array I / F 141.

  In step S42, when it is determined that no sound data is supplied from the microphone array I / F 141, that is, when no speaker is speaking, the sound source localization detection unit 142 is supplied with the sound data. Wait until.

  On the other hand, when it is determined in step S42 that the voice data is supplied from the microphone array I / F 141, that is, the speech is input to the microphone array 3, and the digital voice data obtained by the microphone array I / F 141 is the sound source localization. When supplied to the detection unit 142 and the beam forming processing unit 143, the process proceeds to step S43, and the sound source localization detection unit 142 is based on the audio data from each of the microphones 3-1 to 3-4 constituting the microphone array 3. Estimate the sound source direction of the utterance.

  In step S43, the sound source localization detection unit 142 is the closest to the sound source direction estimated in step S42 based on all the speaker directions supplied from the beam width calculation unit 122 in step S41 and stored in the memory. Speaker IDs associated with the speaker's orientation (the orientation obtained from the captured image) are among the speaker IDs of all the speakers stored in the memory. Select from. The selected speaker ID is supplied to the voice identification processing unit 151.

  In step S44, the sound source localization detection unit 142 supplies the speaker orientation and the beamforming width associated with the speaker ID selected in the process of step S43 to the beamforming processing unit 143. The speaker orientation and beamforming width supplied here are set by the beamforming processing unit 143 as the orientation and beamforming width for beamforming voice data.

  In step S45, the beam forming processing unit 143 performs beam forming on the audio data supplied from the microphone array I / F 141 based on the speaker orientation and the beam forming width supplied from the sound source localization detecting unit 142. Centered on the speaker orientation from the sound source localization detector 142, the orientation of the range represented by the beamforming width (from the orientation of the speaker, from the angle of +1/2 of the range (angle) represented by the beamforming width to -1/2 The voice data from the azimuth up to the angle is emphasized, and the voice data obtained by the enhancement is supplied to the voice identification processing unit 151.

  In step S46, the voice identification processing unit 151 extracts a voice feature amount from the voice data (beamformed voice data) supplied from the beamforming processing unit 143, and generates a standard pattern from the voice feature amount. Further, the voice identification processing unit 151 associates the speaker ID from the sound source localization detection unit 142 with the standard pattern and supplies the standard pattern to the DB management unit 131.

  In step S47, the DB management unit 131 sets the standard pattern associated with the speaker ID supplied from the voice identification processing unit 151 to the same speaker among the speaker IDs registered in the speaker information DB 132. Register in association with the ID, and terminate the process.

  The processes in steps S42 to S47 are performed at least once for each speaker. In the speaker information DB 132, face image data and standard patterns of each speaker are registered in association with the speaker ID. The Based on the information registered in the speaker information DB 132, the speaker identification process of FIG. 8 is performed.

  Next, speaker identification processing performed based on information registered in the speaker information DB 132 will be described with reference to the flowchart of FIG.

  In step S71, the sound source localization detection unit 142 determines whether audio data is supplied from the microphone array I / F 141 and waits until the audio data is supplied.

  If the sound source localization detection unit 142 determines in step S71 that audio data has been supplied from the microphone array I / F 141, the sound source localization detection unit 142 proceeds to step S72 and performs audio from each of the microphones 3-1 to 3-4 constituting the microphone array 3. Estimate the sound source direction of the utterance based on the data.

  In step S72, the sound source localization detection unit 142 detects a speaker having a direction closest to the estimated direction from the speaker directions supplied from the beam width calculation unit 122 in the process of step S41 in FIG. The speaker ID associated with the direction of the speaker is supplied to the identification information adding unit 152.

  In step S73, the sound source localization detection unit 142 supplies the speaker orientation and beamforming width associated with the speaker ID supplied to the identification information providing unit 152 in step S72 to the beamforming processing unit 143. To do. The beam forming processing unit 143 sets the speaker orientation and the beam forming width supplied from the sound source localization detecting unit 142 as the beam forming orientation and the beam forming width.

  In step S74, the beamforming processing unit 143 performs beamforming on the audio data supplied from the microphone array I / F 141 based on the speaker orientation and the beamforming width supplied from the sound source localization detection unit 142, thereby Centered on the speaker orientation from the sound source localization detector 142, the orientation of the range represented by the beamforming width (from the orientation of the speaker, from the angle of +1/2 of the range (angle) represented by the beamforming width to -1/2 The voice data from the azimuth up to the angle is emphasized and supplied to the voice identification processing unit 151.

  In step S <b> 75, the voice identification processing unit 151 extracts a voice feature amount from the beamformed voice data supplied from the beamforming processing unit 143.

  In step S76, the voice identification processing unit 151 refers to the standard pattern of the speaker registered in the speaker information DB 132 obtained through the DB management unit 131, and identifies the speaker identification of the speaker who made the utterance. Do.

  That is, the speech identification processing unit 151 associates with a standard pattern registered in the speaker information DB 132 with a standard pattern having a high likelihood that the speech feature amount of speech data from the beamforming processing unit 143 is observed. Identify the speaker ID being used. Further, the voice identification processing unit 151 supplies the speaker ID and the voice data from the beamforming processing unit 143 to the identification information adding unit 152.

  In step S77, the identification information adding unit 152 determines the speaker ID from the sound source localization detection unit 142 (the speaker associated with the speaker direction closest to the estimated direction estimated by the sound source localization detection unit 142 in step S72). ID) and a speaker ID obtained as a result of speaker identification supplied from the voice identification processing unit 151 in step S76 are determined.

  When the identification information adding unit 152 determines in step S77 that the speaker ID from the sound source localization detection unit 142 and the speaker ID obtained as a result of speaker identification supplied from the voice identification processing unit 151 do not match. In step S78, predetermined error processing is performed.

  As the error processing performed in step S78, for example, all-around image data can be acquired by the camera module 2, and the speaker orientation and the beamforming width can be recalculated. In this case, when the speaker moves and the speaker orientation and the optimum beamforming width change, the speaker orientation and beamforming width can be changed in real time.

  On the other hand, if it is determined in step S77 that the speaker ID from the sound source localization detection unit 142 and the speaker ID obtained as a result of speaker identification supplied from the voice identification processing unit 151 match, the process proceeds to step S79. The identification information adding unit 152 sets the matching speaker ID as the final speaker identification result, and the face image data associated with the speaker ID via the DB management unit 131. Get from. Thereafter, the identification information adding unit 152 supplies the face image data to the output control unit 153 together with the audio data supplied from the audio identification processing unit 151, and ends the speaker identification process.

  In the output control unit 153, a face image is displayed and a sound is output.

  As described above, in the table top type speaker identification device 1, the speaker ID of the speaker in the direction closest to the estimated direction of the sound source and the speech obtained as a result of speaker identification by the speech identification processing unit 151. Only when the speaker ID matches, the speaker ID is used as the final speaker identification result, so that the result is more accurate than when such a determination is not made.

  In addition, since the voice data for generating the standard pattern and the voice data for identifying the speaker are acquired in the same environment, the accuracy of speaker identification in the voice identification processing unit 151 can be improved. it can.

  In step S72 of FIG. 8, the sound source localization detection unit 142 estimates the sound source directions of a plurality of utterances based on the sound data from the microphones 3-1 to 3-4, that is, a plurality of speakers. Are simultaneously spoken, and a plurality of sound source azimuths are thereby estimated, the beamforming processing unit 143 in the subsequent stage associates with the speaker IDs of the plurality of speakers supplied from the sound source localization detection unit 142. Beam forming is performed using a certain speaker direction and beam forming width, and speaker identification is performed by the voice identification processing unit 151 based on each voice data, so that a plurality of speakers speak at the same time. Even in the case, each speaker can be identified.

  Further, since the face is detected from the all-around image data, for example, the task of registering the speaker face image data displayed as the speaker identification result can be omitted. Although the camera module 2 obtains all-around image data, it is also possible to obtain image data of a part of the entire circumference.

  Furthermore, the present invention can be applied not only to the table top type speaker identification device 1 but also to, for example, audiovisual technology of a robot. When the present invention is applied to the audiovisual technology of a robot, each speaker can be identified even when a plurality of speakers speak at the same time within the vision of the robot.

  In the present embodiment, the speech identification processing unit 151 extracts speech feature values by discrete Fourier transform. However, the speech feature value extraction method is not limited to discrete Fourier transform.

  Furthermore, speaker identification in the voice identification processing unit 151 can be performed by means other than the HMM method.

  When determining whether or not the sound data is supplied to the sound source localization detection unit 142 in the process of step S42 of FIG. 7 and the process of step S71 of FIG. 8, the speaker is determined based on the sound signal output from the microphone array 3. It is possible to detect an utterance section that is a section in which utterance is made, and determine that voice data has been supplied when the utterance section is detected.

  Note that the program for executing the above-described series of processing performed by the CPU 51 of FIG. 3 can be downloaded from a download site and installed. The program can be installed from a recording medium.

  The recording medium on which the program is recorded can be distributed as a package medium including the magnetic disk 61, the optical disk 62, the magneto-optical disk 63, or the semiconductor memory 64.

  Here, in the present specification, the processing of each step described in the flow chart described is not limited to processing performed in time series according to the described order, but may be performed in parallel or individually even if not necessarily performed in time series. This includes the processing to be executed.

  The series of processes described above is executed by software, but can also be executed by dedicated hardware.

It is a figure which shows the usage example of one Embodiment of the tabletop type speaker identification device to which this invention is applied. It is a perspective view which expands and shows the tabletop type speaker identification apparatus of FIG. It is a block diagram which shows the hardware structural example of a tabletop type speaker identification apparatus. It is a block diagram which shows the functional structural example of the information processing part of FIG. It is a figure explaining beam forming. It is a flowchart explaining the process which calculates | requires a speaker orientation and a beam forming width | variety. It is a flowchart explaining the process which registers the standard pattern of audio | voice data into speaker information DB. It is a flowchart explaining a speaker identification process.

Explanation of symbols

  1 table top type speaker identification device, 2 camera module, 3 microphone array, 4 information processing section, 5 display section, 51 CPU, 52 ROM, 53 RAM, 54 bus, 55 input / output interface, 56 input section, 57 output section , 58 storage unit, 59 communication unit, 60 drive, 61 magnetic disk, 62 optical disk, 63 magneto-optical disk, 64 semiconductor memory, 121 face detection unit, 122 beam width calculation unit, 131 DB management unit, 132 speaker information DB, 141 microphone array I / F, 142 sound source localization detection unit, 143 beamforming processing unit, 151 voice identification processing unit, 152 identification information adding unit, 153 output control unit, 154 minutes DB

Claims (5)

Imaging means for imaging one or more speakers and outputting the images;
Extraction range calculation means for calculating an extraction range that is a range of orientations for extracting voice data of each speaker based on the positional relationship of the speakers in the image output by the imaging means;
Voice data extraction means for extracting voice data within the extraction range calculated by the extraction range calculation means, out of voice data of speech by the speaker ;
Storage means for storing a standard pattern for speaker identification based on the voice data extracted by the voice data extraction means for each speaker;
Speaker identification means for performing speaker identification using voice data newly extracted by the voice data extraction means and a standard pattern stored in the storage means;
Based on the voice data newly extracted by the voice data extraction means, the sound source direction estimation means for estimating the sound source direction;
Determining means for determining whether or not a speaker existing in the sound source direction estimated by the sound source direction estimating means matches a speaker identified by speaker identification by the speaker identifying means ; A characteristic information processing apparatus. Further comprising face detection means for detecting the face of the speaker from the image output by the imaging means;
The information processing apparatus according to claim 1, wherein the extraction range calculation unit calculates the extraction range based on a positional relationship of faces detected by the face detection unit. An imaging step of imaging one or more speakers and outputting the images;
An extraction range calculating step for calculating an extraction range that is a range of orientations for extracting voice data of each speaker based on the positional relationship of the speakers in the image output by the processing of the imaging step;
A voice data extraction step of extracting voice data within the extraction range calculated by the processing of the extraction range calculation step of the voice data of the utterance by the speaker ;
A storage step of storing a standard pattern for speaker identification based on the voice data extracted by the voice data extraction step for each speaker,
A speaker identification step of performing speaker identification using the voice data newly extracted by the processing of the voice data extraction step and the standard pattern stored by the processing of the storage step;
A sound source direction estimation step for estimating a sound source direction based on the voice data newly extracted by the processing of the voice data extraction step;
A determination step for determining whether or not a speaker existing in the sound source direction estimated by the processing of the sound source direction estimation step matches a speaker identified by speaker identification by the processing of the speaker identification step; An information processing method comprising: A computer executable program,
An imaging step of imaging one or more speakers and outputting the images;
An extraction range calculating step for calculating an extraction range that is a range of orientations for extracting voice data of each speaker based on the positional relationship of the speakers in the image output by the processing of the imaging step;
A voice data extraction step of extracting voice data within the extraction range calculated by the processing of the extraction range calculation step of the voice data of the utterance by the speaker ;
A storage control step for controlling storage for each speaker, a standard pattern for speaker identification based on the voice data extracted by the processing of the voice data extraction step;
A speaker identification step of performing speaker identification using the voice data newly extracted by the processing of the voice data extraction step and the standard pattern stored by the processing of the storage control step;
A sound source direction estimation step for estimating a sound source direction based on the voice data newly extracted by the processing of the voice data extraction step;
A determination step for determining whether or not a speaker existing in the sound source direction estimated by the processing of the sound source direction estimation step matches a speaker identified by speaker identification by the processing of the speaker identification step; The program characterized by including. A recording medium on which a computer executable program is recorded,
An imaging step of imaging one or more speakers and outputting the images;
An extraction range calculating step for calculating an extraction range that is a range of orientations for extracting voice data of each speaker based on the positional relationship of the speakers in the image output by the processing of the imaging step;
A voice data extraction step of extracting voice data within the extraction range calculated by the processing of the extraction range calculation step of the voice data of the utterance by the speaker ;
A storage control step for controlling storage for each speaker, a standard pattern for speaker identification based on the voice data extracted by the processing of the voice data extraction step;
A speaker identification step of performing speaker identification using the voice data newly extracted by the processing of the voice data extraction step and the standard pattern stored by the processing of the storage control step;
A sound source direction estimation step for estimating a sound source direction based on the voice data newly extracted by the processing of the voice data extraction step;
A determination step for determining whether or not a speaker existing in the sound source direction estimated by the processing of the sound source direction estimation step matches a speaker identified by speaker identification by the processing of the speaker identification step; A recording medium on which a program is recorded.

Images