JP2010154260A - Voice recognition device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain an excellent sound even for a sound source which intermittently generates the sound by attenuating noise. <P>SOLUTION: The voice recognition device includes: an image/sound feature information storage unit 31 which stores image/sound feature information; a body detection unit 24a which detects feature information of a subject image; sound detection units 14a and 14b which detect feature information of the sound; a body position detection unit 24b which calculates the distance and the direction to a subject; a sound position detection unit 12 which calculates the distance and the direction to the sound source; a relating unit 40a which relates the subject and the sound source as the same body on the basis of the feature information of the subject image, the feature information of the sound, the distance and the direction to the subject, and the distance and the direction to the sound source; a feature information determination unit 40b which determines whether the feature information of the body matches the image/sound feature information; a tracking control unit 40d which tracks the subject image; and directional characteristic adjustment units 13a and 13b which adjust directional characteristics of a microphone array 11 on the basis of a tracking result and the distance and the direction to the subject or the sound source. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a device for identifying sound emitted from an object, and more particularly to a sound identification device capable of obtaining good sound by attenuating noise even for a sound source that intermittently generates sound.

  In general, a video camera or the like may shoot a large number of people. In such a case, the direction of the speaker that is producing the voice is detected, and the directivity of the microphone is increased with respect to the detected direction. Noise needs to be attenuated.

  Therefore, in Patent Document 1, a direction detection unit that detects the direction of the speaker, a microphone that detects the voice of the speaker, and a directivity characteristic of the microphone are enhanced in the direction of the speaker detected by the direction detection unit. A vehicle-mounted speech recognition device that includes a gain adjustment unit that adjusts and a speech recognition unit that recognizes the speech of a speaker adjusted by the gain adjustment unit has been proposed.

  Further, Patent Document 2 specifies the lip position of a pilot from an image captured by a camera, adjusts the directivity characteristics of a plurality of microphones based on the specified lip position, and outputs audio signals of the plurality of microphones. Has been proposed.

Furthermore, in Patent Document 3, the direction of the speaker is detected based on the sound input from the microphone to match the directivity characteristics of the microphone, and the camera is directed toward the detected speaker and is taken by this camera. There has been proposed an audiovisual cooperative recognition apparatus that performs face detection based on an image and performs dialogue processing when the face is detected.
JP 11-219193 A JP 2000-10589 A JP 2006-251266 A

  However, in the technique of Patent Document 1, since the microphone directivity is adjusted based on the voice uttered by the speaker, it is possible to adjust the microphone directivity while the speaker is not generating speech. It was difficult.

  In the technique described in Patent Document 2, the directional characteristics of the microphone are adjusted based on the position of the pilot's lips identified from the photographed image, so that the pilot's lips are not moving. While it did not occur, it was difficult to adjust the directivity of the microphone.

  Furthermore, in the technique described in Patent Document 3, the direction of the speaker is detected based on the sound input from the microphone. Therefore, for example, when a large number of people who speak at random are photographed, the sound is input from the microphone. In addition, since it is necessary to match the directivity characteristics of the microphone, the load on the apparatus is large, and it is difficult to detect the sound immediately after uttering with high sensitivity.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a speech identification device that can obtain good speech by attenuating noise even for a sound source that intermittently generates speech.

  In order to achieve the above object, a first feature of a voice identification device according to the present invention is that a voice identification device for identifying a voice emitted from an object converts light collected by an optical system into an electrical signal to generate an image. An imaging unit that generates data, a microphone array in which a plurality of microphones that generate sound data by converting sound emitted from a sound source into electrical signals, and features of a subject image included in the image data An image sound feature information storage unit that associates information with sound feature information emitted from the sound source and stores it as image sound feature information, and detects object image feature information from the image data generated by the imaging unit An object detection unit, a voice detection unit that detects voice feature information from the voice data generated by the microphone array, and the imaging unit. Based on the audio data generated by the microphone array, an object position detection unit that calculates the distance from the voice identification device to the subject and the direction of the subject relative to the voice identification device based on the image data formed An audio position detection unit that calculates a distance from the sound identification device to the sound source and a direction of the sound source with respect to the sound identification device, feature information of the subject image detected by the object detection unit, and the sound detection unit. Based on the detected audio feature information, the distance and direction of the subject calculated by the object position detection unit, and the distance and direction of the sound source calculated by the audio position detection unit, the subject and the The associating unit associating the sound source with the same object and the feature information of the object associated by the associating unit A feature information determination unit that determines whether or not the image and sound feature information stored in the image and sound feature information storage unit matches, and a feature information that is determined by the feature information determination unit to match the image and sound feature information A tracking control unit that tracks the subject on the image data, a tracking result of the tracking control unit, a distance and direction of the subject calculated by the object position detection unit, or a sound position detection unit. And a directional characteristic adjusting unit that adjusts the directional characteristic of the microphone array based on the distance and direction of the sound source.

  In order to achieve the above object, the second feature of the voice identification device according to the present invention is that the feature information of the object associated by the association unit is obtained when the feature information judgment unit determines that the feature information does not match. The present invention further includes a storage control unit that stores the new image / audio feature information in the image / audio feature information storage unit.

  In order to achieve the above object, a third feature of the speech recognition device according to the present invention is that the object position detection unit is configured to detect the object position from the speech recognition device based on the angle of view in the imaging unit and the focus information to the subject. It is to calculate the distance to the subject and the direction of the subject with respect to the voice identification device.

  In order to achieve the above object, a fourth feature of the voice identification device according to the present invention is an image reference feature information storage unit that stores feature information of the subject image and a reference dimension of the subject in association with each other as image reference feature information. The object position detection unit extracts a reference dimension of the subject corresponding to the feature information of the subject image included in the image data based on the image reference feature information, and extracts the reference of the subject The object is to calculate the distance from the voice identification device to the subject and the direction of the subject relative to the voice identification device based on the dimensions and the angle of view in the imaging unit.

  In order to achieve the above object, according to a fifth feature of the voice identification device according to the present invention, the voice position detection unit is configured such that the voice position detection unit to the sound source based on a time difference between voices reaching the plurality of microphones. And the direction of the sound source with respect to the voice identification device.

  In order to achieve the above object, a sixth feature of the voice identification device according to the present invention is that, when the tracking control unit determines that the feature information of the subject image matches by the feature information determination unit, the image data The image displayed on the basis of the above is divided into a plurality of blocks, and the movement of the subject is tracked by detecting the movement of each block.

  In order to achieve the above object, according to a seventh feature of the voice identification device according to the present invention, the directivity adjustment unit is configured so that the tracking result of the tracking control unit and the distance of the subject calculated by the object position detection unit are And the audio data generated by the plurality of microphones is overlapped so as to eliminate the time difference between the voices reaching the plurality of microphones based on the direction and direction or the distance and direction of the sound source calculated by the sound position detection unit. There is to match.

  According to the voice identification device of the present invention, it is possible to obtain a good voice by attenuating noise even for a sound source that intermittently generates voice.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  In an embodiment of the present invention, a voice recognition device that obtains a good voice by attenuating noise for a sound source that intermittently generates voice will be described as an example.

<Configuration of voice identification device>
FIG. 1 is a configuration diagram showing a configuration of a voice identification device according to an embodiment of the present invention.

  A voice identification device 1 according to an embodiment of the present invention includes a microphone array 11, a voice position detection unit 12, a first directivity characteristic adjustment unit 13a, a second directivity characteristic adjustment unit 13b, and a first voice. Detection unit 14a, second audio detection unit 14b, camera 21 and camera processing unit 22 having an imaging unit, motion sensor 23, detection unit 24, motion vector detection unit 25, and image / audio feature information storage unit 31, an image reference feature information storage unit 32, an audio reference feature information storage unit 33, a directivity characteristic priority storage unit 34, a CPU 40, an operation unit 41, and a display unit 42.

  The microphone array 11 includes a first microphone 11a, a second microphone 11b, and a third microphone 11c, each of which is arranged at a predetermined interval of, for example, about 10 mm. The sound emitted from the sound source is an electrical signal. To generate audio data.

  The sound position detection unit 12 calculates the distance from the sound identification device 1 to the sound source and the direction of the sound source with respect to the sound identification device 1 based on the sound data generated by the microphone array 11.

  The first directivity adjustment unit 13a is a tracking result of a tracking control unit 40d of a CPU 40 described later, and a distance and direction of a subject calculated by an object position detection unit 24b of a detection unit 24 described later, or an audio position detection unit 12. Is generated by the respective microphones so as to eliminate the time difference between the voices that have reached the first microphone 11a, the second microphone 11b, and the third microphone 11c, based on the distance and direction of the sound source calculated by. The directivity is adjusted by superimposing the recorded audio data.

  The second directivity characteristic adjustment unit 13b has the same configuration as the first directivity characteristic adjustment unit 13a.

  The first voice detection unit 14a extracts voice feature information from the voice data whose directivity characteristics are adjusted by the first directivity characteristic adjustment unit 13a. Specifically, the first sound detection unit 14a extracts component waveforms, formants, and the like from the sound whose directivity characteristics have been adjusted, and supplies these to the CPU 40 as sound feature information.

  The second voice detection unit 14b has the same configuration as the first voice detection unit 14a.

  The camera 21 converts the light collected by the built-in lens into an electrical signal.

  The camera processing unit 22 converts the electrical signal supplied from the camera 21 into image data such as an RGB signal luminance signal Y and color difference signals Cr and Cb signals.

  The motion sensor 23 includes a gyro sensor, for example, and detects the motion of the voice identification device 1.

  The detection unit 24 includes an object detection unit 24a and an object position detection unit 24b.

  The object detection unit 24 a detects the feature information of the subject image from the image data generated by the camera processing unit 22. For example, the object detection unit 24 a detects the shape and color of the subject image from the image data as feature information, and based on the extracted shape and color and the image reference feature information stored in the image reference feature information storage unit 32. Identify the type of subject. The specified subject type and subject image feature information are supplied to the CPU 40.

  The object position detection unit 24 b calculates the distance from the voice identification device 1 to the subject of the image data and the direction of the subject with respect to the voice identification device 1 based on the image data generated by the camera processing unit 22.

  The motion vector detection unit 25 detects the motion of the image data generated by the camera 21.

  The image / sound feature information storage unit 31 stores the feature information of the subject image of the image data and the feature information of the sound emitted from the sound source in association with each other and stores them as image / sound feature information.

  FIG. 2 is a diagram showing an example of image / audio feature information stored in the image / audio feature information storage unit 31 included in the audio identification device 1 according to the embodiment of the present invention.

  As shown in FIG. 2, the column name “feature information ID” (reference numeral 51), the column name “type” (reference numeral 52), the column name “feature information characteristic information” (reference numeral 53), and the column name “voice”. Data feature information "(reference numeral 54) is stored in association with image / audio feature information.

  The subject image feature information 53 includes a column name “shape” (reference numeral 53a) and a column name “color” (reference numeral 53b). The audio data feature information 54 includes a column name “component waveform” (symbol 54a) and a column name “formant” (symbol 54b).

  The image reference feature information storage unit 32 stores the type of subject and the image reference feature information in association with each other.

  FIG. 3 is a diagram illustrating an example of image reference feature information stored in the image reference feature information storage unit 32 included in the voice identification device 1 according to the embodiment of the present invention.

  As shown in FIG. 3, the column name “type” (reference numeral 61) and the column name “image reference feature information” (reference numeral 62) are stored in association with each other. The image reference feature information 62 includes a column name “shape” (reference numeral 62a), a column name “color” (reference numeral 62b), and a column name “reference dimension” (reference numeral 62c).

  The sound reference feature information storage unit 33 stores the sound source type and the sound reference feature information in association with each other.

  FIG. 4 is a diagram illustrating an example of the speech reference feature information stored in the speech reference feature information storage unit 33 included in the speech identification device 1 according to an embodiment of the present invention.

  As shown in FIG. 4, the column name “type” (reference numeral 71) and the column name “voice reference feature information” (reference numeral 72) are stored in association with each other. The audio reference feature information 72 includes a column name “power spectrum” (reference numeral 72a) and a column name “sound spectrum” (reference numeral 72b).

  The directivity characteristic priority storage unit 34 stores the priority order of the types of subjects and sound sources supplied from the operation unit 41 described later. Note that the CPU 40 described later performs processing in accordance with a predetermined priority order stored in advance in the directivity characteristic priority storage unit 34 until a priority order in the operation unit 41 is designated.

  The CPU 40 performs central control of the voice identification device 1. Moreover, CPU40 is provided with the correlation part 40a, the characteristic information determination part 40b, the memory | storage control part 40c, the tracking control part 40d, and the orientation adjustment control part 40e on the function.

  The associating unit 40a includes the feature information of the subject image detected by the object detecting unit 24a, the feature information of the sound detected by the first sound detecting unit 14a or the second sound detecting unit 14b, and the object position detecting unit 24b. The subject and the sound source are associated as the same object based on the distance and direction of the subject calculated by the above and the distance and direction of the sound source calculated by the audio position detection unit 12.

  The feature information determination unit 40 b determines whether or not the feature information of the object associated by the association unit 40 a matches the image and sound feature information stored in the image and sound feature information storage unit 31.

  When the feature information determining unit 40b determines that the feature information does not match, the storage control unit 40c uses the feature information of the object associated by the associating unit 40a as new image / sound feature information, and the image / sound feature information storage unit 31. Remember me.

  The tracking control unit 40d divides the image displayed on the display unit 42 into a plurality of blocks on the basis of the image data when the feature information determination unit 40b determines that the feature information of the subject image matches. The movement of the subject is tracked by detecting the movement of the subject.

  The directivity adjustment control unit 40e is based on the tracking result of the tracking control unit 40d and the distance and direction of the subject calculated by the object position detection unit 24b or the distance and direction of the sound source calculated by the audio position detection unit 12. The first directivity adjustment unit 13a or the second directivity adjustment unit 13b is caused to adjust the directivity.

  The operation unit 41 performs various operations such as an operation signal for requesting the start and end of photographing and an operation signal for requesting storage of the priority order of the types of subjects in the directivity characteristic priority storage unit 34 based on a user operation. A signal is generated and the generated operation signal is supplied to the CPU 40.

  The display unit 42 includes an image output device such as an organic EL (electroluminescence) display or a liquid crystal display, and displays various screens based on image data supplied from the CPU 40.

<Operation of the voice identification device 1>
Next, the operation of the voice identification device 1 according to an embodiment of the present invention will be described.

  FIG. 5 is a flowchart showing a processing flow of the voice identification device 1 according to the embodiment of the present invention.

  First, when an electrical signal is supplied from the camera 21 (step S101), the camera processing unit 22 of the voice identification device 1 converts the supplied electrical signal into an RGB signal, a luminance signal Y, and color difference signals Cr and Cb signals. Convert to generate image data.

  Next, the object position detection unit 24b corrects the shake based on the movement of the voice identification device 1 detected by the movement sensor 23 (step S102). For example, the object position detection unit 24b selects a range of image data to be cut out from the image data supplied from the camera processing unit 22 so as to cancel the movement of the voice identification device 1 detected by the motion sensor 23. The obtained image data is supplied to the object detection unit 24a.

  Then, the object detection unit 24a detects the feature information of the subject image from the image data whose shake has been corrected (step S103). For example, the object detection unit 24 a detects the shape and color of the subject image from the image data as feature information, and based on the extracted shape and color and the image reference feature information stored in the image reference feature information storage unit 32. Identify the type of subject. The specified subject type and subject image feature information are supplied to the CPU 40.

  FIG. 6 is a diagram illustrating a detection process performed by the object detection unit 24a included in the voice identification device 1 according to the embodiment of the present invention.

  As shown in FIG. 6, the subject A and subject B, which are men, are shown on the screen imaged by the camera 21, and the object detection unit 24a detects the shape and color of the subject A and subject B as feature information. To do. Then, the object detection unit 24a extracts the type of the subject that matches the detected shape and color from the image reference feature information stored in the image reference feature information storage unit 32, and extracts the extracted subject A and subject B. And the feature information of the subject image are supplied to the CPU 40. In the example illustrated in FIG. 6, the object detection unit 24a extracts “male” as the type of the subject A and the subject B, and extracts the “male” that is the type of the extracted subject and the feature information of each subject image. It supplies to CPU40.

  Next, the object position detection unit 24b calculates the distance from the voice identification device 1 to the subject and the direction of the subject with respect to the voice identification device 1 based on the image data whose shake has been corrected (step S104). For example, the object position detection unit 24b calculates the distance from the voice identification device 1 to the subject of the image data and the direction of the subject with respect to the voice identification device 1 based on the angle of view in the camera 21 and the focus information to the subject.

  FIG. 7 is a diagram for explaining subject direction calculation processing by the object position detection unit 24b included in the voice identification device 1 according to the embodiment of the present invention.

  As shown in FIG. 7, the subject A and the subject B shown in FIG. 6 are shown on the screen imaged by the camera 21. Assuming that the angle of view of the camera 21 is ± Φ, the object position detection unit 24b has the subject A detected by the object detection unit 24a in the + θ3 direction on the xy plane when the voice identification device 1 is viewed from above. That is, it is determined that the subject A exists on the straight line 201 in the + θ3 direction.

  Then, the object position detection unit 24b calculates the distance from the voice identification device 1 to the subject based on the image data whose shake has been corrected.

  FIG. 8 is a diagram for explaining subject distance calculation processing by the object position detection unit 24b included in the voice identification device 1 according to the embodiment of the present invention.

  When the subject A or B is within the focus range of the camera 21, the object position detection unit 24b calculates the distance from the focus focus information.

  As shown in FIG. 8, when the subject A is within the focus range, the object position detection unit 24b calculates the distance d1 between the camera 21 and the subject A from the focus focus information.

  When the subject A or B is outside the focus range of the camera 21, the object position detection unit 24 b uses the image reference feature information stored in the image reference feature information storage unit 32 to detect the subject image of the image data. A reference dimension of the subject corresponding to the feature information is extracted, and a distance from the camera 21 to the subject of the image data is calculated based on the extracted reference dimension of the subject and the angle of view of the camera 21.

  Specifically, when the subject B shown in FIG. 8 is out of the focus range, the object position detection unit 24b determines the subject specified in step S103 from the image reference feature information stored in the image reference feature information storage unit 32. The reference dimension L2 corresponding to the type of is extracted.

  Then, the object position detection unit 24b uses the following formula 1 to calculate the object B's height when the screen height shown in FIG. 6 is Hc, the length H2 of the face of the subject B is vertical, and the field angle is θc. The angle θ2 is calculated.

θ2 = θc × H2 / Hc (Formula 1)
Next, the object position detection unit 24b calculates the distance d2 using the following formula 2 from the extracted reference dimension L2 and the calculated angle θ2.

d2 = L2 / tan θ2 (Formula 2)
Accordingly, the object position detection unit 24b can calculate the distance from the voice identification device 1 to the subject and the direction of the subject with respect to the voice identification device 1 based on the image data whose shake is corrected.

  Next, when the voice data is supplied from the first microphone 11a, the second microphone 11b, and the third microphone 11c (step S105), the voice position detection unit 12 recognizes the voice detected by the motion sensor 23. The shake is corrected based on the movement of the device 1 (step S106).

  Next, the first sound detection unit 14a or the second sound detection unit 14b is the feature information of the sound in which the shake supplied from the first directivity characteristic adjustment unit 13a or the second directivity characteristic adjustment unit 13b is corrected. Is detected (step S107). For example, the first sound detection unit 14 a extracts component waveforms, formants, and the like as sound feature information from the sound data whose shake has been corrected, and the extracted component waveforms, formants, and sound reference feature information storage unit 33. Based on the stored sound reference feature information, the types of sound sources are ranked. The ranked sound source types and audio feature information are supplied to the CPU 40.

  FIG. 9 is a diagram illustrating a detection process performed by the first voice detection unit 14a or the second voice detection unit 14b included in the voice identification device 1 according to an embodiment of the present invention. (A) shows an example of a waveform of corrected audio data, (b) shows a power spectrum generated based on (a), and (c) shows audio reference feature information storage. An example of the power spectrum of the speech reference feature information stored in the unit 33 is shown.

  In addition, since the 1st audio | voice detection part 14a or the 2nd audio | voice detection part 14b has the same structure, the 1st audio | voice detection part 14a is demonstrated.

  As shown in FIG. 9, the first voice detection unit 14a generates the power spectrum 302 shown in FIG. 9B from the voice waveform 301 shown in FIG. 9A.

  Then, the first sound detection unit 14a determines the degree of coincidence between the generated power spectrum 302 and the power spectrum 303 of the sound reference feature information stored in the sound reference feature information storage unit 33 illustrated in FIG. And ranking based on the calculated degree of coincidence.

  Specifically, the first sound detection unit 14a calculates values for each frequency component (A1 to A7) of the power spectrum illustrated in FIG. 9B, and the power spectrum illustrated in FIG. 9C. The value for each frequency component (A1 to A7) is calculated, and the absolute value of the difference between the calculated values is calculated for each frequency component (A1 to A7).

  The sum of the absolute values of the differences calculated for each frequency component (A1 to A7) is the smaller the generated power spectrum 302 and the power spectrum 303 of the speech reference feature information stored in the speech reference feature information storage unit 33. Since the degree of coincidence is high, the first sound detection unit 14a performs prioritization by rearranging the types of sound sources in ascending order of the sum of the absolute values of the differences calculated for each frequency component (A1 to A7). .

  For example, the first sound detection unit 14a calculates evaluation points that increase as the sum of absolute values of differences calculated for each frequency component (A1 to A7) decreases, and selects the type of sound source in descending order of the evaluation points. Rearrange.

  As a result, the type of the detected sound source and the evaluation point are set as “male” (evaluation point 90), “female” (evaluation point 70), “dog” (evaluation point 50), “car” (evaluation point 20). In this way, the types of sound sources are rearranged in descending order of evaluation score.

  Note that the first sound detection unit 14a may prioritize based on the sound spectrogram instead of ranking based on the power spectrum.

  Also in this case, similarly, the first sound detection unit 14a matches the degree of coincidence between the sound spectrogram generated based on the sound data and the sound spectrogram of the sound reference feature information stored in the sound reference feature information storage unit 33. And ranking is performed based on the calculated degree of coincidence.

  As shown in FIG. 5, next, the voice position detection unit 12 calculates the distance from the voice identification device 1 to the sound source and the direction of the sound source with respect to the voice identification device 1 based on the corrected voice data (step S108). ).

  FIG. 10 is a diagram for explaining calculation processing of the direction and distance of the sound source by the voice position detection unit 12 included in the voice identification device 1 according to the embodiment of the present invention.

  As shown in FIG. 10, since the first microphone 11a, the second microphone 11b, and the third microphone 11c are arranged at a predetermined distance from each other, the sound uttered by the sound source A is Delay time to input is different.

  Specifically, as shown in FIG. 10, when the time from when the sound is emitted from the sound source A to the arrival at the first microphone 11a is t0, the second microphone is obtained after the sound is emitted from the sound source A. The time until reaching 11b is (t0 + t1), and the time from when the sound source A emits sound until it reaches the third microphone 11c is (t0 + t2).

  Therefore, the voice position detection unit 12 compares the phases of the voices input to the first microphone 11a, the second microphone 11b, and the third microphone 11c, thereby delaying the delay time t1 of the voice input to the microphone. , T2 is calculated, and the distance from the voice identification device 1 to the sound source and the direction of the sound source relative to the voice identification device 1 are calculated based on the calculated delay times t1 and t2.

  FIG. 11 shows an example of phase comparison of speech waveforms input to the first microphone 11a, the second microphone 11b, and the third microphone 11c included in the speech identification device 1 according to an embodiment of the present invention. FIG.

  As shown in FIG. 11, since the sound that has been emitted from the sound source A and reached the first microphone 11a has a peak at time T10, the sound position detection unit 12 determines T10 that is the peak time. The standard. Then, the voice position detection unit 12 sets the time from T10 to time T11 when the similar peak waveform arrives in the voice waveform that has reached the second microphone 11b as the delay time t1. The voice position detection unit 12 sets a delay time t2 from T10 to a time T12 when a similar peak waveform arrives in the voice waveform that has reached the third microphone 11c.

  Then, the voice position detection unit 12 calculates the distance from the voice identification device 1 to the sound source and the direction of the sound source with respect to the voice identification device 1 based on the calculated delay times t1 and t2. Specifically, when the sound speed is v, the audio position detector 12 determines that the distance from the sound source A to the first microphone 11a is v · t0, and the distance from the sound source A to the second microphone 11b is v · (t0 + t1). ), The distance from the sound source A to the third microphone 11c is v · (t0 + t2). The voice position detection unit 12 is separated from the first microphone 11a, the second microphone 11b, and the third microphone 11c by v · t0, v · (t0 + t1), and v · (t0 + t2), That is, with the first microphone 11a, the second microphone 11b, and the third microphone 11c as the centers, the radii from the centers are v · t0, v · (t0 + t1), and v · (t0 + t2), respectively. A point where the sound source A is located is defined as a point where the points overlap each other when drawn.

  Thereby, the audio | voice position detection part 12 can calculate the distance from the audio | voice identification apparatus 1 to a sound source, and the direction of the sound source with respect to the audio | voice identification apparatus 1 based on the corrected audio | voice data.

  For example, when the sound source A and the sound source B simultaneously emit sound, the sound position detection unit 12 uses, for example, the technology described in Japanese Patent Application Laid-Open No. 2006-227328 and the distance from the sound identification device 1 to the sound source. The direction of the sound source with respect to the voice identification device 1 is calculated. Specifically, the audio position detection unit 12 determines whether a band division signal obtained by band division is a signal in which a plurality of sound sources overlap or a signal composed of only one sound source, The sound source direction is calculated using only non-overlapping frequency components.

  Next, the associating unit 40a of the CPU 40 calculates the distance from the voice identification device 1 to the subject calculated in step S104, the direction of the subject with respect to the voice identification device 1, and the distance from the voice identification device 1 to the sound source calculated in step S108. Whether the sound source and the subject can be associated based on the distance and the direction of the sound source with respect to the voice identification device 1, the type of the subject specified in step S103, and the ranking of the type of the sound source determined in step S109. Is determined (step S109).

  For example, the associating unit 40a includes the predetermined peripheral range of the position specified by the distance from the voice identification device 1 to the subject calculated in step S104 and the direction of the subject with respect to the voice identification device 1, and the voice calculated in step S108. There is an overlapping portion in the predetermined peripheral range of the position specified by the distance from the identification device 1 to the sound source and the direction of the sound source with respect to the voice identification device 1, and the type of the subject specified in step S103 is determined in step S109. When the determined evaluation score is included in the type of the sound source having 80 points or more, it is determined that the subject and the sound source can be associated as the same object.

  When it is determined in step S109 that the sound source and the subject can be associated with each other, the associating unit 40a detects the feature information of the subject image detected in step S103 and the information from the voice identification device 1 to the subject calculated in step S104. The distance and the direction of the subject with respect to the voice identification device 1, the characteristic information of the sound source detected in step S107, the distance from the voice identification device 1 to the sound source calculated in step S108, and the direction of the sound source with respect to the voice identification device 1. Associate (step S110).

  Next, the feature information determination unit 40b of the CPU 40 determines whether the feature information of the subject image and the feature information of the sound source associated in step S110 match the image / sound feature information stored in the image / sound feature information storage unit 31. It is determined whether or not (step S111).

  If it is determined in step S111 that the feature information does not match the audio / video feature information (in the case of NO), the storage control unit 40c of the CPU 40 determines the feature information of the subject image and the feature information of the sound source associated in step S110. Is stored in the image / audio feature information storage unit 31 as new image / audio feature information (step S112).

  Next, the tracking control unit 40d of the CPU 40 divides the image displayed on the display unit 42 into a plurality of blocks based on the image data, and tracks the movement of the subject by detecting the movement of each block (step S113). ).

  Specifically, the tracking control unit 40d divides the screen displayed based on the image data into a plurality of blocks, and the subject moves based on the motion vector for each block detected by the motion vector detection unit 25. Detect if there is. The motion vector may be detected using a luminance signal or a color signal.

  Further, the tracking control unit 40d always recognizes the image on the entire screen even when there is no moving object in the screen, and estimates the subject from the outline and color. Image recognition is performed on the subject based on the feature information, and a comparison with the subject detected so far is performed. If the difference between the subject and the characteristic information of the subject so far is smaller than a predetermined value, the subject is determined to be the same object. Thereby, the tracking control unit 40d can track the subject in the screen.

  Then, according to an instruction from the directivity adjustment control unit 40e of the CPU 40, the first directivity characteristic adjustment unit 13a or the second directivity characteristic adjustment unit 13b performs the first microphone 11a, the second microphone 11b, and the third microphone 11c. The directivity characteristics are adjusted by superimposing the sound data generated by the first microphone 11a, the second microphone 11b, and the third microphone 11c so as to eliminate the time difference between the sounds that have reached (step S114). This directivity adjustment process will be described later.

  Next, the CPU 40 determines whether or not an operation signal for requesting the end of shooting is supplied from the operation unit 41 (step S115), and when it is determined that an operation signal for requesting the end of shooting is supplied (in the case of YES). The process is terminated.

  FIG. 12 is a flowchart showing a process flow of the directivity adjustment process in the voice identification device 1 according to the embodiment of the present invention.

  As illustrated in FIG. 12, the directivity adjustment control unit 40e of the CPU 40 determines whether or not at least one of the first directivity characteristic adjustment unit 13a and the second directivity characteristic adjustment unit 13b is usable ( Step S201). Specifically, the CPU 40 determines whether or not there is a first directivity adjustment unit 13a or a second directivity adjustment unit 13b that has not performed directivity adjustment.

  If it is determined in step S201 that both cannot be used, that is, both the first directivity adjustment unit 13a and the second directivity adjustment unit 13b are performing the directivity adjustment (in the case of NO), the directivity adjustment control is performed. The unit 40e extracts the directional characteristic priority stored in the directional characteristic priority storage unit 34 (step S202). Specifically, the directivity adjustment control unit 40e receives from the image / audio feature information storage unit 31 the type of the subject whose movement is being tracked in step S113, and the first directivity adjustment unit 13a and the second directivity adjustment. The type of subject whose directivity characteristics are adjusted by the unit 13b is extracted. Then, the directional adjustment control unit 40e extracts the directional characteristic priority corresponding to the extracted subject type from the directional characteristic priority storage unit 34.

  Next, the directivity adjustment control unit 40e adjusts the directivity characteristic priority of the subject whose movement is being tracked in step S113 by the first directivity characteristic adjustment unit 13a or the second directivity characteristic adjustment unit 13b. It is determined whether or not the directivity priority of the subject is higher (step S203).

  In step S203, the directivity priority of the subject whose movement is being tracked in step S113 is the directivity priority of the subject whose directivity has been adjusted by the first directivity adjustment unit 13a or the second directivity adjustment unit 13b. When it is determined that it is higher than the degree (in the case of YES), the first directivity characteristic adjustment unit 13a or the second directivity characteristic adjustment unit 13b performs directivity adjustment based on an instruction from the directivity adjustment control unit 40e (step S204). ). Specifically, the first directivity characteristic adjustment unit 13a or the second directivity characteristic adjustment unit 13b is based on the tracking result of the tracking control unit 40d, and the first microphone 11a, the second microphone 11b, and the third microphone. The directivity characteristics are adjusted by superimposing the sound data generated by the first microphone 11a, the second microphone 11b, and the third microphone 11c so as to eliminate the time difference of the sound that has reached the microphone 11c.

  As described above, according to the voice identification device 1 according to an embodiment of the present invention, based on the subject image feature information, the voice feature information, the subject distance and direction, and the sound source distance and direction. When the subject and the sound source are associated as the same object, and the feature information of the associated object matches the image sound feature information, the tracking control unit 40d tracks the subject image on the image data, and the first The directivity characteristic adjustment unit 13a and the second directivity characteristic adjustment unit 13b adjust the directivity characteristics of the microphone array 11 based on the tracking result of the tracking control unit 40d and the distance and direction of the subject or the distance and direction of the sound source. Therefore, even when the sound source goes out of the angle of view of the camera 21 or when the sound source intermittently generates sound, the sound position detection unit 12 and the object position detection unit 24b are each Without re-calculating the position, it is possible to obtain a good sound attenuates the noise by adjusting the directional characteristic of the microphone array 11.

  In the voice identification device 1 according to an embodiment of the present invention, two directivity characteristic adjustment units (first directivity characteristic adjustment unit 13a and second directivity characteristic adjustment unit 13b) and two voice detection units (first However, the present invention is not limited to this, and may be configured to include a large number of directional characteristic adjustment units and a large number of sound detection units.

It is the block diagram which showed the structure of the audio | voice identification apparatus which is one Embodiment of this invention. It is the figure which showed an example of the image sound feature information memorize | stored in the image sound feature information storage part with which the audio | voice identification apparatus which is one Embodiment of this invention is provided. It is the figure which showed an example of the image reference | standard feature information memorize | stored in the image reference | standard feature information storage part with which the audio | voice identification apparatus which is one Embodiment of this invention is provided. It is the figure which showed an example of the audio | voice reference | standard feature information memorize | stored in the audio | voice reference | standard feature information storage part with which the audio | voice identification apparatus which is one Embodiment of this invention is provided. It is the flowchart which showed the processing flow of the speech identification device which is one Embodiment of this invention. It is a figure explaining the detection process by the object detection part with which the speech identification device which is one Embodiment of this invention is provided. It is a figure explaining the calculation process of the direction of a to-be-photographed by the object position detection part with which the audio | voice identification apparatus which is one Embodiment of this invention is provided. It is a figure explaining the calculation process of the distance of a subject by the object position detection part with which the audio | voice identification apparatus which is one Embodiment of this invention is provided. It is a figure explaining the detection process by the 1st audio | voice detection part or the 2nd audio | voice detection part with which the audio | voice identification apparatus which is one Embodiment of this invention is provided. (A) shows an example of a waveform of corrected audio data, (b) shows a power spectrum generated based on (a), and (c) shows audio reference feature information storage. An example of the power spectrum of the speech reference feature information stored in the unit 33 is shown. It is a figure explaining the calculation process of the direction and distance of a sound source by the audio | voice position detection with which the audio | voice identification apparatus which is one Embodiment of this invention is provided. It is the figure which showed an example of the phase comparison of the audio | voice waveform input into the 1st microphone, 2nd microphone, and 3rd microphone with which the speech identification device which is one Embodiment of this invention is provided. It is the flowchart which showed the processing flow of the directional characteristic adjustment process in the audio | voice identification apparatus which is one Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Voice identification device 11 ... Microphone array 12 ... Voice position detection part 13a ... 1st directivity characteristic adjustment part 13b ... 2nd directivity characteristic adjustment part 14a ... 1st voice detection part 14b ... 2nd voice detection part 21 ... Camera 22 ... Camera processing unit 23 ... Motion sensor 24 ... Detection unit 24a ... Object detection unit 24b ... Object position detection unit 25 ... Vector detection unit 31 ... Image audio feature information storage unit 32 ... Image reference feature information storage unit 33 ... Audio Reference feature information storage unit 34 ... Directional characteristic priority storage unit 40 ... CPU
40a ... Association unit 40b ... Feature information determination unit 40c ... Storage control unit 40d ... Tracking control unit 40e ... Direction adjustment control unit 41 ... Operation unit

Claims (7)

In a voice identification device for identifying voice emitted from an object,
An imaging unit that converts light collected by the optical system into an electrical signal to generate image data;
A microphone array in which a plurality of microphones that generate sound data by converting sound emitted from a sound source into electrical signals are arranged at predetermined intervals;
An image / sound feature information storage unit that stores the feature information of the subject image included in the image data and the feature information of the sound emitted from the sound source in association with each other and stores it as image / sound feature information
An object detection unit for detecting feature information of a subject image from image data generated by the imaging unit;
A voice detector for detecting voice feature information from the voice data generated by the microphone array;
An object position detection unit that calculates a distance from the voice identification device to the subject and a direction of the subject with respect to the voice identification device based on image data generated by the imaging unit;
A voice position detection unit that calculates a distance from the voice identification device to the sound source and a direction of the sound source with respect to the voice identification device based on voice data generated by the microphone array;
Feature information of the subject image detected by the object detection unit, feature information of the voice detected by the voice detection unit, distance and direction of the subject calculated by the object position detection unit, and voice position detection An association unit that associates the subject and the sound source as the same object based on the distance and direction of the sound source calculated by the unit;
A feature information determination unit that determines whether or not the feature information of the object associated by the association unit matches the image and sound feature information stored in the image and sound feature information storage unit;
A tracking control unit that tracks the subject on the image data when the feature information determination unit determines that the feature information matches the image audio feature information;
Based on the tracking result of the tracking control unit and the distance and direction of the subject calculated by the object position detection unit or the distance and direction of the sound source calculated by the audio position detection unit, the orientation of the microphone array is determined. A directivity adjustment unit for adjusting the characteristics;
A voice identification device comprising: A storage control unit that stores feature information of an object associated by the association unit in the image / audio feature information storage unit as new image / audio feature information when the feature information determination unit determines that the feature information does not match The voice identification device according to claim 1, further comprising: The object position detector
The distance from the voice identification device to the subject and the direction of the subject with respect to the voice identification device are calculated based on an angle of view in the imaging unit and focus information to the subject. Voice identification device. An image reference feature information storage unit that stores the feature information of the subject image and the reference dimension of the subject as image reference feature information in association with each other;
The object position detector
Based on the image reference feature information, a reference size of the subject corresponding to the feature information of the subject image included in the image data is extracted, and the extracted reference size of the subject is used as an angle of view in the imaging unit. The voice identification device according to claim 1, further comprising: calculating a distance from the voice identification device to the subject and a direction of the subject with respect to the voice identification device. The voice position detector
The voice identification device according to claim 1, wherein a distance from the voice identification device to the sound source and a direction of the sound source with respect to the voice identification device are calculated based on a time difference between voices reaching the plurality of microphones. . The tracking control unit
When the feature information determination unit determines that the feature information of the subject image matches, the image displayed based on the image data is divided into a plurality of blocks, and the movement of each block is detected to detect the motion of the subject. The voice identification device according to claim 1, wherein the movement is tracked. The directivity adjusting unit is
Based on the tracking result of the tracking control unit and the distance and direction of the subject calculated by the object position detection unit or the distance and direction of the sound source calculated by the audio position detection unit, the plurality of microphones The voice identification device according to claim 1, wherein voice data generated by the plurality of microphones are superimposed so as to eliminate a time difference between the reached voices.

Images