JP2021021852A - Voice recognition device, electronic apparatus, control method and control program - Google Patents

Abstract

To provide a voice recognition device, an electronic apparatus, a control method and a control program, which improve recognition precision of voice which a person utters in collected sound.SOLUTION: A voice recognition device (control unit 10) in an interactive robot R comprises: a detected sound direction specification unit for specifying a sound generation direction of a sound source from a plurality of detected sounds acquired from a plurality of microphones; a human image detection unit for detecting a human image on the basis of imaging data acquired by imaging the sound generation direction of the sound source or/and a sensor signal acquired by sensing the sound generation direction of the sound source; and a detected sound acquisition validity/invalidity determination unit for determining acquisition of the plurality of detected sounds to be valid or invalid on the basis of information acquired from the human image detection unit.SELECTED DRAWING: Figure 1

Description

The present invention relates to a voice recognition device and the like.

In recent years, various voice recognition devices have been developed that collect utterances with a sensor, a microphone, or the like and recognize the collected sounds as voices uttered by a person.

As a technique related to such a voice recognition device, a technique for preventing malfunctions other than human speech is disclosed. For example, Patent Document 1 includes three microphones, extracts voice data in a specific direction from the sounds collected by these microphones, and presumes that the sound is human voice data based on the extracted voice data. The device is disclosed.

Japanese Unexamined Patent Publication No. 2016-46769 (published on April 4, 2016)

However, in the above-mentioned sound collecting device, when the human voice and the environmental sound such as the sound of a TV program have the same sound quality and the same volume, it may not be possible to determine which sound is the human voice.

One aspect of the present invention is in view of the above-mentioned problems, and an object of the present invention is to realize a voice recognition device or the like that improves the recognition accuracy of a voice spoken by a person among the collected sounds.

In order to solve the above problems, the voice recognition device according to one aspect of the present invention includes a detection sound direction specifying unit that specifies a sound generation direction of a sound source from a plurality of detected sounds acquired from a plurality of microphones, and an imaging unit. Is an imaging data acquired by imaging the sound generation direction of the sound source and / or a human image detection unit that detects a human image based on a sensor signal acquired by the human sensor unit sensing the sound generation direction of the sound source. Based on the information acquired from the human image detection unit, it is determined that the acquisition of the plurality of detected sounds is valid when the human image can be confirmed, or the acquisition of the plurality of detected sounds is invalid when the human image cannot be confirmed. It is characterized by including a detection sound acquisition valid / invalid determination unit for determining that.

In order to solve the above problems, the electronic device according to one aspect of the present invention includes a detection sound direction specifying unit that specifies a sound generation direction of a sound source from a plurality of detected sounds acquired from a plurality of microphones, and a detection sound direction specifying unit of the sound source. A human image detection unit that detects a human image and information acquired from the human image detection unit based on the imaging data acquired by imaging the sound generation direction and / and the sensor signal acquired by sensing the sound generation direction of the sound source. Based on the above, the detection sound acquisition valid / invalid determination unit that enables the acquisition of the plurality of detected sounds when the human image can be confirmed or determines that the acquisition of the plurality of detected sounds is invalid when the human image cannot be confirmed. It is characterized by including a voice recognition device having the above, and a drive unit that drives the image pickup unit in the direction in which the detection sound is generated.

In order to solve the above problems, the control method of the voice recognition device according to one aspect of the present invention includes a detection sound direction specifying step for specifying a sound generation direction of a sound source from a plurality of detected sounds acquired from a plurality of microphones. , A human image detection step for detecting a human image and a human image detection step based on the imaging data acquired by imaging the sound generation direction of the sound source and / and the sensor signal acquired by sensing the sound generation direction of the sound source. Based on the information acquired from, it is determined that the acquisition of the plurality of detected sounds is valid when the human image can be confirmed, or the acquisition of the plurality of detected sounds is determined to be invalid when the human image cannot be confirmed. It is characterized by including an valid / invalid judgment step.

According to one aspect of the present invention, it is possible to improve the recognition accuracy of the voice uttered by a person among the collected sounds.

It is a block diagram which shows the main part structure of the dialogue robot which concerns on Embodiment 1 of this invention. It is a figure which shows the dialogue robot. It is a figure which shows the operation example of an interactive robot. It is a flowchart which shows an example of the processing flow of an interactive robot. It is a figure which shows the operation example of the interactive robot which concerns on Embodiment 2 of this invention. It is a flowchart which shows an example of the processing flow of an interactive robot. It is a block diagram which shows the main part structure of the dialogue robot which concerns on Embodiment 3 of this invention. It is a figure which shows the operation example of an interactive robot. It is a flowchart which shows an example of the processing flow of an interactive robot. It is a block diagram which shows the main part structure of the dialogue robot which concerns on Embodiment 4 of this invention. It is a figure which shows the operation example of an interactive robot. It is a flowchart which shows an example of the processing flow of an interactive robot.

[Embodiment 1]
Embodiment 1 of the present disclosure will be described with reference to FIGS. 1 to 4. FIG. 1 is a block diagram showing a main configuration of the dialogue robot R according to the present embodiment. In the following description, since the control unit functions as a voice recognition device, the duplicate description of the voice recognition device will be omitted.

The dialogue robot R is an electronic device that recognizes a person's utterance as a user and other sounds (for example, the sound of a television program), enables the person's utterance, and invalidates the other sounds.

As shown in FIG. 1, the dialogue robot R includes a microphone 20, a speaker 30, a control unit 10 that functions as a voice recognition device, a drive unit 40, an imaging unit 50, a motion sensor unit 60, and a storage unit. 70 and.

The microphone 20 is an input device that detects sound. The type of microphone is not limited, but the detection sound direction specifying unit 12 described later has detection accuracy and directivity to the extent that the direction of the detected sound can be specified. The microphone 20 is controlled to start and stop sound detection by a detection control unit 18 described later. The dialogue robot R includes a plurality of microphones 20. Further, it is desirable that the dialogue robot R arranges a plurality of microphones 20, 20, 20 in different directions. As a result, it is possible to improve the accuracy of specifying the direction of the detected sound (sound source) by the detected sound direction specifying unit 12 described later.

The speaker 30 outputs a message as a response content by voice according to the control of the output control unit 19 described later. The dialogue robot R may include a plurality of speakers 30. Here, the "response" means a reaction to the utterance of the interactive robot R, which is indicated by voice, motion, light, and a combination thereof.

The control unit 10 is a CPU (Central Processing Unit) that collectively controls the interactive robot R. The control unit 10 includes a detection sound acquisition unit 11, a detection control unit 18, and an output control unit 19 as functional blocks. The control unit functions as a voice recognition device.

The detection sound acquisition unit 11 acquires the detection sound from the microphone 20. The detection sound acquisition unit 11 distinguishes and acquires each detection sound from the plurality of microphones 20. Further, the detection sound acquisition unit 11 divides the detection sound of each microphone 20 by an arbitrary length and acquires the detection sound a plurality of times. The detection sound acquisition unit 11 includes a detection sound direction specifying unit 12, a human image detection unit 13, and a detection sound acquisition valid / invalid determination unit 14.

The detection sound direction specifying unit 12 specifies from which direction the detection sound collected by the microphone 20 comes from. That is, the detection sound direction specifying unit 12 estimates the direction of the sound source from the phase difference and volume difference of the detected sound, and specifies the arrival (generation) direction of the detected sound (voice or the like). The detected sound direction specifying unit 12 comprehensively identifies the sound generation direction of the sound source from the plurality of detected sounds acquired from the plurality of microphones 20, 20, 20. The control unit 10 drives the drive unit 40, which will be described later, based on the arrival direction information indicating the arrival (generation) direction of the detection sound specified by the detection sound direction identification unit 12.

The human image detection unit 13 senses the arrival (generation) direction of the detection sound specified by the detection sound direction identification unit 12 by the imaging data acquired by imaging with the imaging unit 50 described later and / or by the motion sensor unit 60 described later. Based on the sensor signal obtained in the above, it is detected whether or not a human image exists.

Based on the information acquired from the human image detection unit 13, the detection sound acquisition valid / invalid determination unit 14 determines that the detection sound acquisition is valid when the detection sound is derived from a person's utterance, or the detection sound is a person. Judgment is made to invalidate the income of the detected sound when it is not derived from the utterance of.

The imaging unit 50 images the generation direction of the detected sound specified by the detection sound direction specifying unit 12, and transmits the acquired imaging data to the human image detecting unit 13. For example, the image pickup unit 50 includes an image pickup element such as a CCD or CMOS and a circuit such as A / D conversion.

The motion sensor unit 60 senses the generation direction of the detected sound specified by the detection sound direction specifying unit 12, and transmits the acquired sensing signal to the human image detecting unit 13. For example, the motion sensor unit 60 includes a temperature sensor that can detect the temperature of a person's face or the like. Further, as the motion sensor unit, in addition to the temperature sensor, an infrared sensor or an infrared sensor and an ultrasonic sensor may be combined.

The drive unit 40 drives the image pickup unit 50 and / and the motion sensor unit 60 so as to direct the sound (detection sound) of the sound source specified by the detection sound direction specification unit 12. The drive unit 40 includes a moving unit 41 that moves the interactive robot R, which will be described later.

The storage unit 70 is a memory for storing data necessary for processing executed by the control unit 10. The storage unit 70 includes at least a response sentence table and the like. The response statement table is a data table in which the response contents are registered in association with a predetermined sentence or keyword. For example, as the response content, the character string of the message that is the answer to the sentence or keyword is registered.

When there is a response to the recognized word, the output control unit 19 causes the speaker 30 to output a pre-registered response message.

The detection control unit 18 starts the detection of the sound of the microphone 20 and stops the detection of the sound of the microphone 20.

Next, the specific structure and operation of the interactive robot R will be described with reference to FIGS. 2 and 3. FIG. 2 is a diagram showing a dialogue robot R. FIG. 3 is a diagram showing an operation example of the interactive robot R.

As shown in FIG. 2, for example, the interactive robot R has a humanoid structure including a head portion R20, a body portion R30, both arm portions R40, and both foot portions R50. A microphone 20 is arranged on the forehead R20b of the head R20 of the dialogue robot R, one microphone each upward in each of the binocular portions R21 and R21. Further, one microphone 20 is arranged on the back of the head R20c. The dialogue robot R disables the acquisition of the sound (detection sound) of the TV program from the TV device T arranged behind the robot body R10, and then the human voice (detection sound) uttered from the front of the robot body R10. The case where the acquisition of is enabled will be described.

As shown in FIG. 3A, when each microphone 20 of the interactive robot R detects the sound of the TV program (one-dot chain line shown in the figure), the detection sound acquisition unit 11 of the control unit 10 detects the sound of the TV program. To get. The detection sound direction specifying unit 12 specifies the arrival direction of the detected sound (behind the head, the direction of the arrow shown in the figure) from the detected sound of the television program. In this example, the direction of arrival of this detected sound is specified from the volume difference and phase difference of the sound acquired by the microphone 20 of the occipital region R20c (FIG. 3 (b)).

The control unit 10 operates the drive unit 40 (for example, the drive motor) in the body unit R30 of the robot body R10 based on the arrival direction information of the detected sound from the detection sound direction specifying unit 12, and images the crown R20a. The unit 50 or the human sensor unit 60 between the head and eyebrows is directed in the direction of arrival of the detection sound. That is, the control unit 10 drives the drive unit 40 and rotates the interlocking head R20 by 180 degrees to detect the image pickup unit 50 of the crown R20a or the motion sensor unit 60 between the head eyebrows in the direction of arrival of the detection sound. (Fig. 3 (c)). The imaging unit 50 images the television screen (detection target) of the television device T located in the direction of arrival of the detection sound. The imaging unit 50 transmits the imaged data acquired by imaging to the human image detection unit 13. Alternatively, the motion sensor unit 60 senses the television screen (detection target) of the television device T located in the direction of arrival of the detected sound. The motion sensor unit 60 transmits the sensor signal obtained by sensing to the human image detection unit 13.

In the above description, either the imaging unit 50 or the motion sensor unit 60 is operated to acquire the imaging data or the sensor signal, but both the imaging unit 50 and the motion sensor unit 60 are operated to acquire the imaging data. And the sensor signal may be acquired and transmitted to the human image detection unit 13. As shown in FIG. 3C, the front surface of the head R20 faces the television device T, but after the image pickup unit 50 has taken an image or the motion sensor unit 60 has sensed, the drive unit 40 is used. The head R20 is rotated by an angle of 180 ° to return to the original position (FIG. 3 (d)).

When the human image detection unit 13 acquires the image data from the image pickup unit 50 or the sensing signal from the motion sensor unit 60, the human image detection unit 13 detects whether or not a human image exists in the image data or the sensing signal based on the image data or the sensing signal. To do. In this example, the human image detection unit 13 outputs that the human image has not been detected from the image data including the television image of the television device T. Alternatively, the human image detection unit 13 outputs from this sensing signal that the human image has not been detected. The human image detection unit 13 transmits information that the human image has not been detected to the voice acquisition valid / invalid determination unit 14. When detecting a human image from image data, a human image is detected by using a program for recognizing a person (so-called person recognition engine).

The voice acquisition valid / invalid determination unit 14 determines that the acquisition of the detected sound (television screen sound) in the arrival direction of the detected sound is invalid based on the information from the human image detection unit 13 that the human image has not been detected (FIG. 3 (FIG. 3). d)). The area shown by the dotted line in FIG. 3D is the voice invalidation area.

Here, as shown in FIG. 3 (e), when each microphone 20 of the dialogue robot R detects the utterance sound of the person P (one-dot chain line shown in the figure) from the front, the detection sound acquisition unit 11 of the control unit 10 Gets the detection sound derived from a person's utterance. The detection sound direction specifying unit 12 specifies the arrival direction (front of the head) of the detected sound.

The control unit 10 directs the image pickup unit 50 or the motion sensor unit 60 in the arrival direction of the detection sound based on the arrival direction information of the detection sound from the detection sound direction identification unit 12. The image pickup unit 50 images a person P (detection target) located in the direction of arrival of the detection sound, and transmits the image pickup data to the person image detection unit 13. Alternatively, the motion sensor unit 60 senses a person (detection target) located in the direction of arrival of the detection sound, and transmits the sensor signal to the human image detection unit 13.

The human image detection unit 13 detects the existence of a human image from the image data, and transmits the human detection information to the detection sound acquisition valid / invalid determination unit 14. Alternatively, the human image detection unit 13 detects the presence of a human image from the sensing signal and transmits the human detection information to the detection sound acquisition valid / invalid determination unit 14.

The detection sound acquisition valid / invalid determination unit 14 determines that the acquisition of the detection sound (human voice) in the direction of arrival of the detected sound is effective based on the human detection information. The detection sound acquisition unit 11 continues to acquire voice data.

The processing flow of the interactive robot R will be described with reference to FIG. FIG. 4 is a flowchart showing an example of the processing flow of the interactive robot R.

The microphones 20, 20, 20 are in a standby state until they detect sound (S1). When a sound arrives (S2, FIG. 3A), a plurality of microphones 20, 20, 20 detect the sound, the detected sound acquisition unit 11 separately acquires the detected sound, and the detected sound direction specifying unit 12 However, when the arrival direction of the detected sound can be specified from the detected sound and the arrival direction information of the detected sound can be acquired (S3, YES, FIG. 3B), the arrival direction information of the detected sound from the detected sound direction specifying unit 12 Based on the above, the drive unit 40 is operated to direct the image pickup unit 50 of the crown R20a of the head R20 or the human sensor unit 60 between the head eyebrows in the direction of arrival of the detection sound (FIG. 3 (c)). The imaging unit 50 images the direction of arrival of the detected sound. Alternatively, the motion sensor unit 60 senses the arrival direction of the detected sound (S4). The image data or the sensing signal is transmitted to the human image detection unit 13.

When the human image detection unit 13 can detect the presence of the human image based on the image data or the sensing signal (S5, YES), the detection sound acquisition valid / invalid determination unit 14 uses the human image detection information from the human image detection unit 13 as information. Based on this, it is determined that the acquisition of the detected sound (human voice) in the direction of arrival of the detected sound is effective, and the control unit 10 (detection sound acquisition unit 11) acquires voice data which is a human-derived detected sound (S6). , FIG. 3 (e)), and the process returns to the sound arrival process (S2).

If the human image detection unit 13 cannot detect the presence of the human image based on the image data or the sensing signal (S5, NO), the detection sound acquisition valid / invalid determination unit 14 does not detect the human image from the human image detection unit 13. Based on the information in the above, a determination is made to invalidate the acquisition of the detected sound in the direction of arrival of the detected sound, and the control unit 10 (detection sound acquisition unit 11) does not acquire the voice data (S7, FIG. 3D). , Return to the processing of sound arrival (S2).

If the detected sound direction specifying unit 12 specifies the arrival direction of the sound from the detected sound and cannot acquire the arrival direction information of the detected sound (S3, NO), the process returns to the sound arrival process (S2).

According to the above processing, the dialogue robot R can create an environment in which voice recognition is easy by disabling sound collection from an area that is clearly not spoken by a person. That is, in the dialogue robot R, the detection sound direction specifying unit 12 specifies the arrival direction of the sound from the detected sound, operates the drive unit 40 based on the arrival direction information of the detected sound, and operates the drive unit 40, and the head portion R20a of the head R20. It is possible to accurately point the image pickup unit 50 or the human sensor unit 60 between the head and eyebrows in the direction of arrival of the detection sound. The image capturing unit 50 captures the arrival direction of the detection sound and transmits the image data to the human image detecting unit 13. Alternatively, the motion sensor unit 60 senses the arrival direction of the detection sound and transmits the sensing signal to the human image detection unit 13. The human image detection unit 13 detects the existence of a human image based on the image data or the sensing signal, and transmits the information of the human image detection to the detection sound acquisition valid / invalid determination unit 14. The detection sound acquisition enable / disable determination unit 14 accurately determines that the acquisition of the detection sound (human voice) in the direction of arrival of the detection sound is valid based on the information of the human image detection, and thereby the control unit 10 ( The detection sound acquisition unit 11) can reliably acquire only voice data which is a human-derived detection sound. Alternatively, if the human image detection unit 13 cannot detect the presence of the human image based on the image data or the sensing signal, the human image detection unit 13 transmits the information that the human image has not been detected to the detection sound acquisition valid / invalid determination unit 14. The detection sound acquisition valid / invalid determination unit 14 accurately determines that the acquisition of the detected sound in the arrival direction of the detected sound is invalid based on the information that the human image has not been detected from the human image detection unit 13, and thereby the control unit. 10 (detection sound acquisition unit 11) does not acquire detection sounds other than humans.

By performing such processing, it is possible to effectively identify whether the detected sound collected from each microphone 20 is a voice spoken by a person or a sound other than a person, and then effectively acquire the human voice. On the other hand, since it is possible to invalidate the acquisition of sounds other than those spoken by humans, it is possible to suppress external noise that may interfere with voice recognition and improve the recognition accuracy of voices spoken by humans.

[Embodiment 2]
Embodiment 2 of the present disclosure will be described with reference to FIGS. 5-6. For convenience of explanation, the block diagram of the second embodiment and the structure of the dialogue robot are the same as the block diagram and the structure of the dialogue robot of the first embodiment, so duplicate description will be omitted. The same reference numerals are added to the members having the same functions as the members described in the first embodiment, and the description thereof will be omitted.

Next, the specific operation of the interactive robot will be described with reference to FIG. FIG. 5 is a diagram showing an operation example of the interactive robot. A microphone 20 is arranged on the head R20 of the dialogue robot R in each of the binocular portions R21 and R21 in the upward direction (see FIG. 2). Further, one microphone 20 is arranged on the back of the head R20c. As an example in FIG. 5, in the interactive robot R, the acquisition of the sound of the TV program of the TV device T arranged behind the robot body R10 and the sound of the radio program of the radio device Q arranged on the side of the robot body R10 are acquired. A case will be described in which the acquisition of the voice of a person spoken from the front of the robot main body R10 is enabled after the invalidation.

As shown in FIG. 5A, each microphone 20 of the interactive robot R has the sound of the TV program of the TV device T (one-point chain line shown in the figure) and the sound of the radio program of the radio device Q (one-point chain line shown in the figure). ) Is detected, the detection sound acquisition unit 11 of the control unit 10 acquires the detection sound of the TV program and the detection sound of the radio program. The detection sound direction specifying unit 12 specifies the arrival direction of each detection sound (backward and sideways of the head, arrow direction shown in the drawing) from each detection sound. The arrival direction of each detected sound is specified from the volume difference and phase difference of the sounds acquired by each microphone 20 (FIG. 5 (b)).

The control unit 10 operates the drive unit 40 in the body unit R30 based on the information of the arrival direction of each detected sound from the detection sound direction specifying unit 12, and operates the image pickup unit 50 of the crown R20a and the person between the head eyebrows. The motion sensor unit 60 is directed in the direction of arrival of each detection sound.

That is, the control unit 10 drives the drive unit 40 and rotates the interlocking head R20 by an angle of 180 ° to detect the arrival of the detection sound of the image pickup unit 50 of the crown R20a and the motion sensor unit 60 between the head eyebrows. Turn in the direction (Fig. 5 (c)).

The imaging unit 50 images the television screen (detection target) of the television device T located in the direction of arrival of the detection sound. The imaging unit 50 transmits the imaged data acquired by imaging to the human image detection unit 13. In addition, the motion sensor unit 60 senses the television screen (detection target) of the television device T located in the direction of arrival of the detected sound. The motion sensor unit 60 transmits the sensor signal obtained by sensing to the human image detection unit 13.

Further, the control unit 10 further rotates the head R20, which is interlocked by driving the drive unit 40, by an angle of 90 ° to detect the image pickup unit 50 of the crown R20a and the motion sensor unit 60 between the head eyebrows. Turn to the direction of arrival. The imaging unit 50 images the radio device Q (detection target) located in the direction of arrival of the detection sound. The imaging unit 50 transmits the imaged data acquired by imaging to the human image detection unit 13. In addition, the motion sensor unit 60 senses the radio device Q (detection target) located in the direction of arrival of the detected sound. The motion sensor unit 60 transmits the sensor signal obtained by sensing to the human image detection unit 13.

In the above description, both the imaging data and the sensor signal are acquired by operating the imaging unit 50 and the motion sensor unit 60, but either the imaging data of the imaging unit 50 or the sensor signal of the motion sensor unit 60 is used. It may be acquired and transmitted to the human image detection unit 13.

As shown in FIG. 5C, the front surface of the head faces the television device T, but after the image pickup unit 50 takes an image or the motion sensor unit 60 senses the head, the drive unit 40 takes the head. Is rotated to return to the original position (FIG. 5 (d)).

When the human image detection unit 13 acquires each image data from the imaging unit 50 and each sensing signal from the motion sensor unit 60, is there a human image in each image data and each sensing signal based on the image data and the sensing signal? Detect whether or not. In this example, the human image detection unit 13 outputs that the human image has not been detected from the image data including the television image and the image data including the radio device Q. In addition, the human image detection unit 13 outputs that the human image has not been detected from the sensing signal including the television image and the sensing signal including the radio device Q. The human image detection unit 13 transmits information that the human image has not been detected to the detection sound acquisition valid / invalid determination unit 14.

The detection acquisition valid / invalid determination unit 14 determines that the acquisition of both detected sounds (television screen sound and radio program sound) is invalid based on the information from the human image detection unit 13 that the human image has not been detected. The detection sound acquisition unit 11 of the control unit 10 stops the acquisition of both detection sounds. The area shown by the dotted line in FIG. 5C is the voice invalidation area.

Here, as shown in FIG. 5D, when each microphone 20 of the dialogue robot R detects the sound of a person's utterance (one-dot chain line shown in the figure) from the front, the detection sound acquisition unit 11 of the control unit 10 , Acquire the detection sound derived from human utterance. The detection sound direction specifying unit 12 specifies the arrival direction (front of the head) of the detected sound.

The control unit 10 directs the imaging unit 50 and the motion sensor unit 60 in the direction of arrival of the detected sound based on the information of the direction of arrival of the detected sound from the detection sound direction specifying unit 12. The image pickup unit 50 takes an image of the person P located in the direction of arrival of the detection sound, and transmits the image pickup data to the person image detection unit. In addition, the motion sensor unit 60 senses the person P located in the direction of arrival of the detection sound, and transmits the sensor signal to the human image detection unit 13.

The human image detection unit 13 detects the existence of a human image from the image data, and transmits the human detection information to the detection sound acquisition valid / invalid determination unit 14. In addition, the human image detection unit 13 detects the presence of a human image from the sensing signal, and transmits the human detection information to the detection sound acquisition valid / invalid determination unit 14.

The detection sound acquisition enable / disable determination unit 14 determines that the acquisition of the detection sound (human voice) in the direction of arrival of the detection sound is valid based on the information of the person detection. The detection sound acquisition unit 11 of the control unit 10 continues to acquire the detection sound.

The processing flow of the interactive robot will be described with reference to FIG. FIG. 6 is a flowchart showing an example of the processing flow of the interactive robot.

It goes into a standby state until a plurality of microphones 20, 20, 20 detect sound (S11). When a sound arrives (S12), a plurality of microphones 20, 20, and 20 detect the sound, the detected sound acquisition unit 11 separately acquires the detected sound, and the detected sound direction specifying unit 12 detects the sound from the detected sound. When the arrival direction of the sound is specified and the arrival direction information of the detected sound is acquired (S13, YES), and when the number of detected sounds is one (S14, YES), the detected sound direction specifying unit 12 The drive unit 40 is operated based on the information on the arrival direction of the detected sound, and the imaging unit 50 of the crown R20a of the head R20 and the human sensor unit 60 between the head eyebrows are directed in the direction of arrival of the sound. The imaging unit 50 images the direction of arrival of the detected sound. At the same time, the motion sensor unit 60 senses the arrival direction of the sound (S15). The image data and the sensing signal are transmitted to the human image detection unit 13.

When the human image detection unit 13 detects the presence of a human image from the image data and the sensing signal (S16, YES), the detection sound acquisition valid / invalid determination unit 14 determines the direction of arrival of the detected sound based on the human image detection information. The control unit 10 (detection sound acquisition unit 11) acquires voice data to be detected sound derived from a person (S17) after determining that the acquisition of the detected sound (human voice) is valid, and returns to the sound arrival process. (S12).

Here, when there is not one detected sound (S14, NO), the driving unit 40 is operated based on the arrival direction information of each detected sound from the detected sound direction specifying unit 12, and the head portion R20a of the head R20 is operated. The imaging unit 50 and the human sensor unit 60 between the head and eyebrows are directed in the direction of arrival of the detection sound. The imaging unit 50 captures a television image (detection target) of the television device T in the direction of arrival of one detection sound and a radio device Q (detection target) in the direction of arrival of the other detection sound. In addition, the motion sensor unit 60 senses the television image (detection target) of the television device T in the direction of arrival of one detection sound and the radio device Q (detection target) in the direction of arrival of the other detection sound. The image data including the television image, the image data including the radio device Q, the sensing signal including the television image, and the sensing signal including the radio device Q are transmitted to the human image detection unit 13 (S21).

When the human image detection unit 13 detects the presence of a human image from the image data and the sensing signal in the first sound arrival direction (S22, YES), and the presence of the human image from the image data and the sensing signal in the second sound arrival direction. (S23, YES), the detection sound acquisition valid / invalid determination unit 14 determines both of the detection sounds in the arrival directions of the first and second detection sounds based on the information of the human image detection from the human image detection unit 13. After determining to invalidate the acquisition, the control unit 10 (detection sound acquisition unit 11) does not acquire the sound data (S24), and returns to the sound arrival process in order to acquire the sound again (S12).

Alternatively, when the human image detection unit 13 cannot detect the presence of a human image from the image data including the television image in the first detection sound arrival direction and the sensing signal including the television image in the first detection sound arrival direction (S22, NO). When the presence of a human image cannot be detected from the image data including the radio device Q in the second detection sound arrival direction and the sensing signal including the radio device Q (S25, NO), the detection sound acquisition valid / invalid determination unit 14 determines the human image. Based on the undetected information, it is determined to invalidate the acquisition of both detected sounds (sound of TV image, sound of radio device) in the area surrounded by the arrival direction axes L and L of the first and second detected sounds. (S27, FIG. 5 (c)), the detection sound acquisition unit 11 of the control unit 10 does not acquire the sound data (S17, FIG. 5 (d)), and returns to the sound arrival process in order to acquire the sound again. (S12). For example, in the region surrounded by the arrival direction axes L and L of the first and second detected sounds, the direction of the first detected sound (first sound source) is detected at a reference of 0 ° ± 10 °, and the second detection is performed. When the direction of the sound (second sound source) can be detected at 100 ° ± 10 °, it can be obtained from the maximum and minimum values of both detected ranges. In this case, the region surrounded by the arrival direction axes L and L is −10 ° to 110 °.

When detecting the presence of a human image from the image data in the arrival direction of the second detection sound and the sensing signal (S25, YES), the detection sound acquisition valid / invalid determination unit 14 performs the second detection based on the information of the human image detection. The control unit 10 (detection sound acquisition unit 11) acquires voice data after determining that the acquisition of the detected sound in the sound arrival direction is effective (S17).

When the human image detection unit 13 cannot detect the presence of a human image from the image data and the sensing signal (S16, NO), the human image detection unit 13 detects the existence of the human image from the image data and the sensing signal in the second detection sound arrival direction. If it cannot be done (S23, NO), the detection sound acquisition valid / invalid determination unit 14 invalidates the acquisition of the detection sound in the arrival direction of the detection sound based on the information that the human image has not been detected from the human image detection unit 13. After making a determination, the control unit 10 (detection sound acquisition unit 11) does not acquire voice data (S26), and returns to the sound arrival process (S12).

According to the above processing, the dialogue robot R acquires the detected sound from a plurality of sound sources and is clearly not a human, and the area formed by the arrival axis of the detected sound from the plurality of sound sources is invalid. Can be.

That is, the dialogue robot R detects a plurality of detected sounds, the detected sound direction specifying unit 12 specifies the arrival direction of each detected sound, and is based on the arrival direction information of each detected sound from the detected sound direction specifying unit 12. Therefore, the drive unit 40 can be operated to direct the image pickup unit 50 of the crown R20a of the head R20 and the human sensor unit 60 between the head eyebrows in the direction of arrival of the detection sound. The imaging unit 50 can image a television image (detection target) of the television device T in the direction of arrival of one detection sound and a radio device Q (detection target) in the direction of arrival of the other detection sound. In addition, the motion sensor unit 60 can sense the television image (detection target) of the television device T in the arrival direction of one detection sound and the radio device Q (detection target) in the arrival direction of the other detection sound. The image data including the television image, the image data including the radio device Q, the sensing signal including the television image, and the sensing signal including the radio device Q are transmitted to the human image detection unit 13. When the human image detection unit 13 cannot detect the presence of the human image from the two image data including the television images in the first and second detection sound arrival directions and the two sensing signals, the human image detection unit 14 determines whether the detection sound acquisition is valid / invalid. , Based on the information that the human image has not been detected, it is determined that the acquisition of both detected sounds (sound of TV image, sound of radio device) in the area surrounded by the arrival direction axes L and L of the first and second detected sounds is invalid. After that, since the voice of the person detected from the area other than the invalidated area can be effectively acquired, the recognition accuracy of the voice spoken by the person can be improved.

Further, in the dialogue robot R, when a plurality of detected sounds due to speech by a plurality of people are detected, the detected sound direction specifying unit 12 specifies the arrival direction of each detected sound, and the detected sound direction specifying unit 12 sends the detected sound. Based on the arrival direction information of each detection sound, the drive unit 40 can be operated to direct the image pickup unit of the crown R20a of the head R20 and the human sensor unit 60 between the head eyebrows in the arrival direction of the detection sound. .. The imaging unit 50 can image the arrival directions of the two detection sounds, respectively. In addition, the motion sensor unit 60 can sense the arrival directions of the two detection sounds. These image data and sensing signals are transmitted to the human image detection unit 13. When the human image detection unit 13 detects the presence of a human image from the image data and the sensing signal in the first sound arrival direction, and when detecting the existence of the human image from the image data and the sensing signal in the second sound arrival direction, The detection sound acquisition enable / disable determination unit 14 determines to invalidate the acquisition of both detection sounds in the arrival directions of the first and second detection sounds based on the information of the human image detection from the human image detection unit 13, and controls the detection sound acquisition. The unit 10 (detection sound acquisition unit 11) does not acquire the voice data of a plurality of people.

[Embodiment 3]
Embodiment 3 of the present disclosure will be described with reference to FIGS. 7-9. For convenience of explanation, the same reference numerals will be added to the members having the same functions as the members described in the first embodiment, and the description thereof will be omitted. FIG. 7 is a block diagram showing a main configuration of the dialogue robot R according to the present embodiment. In the block diagram of this interactive robot, the functions of the sound source detection unit and the drive control unit are added, while the motion sensor unit is excluded.

The detection sound acquisition unit 11 acquires the detection sound from the microphone 20. The detection sound acquisition unit 11 distinguishes and acquires each detection sound from the plurality of microphones 20. Further, the detection sound acquisition unit 11 divides the detection sound of each microphone 20 by an arbitrary length and acquires the detection sound a plurality of times. The detection sound acquisition unit 11 includes a detection sound direction specifying unit 12, a human image detection unit 13, a detection sound acquisition valid / invalid determination unit 14, a sound source detection unit 15, and a drive control unit 16.

The human image detection unit 13 detects whether or not a human image exists based on the imaging data acquired by imaging the arrival direction of the detection sound specified by the detection sound direction specifying unit 12 by the imaging unit 50 described later.

Based on the information acquired from the human image detection unit 13, the detection sound acquisition enable / disable determination unit 14 determines that the detection sound acquisition is valid when the detection sound is a human utterance, or the detection sound is a person. Judgment is made to invalidate the income of the detected sound when it is not the utterance of.

The imaging unit 50 images the arrival direction of the detected sound specified by the detection sound direction specifying unit 12, and transmits the acquired imaging data to the human image detecting unit 13 and the sound source detecting unit 15 described later.

The sound source detection unit 15 detects the presence of a sound source based on the information of the human image from the human image detection unit 13 and the image pickup data from the image pickup unit 50. The sound source is the sound of a television image of the television device T, the sound of the radio device Q, or the sound of other devices other than human utterances.

The drive control unit 16 controls the operation of the drive unit 40 in order to change the direction of the robot body R10 based on the information from the sound source detection unit 15.

The drive unit 40 drives the image pickup unit 50 so as to direct the image pickup unit 50 in the arrival direction of the detection sound specified by the detection sound direction identification unit 12. Further, the drive unit 40 performs a designated drive based on the control information from the drive control unit 16. The drive unit 40 includes a moving unit 41 that moves the interactive robot R.

Next, the specific operation of the dialogue robot R will be described with reference to FIG. FIG. 8 is a diagram showing an operation example of the interactive robot R. This is a case where the television device T and the person P are in the same direction.

The dialogue robot R detects the sound (detection sound) of the TV program of the TV device T from the front of the robot main body R10, and the imaging unit 50 images the direction of the TV device T. The television screen of the television device T and a human image are reflected in the captured data, and it cannot be determined whether the detected sound is the sound of a television program or the voice of a person.

Therefore, the dialogue robot R changes the direction of the robot main body R10 based on the sound source detection information from the newly provided sound source detection unit 15, utters from the speaker 30, prompts the person P to move, and then newly. The case where the acquisition of the voice (detection sound) of the person spoken in is enabled will be described.

As shown in FIG. 8A, when each microphone 20 of the interactive robot R detects the sound of the TV program (one-dot chain line shown in the figure), the detection sound acquisition unit 11 of the control unit 10 detects the sound of the TV program. To get. The detection sound direction specifying unit 12 specifies the arrival direction of the detected sound (behind the head, the direction of the arrow shown in the figure) from the detected sound of the television program. The arrival direction of the detected sound is specified from the volume difference and the phase difference of the sound acquired by the microphone 20.

The control unit 10 directs the imaging unit 50 of the crown R20a to the arrival direction of the detected sound based on the arrival direction information of the detected sound from the detection sound direction specifying unit 12. The imaging unit 50 images a television device T (detection target) located in the direction of arrival of the detection sound and a person P (detection target) behind the television device T (FIG. 8B). The imaging unit 50 transmits the imaged data acquired by imaging to the human image detection unit 13 and the sound source detection unit 15. This imaging data includes a television device T and a human image.

When the human image detection unit 13 acquires the image data from the imaging unit 50, the human image detection unit 13 detects whether or not a human image exists in the image data based on the image data. In this example, the human image detection unit 13 detects a human image from the television device T and image data including the human image. The human image detection unit 13 transmits the human image detection information to the voice acquisition valid / invalid determination unit 14 and the sound source detection unit 15.

The sound source detection unit 15 detects whether or not a sound source exists based on the information on the human image detection from the human image detection unit 13 and the image data from the image pickup unit 50.

The sound source detection unit 15 reads out sound source image data indicating a sound source stored in the storage unit 70, and image-matches the sound source image data with the image data from the imaging unit 50. As a result of image matching, if the sound source image data and the image data match, the sound source is detected. If the sound source image data and the image data do not match, the sound source is not detected. The sound source detection unit 15 transmits the sound source detection information to the drive control unit 16. Specifically, the sound source detection unit 15 matches the sound source image data with the image data from the image pickup unit 50 including the TV device T and the human image, and the sound source image data and the image data match. Device T (sound source) is detected. The sound source detection unit 15 transmits the detected sound source detection information to the drive control unit 16.

The drive control unit 16 operates the drive unit 40 (moving unit 41) to move the foot portion R50 (see FIG. 2) based on the sound source detection information from the sound source detection unit 15, and directs the direction of the robot body R10 in the designated direction. Change to 90 degrees to the left of (Fig. 8 (c)). The output control unit 19 of the control unit 10 makes an utterance prompting the person P to move from the speaker 30. The person P who has moved according to the movement instruction from the dialogue robot R speaks to the robot main body R10 in front of the robot. The detection sound direction specifying unit 12 identifies the arrival direction of the sound of the sound source (front of the head, the direction of the arrow shown in the figure) from the detected sound, and based on the arrival direction information of the detected sound, the imaging unit 50 of the crown R20a To the direction of arrival of the detection sound. The imaging unit 50 images a person P (detection target) located in the direction of arrival of the detection sound. The imaging unit 50 transmits image data including a human image acquired by imaging to the human image detection unit 13. In FIG. 8C, the moving portion 41 (for example, the drive motor) is operated to move the foot portion R50 to change the direction of the robot body R10 in the designated direction, but the drive portion 40 is driven to drive only the head. You may change it in the designated direction.

When the human image detection unit 13 detects a human image based on the arrival direction information of the detected sound of the detection sound direction specifying unit 12 and the imaging data of the imaging unit 50, the human image detection unit 13 transmits the human image detection information to the voice acquisition valid / invalid determination unit 14. To do.

The voice acquisition valid / invalid determination unit 14 determines that the acquisition of the human voice in the direction of arrival of the detected sound by the human utterance is effective based on the human image detection information from the human image detection unit 13. The detection sound acquisition unit 11 continues to acquire voice data (FIG. 8 (d)).

Here, the processing related to the television device T as a sound source will be described in detail.

The detection sound direction specifying unit 12 identifies the arrival direction of the detected sound from the television device T, drives the drive unit 40 as necessary based on the arrival direction information of the detected sound, and drives the image pickup unit 50 of the crown R20a. To the direction of arrival of the detection sound. The imaging unit 50 images a television device (detection target) located in the direction of arrival of the detected sound (FIG. 8 (b)). The image pickup unit 50 transmits the image pickup data including the television device T that has been imaged and acquired to the human image detection unit 13.

The human image detection unit 13 cannot detect a human image based on the arrival direction information of the detected sound of the detected sound direction specifying unit 12 and the imaging data of the imaging unit 50, and the voice acquisition valid / invalid determination unit obtains the information that the human image has not been detected. Send to 14.

The voice acquisition valid / invalid determination unit 14 determines that the acquisition of the detected sound by the television device T in the direction of arrival of the detected sound by the television device T is invalid based on the information from the human image detection unit 13 that the human image has not been detected. The detection sound acquisition unit 11 of the control unit 10 stops the acquisition of the detection sound. The area indicated by the dotted line in the figure is the voice invalidation area (FIG. 8 (d)).

The processing flow of the interactive robot will be described with reference to FIG. FIG. 9 is a flowchart showing an example of the processing flow of the interactive robot.

It goes into a standby state until a plurality of microphones 20, 20, 20 detect sound (S31). When a sound arrives (S32), a plurality of microphones 20, 20, 20 detect the sound, the detected sound acquisition unit 11 separately acquires the detected sound, and the detected sound direction specifying unit 12 detects the detected sound. When specifying the arrival direction of the sound and acquiring the arrival direction information of the detected sound (S33, YES), the drive unit is operated based on the arrival direction information of the detected sound from the detected sound direction specifying unit 12 to operate the head. The imaging unit 50 of the crown R20a of R20 is directed in the direction of arrival of the detection sound. The imaging unit 50 images the direction of arrival of the detected sound (S34). The image data is transmitted to the human image detection unit 13 and the sound source detection unit 15.

When the human image detection unit 13 can detect the presence of a human image based on the image data (S35, YES), the sound source detection unit 15 uses the human image detection information from the human image detection unit 13 and the image data from the imaging unit 50. Based on this, the sound source image data indicating the sound source stored in the storage unit 70 is read out, the sound source image data and the image data of the imaging unit 50 are image-matched, and as a result of the image matching, the sound source image data and the image data are obtained. If they match (S36, YES), the drive control unit 16 operates the drive unit 40 based on the sound source detection information from the sound source detection unit 15 to move the foot portion R50 and direct the direction of the robot main body R10 in the designated direction. Control to change to 90 degrees to the left of (S37).

The output control unit 19 of the control unit 10 controls the speaker 30 to make an utterance prompting the person P to move. Person P moves to the front of the robot body R10 and speaks. The detection sound direction specifying unit 12 identifies the arrival direction of the detected sound from the detected sound of the human utterance, drives the driving unit 40 based on the arrival direction information of the detected sound, and detects the imaging unit 50 of the crown R20a. Turn to the direction of arrival of sound. The imaging unit 50 images a person (detection target) located in the direction of arrival of the detection sound. The imaging unit 50 transmits image data including a human image acquired by imaging to the human image detection unit 13.

When the human image detecting unit 13 detects a human image based on the arrival direction information of the detected sound of the detected sound direction specifying unit 12 and the imaging data of the imaging unit 50, it means that the person has moved (S38, YES). Returning to the sound arrival process (S32), the processes of S33, S34, S35, and YES are performed, and the sound source detection unit 15 is based on the human image detection information from the human image detection unit 13 and the image data from the imaging unit 50. When the sound source image data indicating the sound source stored in the storage unit 70 is read out, the sound source image data and the image data of the imaging unit 50 are image-matched, and as a result of the image matching, the sound source image data and the image data do not match. (S36, NO), the voice acquisition valid / invalid determination unit 14 determines that the acquisition of the human voice in the direction of arrival of the detected sound is effective based on the human image detection information from the human image detection unit 13. The detection sound acquisition unit 11 continues to acquire voice data of human utterances.

In the process of S35, when the human image detection unit 13 cannot detect the existence of the human image based on the image data (S35, NO), the voice acquisition valid / invalid determination unit 14 does not detect the human image from the human image detection unit 13. Based on the information in (S40), it is determined that the acquisition of the detected sound in the direction of arrival of the detected sound is invalid (S40).

If the movement of a person cannot be confirmed in the process of S38 (S38, NO), the process returns to the process of S38.

According to the above processing, even if the person (speaker) and the other sound source are in the same imaging direction, the dialogue robot prompts the person to move in a different direction by the speaker, and the person and the other sound source communicate with each other. By acquiring human voice after being located in a different direction, it is possible to invalidate the acquisition of detected sounds that are clearly not spoken by humans, suppress external noise that can interfere with voice recognition, and improve the accuracy of voice recognition. It will be possible.

That is, the dialogue robot R detects the detected sound, the detected sound direction specifying unit 12 specifies the arrival direction of the detected sound, and the crown portion is based on the arrival direction information of the detected sound from the detected sound direction specifying unit 12. The imaging unit 50 of R20a can be directed in the direction of arrival of the detection sound. The imaging unit 50 can simultaneously capture a television image (detection target) of the television device T and a person P (detection target). The image data including the television image and the image data including the human image is transmitted to the human image detection unit 13. When detecting the presence of a human image, the human image detection unit 13 transmits the human image detection information to the voice acquisition valid / invalid determination unit 14 and the sound source detection unit 15. When the sound source detection unit 15 detects the presence of a sound source based on the human image detection information from the human image detection unit 13 and the image data from the imaging unit 50, the sound source detection unit 15 transmits the sound source detection information to the drive control unit 16 for drive control. Based on the sound source detection information from the sound source detection unit 15, the unit 16 can control to move the foot portion R50 of the drive unit 40 to change the direction of the robot main body R10 in a designated direction. The dialogue robot R makes an utterance prompting the person P to move from the speaker 30, and effectively acquires the newly spoken voice of the person P who has moved according to the movement instruction. It is possible to invalidate the acquisition of the detected sound by the television device T in the direction of arrival of the detected sound.

[Embodiment 4]
Embodiment 4 of the present disclosure will be described with reference to FIGS. 10-12. For convenience of explanation, since the structure of the dialogue robot in the fourth embodiment is the same as the structure of the dialogue robot in the first embodiment, duplicate description will be omitted. The same reference numerals are added to the members having the same functions as the members described in the first embodiment, and the description thereof will be omitted. FIG. 10 is a block diagram showing a main configuration of the dialogue robot R according to the present embodiment. The difference between the block diagram of the dialogue robot of the fourth embodiment and the block diagram of the dialogue robot of the first embodiment described above is that the function of the drive control unit is added, but the motion sensor unit is excluded.

The detection sound acquisition unit 11 acquires the detection sound from the microphone 20. The detection sound acquisition unit 11 distinguishes and acquires each detection sound from the plurality of microphones 20. Further, the detection sound acquisition unit 11 divides the detection sound of each microphone 20 by an arbitrary length and acquires the detection sound a plurality of times. The detection sound acquisition unit 11 includes a detection sound direction specifying unit 12, a human image detection unit 13, a detection sound acquisition valid / invalid determination unit 14, and a drive control unit 16.

The human image detection unit 13 detects whether or not a human image exists based on the imaging data acquired by imaging the arrival direction of the detection sound specified by the detection sound direction specifying unit 12 by the imaging unit 50 described later. The human image detection unit 13 transmits the detected information to the detection sound acquisition valid / invalid determination unit 14 and the detection sound direction identification unit 12.

Based on the information acquired from the human image detection unit 13, the detection sound acquisition enable / disable determination unit 14 determines that the detection sound acquisition is valid when the detection sound is a human utterance, or the detection sound is a person. Judgment is made to invalidate the income of the detected sound when it is not the utterance of.

The imaging unit 50 images the arrival direction of the detected sound specified by the detection sound direction specifying unit 12, and transmits the acquired imaging data to the human image detecting unit 13.

The drive control unit 16 controls the operation of the drive unit 40 based on the information from the detection sound direction specifying unit 12.

The drive unit 40 drives the image pickup unit 50 so as to direct the image pickup unit 50 in the arrival direction of the detection sound specified by the detection sound direction identification unit 12. Further, the drive unit 40 performs a designated drive based on the control information from the drive control unit 16. The drive unit 40 includes a moving unit 41 that moves the interactive robot R. The designated drive means, for example, that the moving unit 41 is driven to move a predetermined distance to the side (FIG. 11 (c)).

Next, the specific operation of the dialogue robot R will be described with reference to FIG. FIG. 11 is a diagram showing an operation example of the interactive robot R. There is a radio device Q in front of the dialogue robot R, and a television device T in the back of the dialogue robot R. The dialogue robot R detects the sound of the radio device Q (detection sound) from the front of the robot body R10 and the sound of the TV program (detection sound) from the rear of the robot body R10, and the imaging unit 50 determines the direction of the detection sound. Take an image of each. Only the radio device Q and the television screen are reflected in each of the captured data, and there is no person. Therefore, the dialogue robot R moves the robot main body R10 in order to narrow the invalidation range of the detection sound, invalidates the acquisition of both detection sounds in the invalidation range, and then the newly spoken person P. A case where acquisition of voice (detection sound) is enabled will be described.

As shown in FIG. 11A, each microphone 20 of the interactive robot R is the sound of a radio program in front of the robot body R10 (one-point chain line shown in the figure) and the sound of a TV program behind the robot body R10 (in the figure). When each of the one-point chain lines shown in (1) is detected, the detection sound acquisition unit 11 of the control unit 10 acquires the detection sound of the radio program and the detection sound of the television program. The detection sound direction specifying unit 12 specifies the arrival direction of each detection sound (front and rear of the head, two arrow directions shown in the figure) from each detection sound. The arrival direction of each detected sound is specified from the volume difference and phase difference of the sound acquired by the microphone 20 (FIG. 11 (b)).

The control unit 10 operates the drive unit 40 in the body portion R30 based on the information of the arrival direction of each detection sound from the detection sound direction specifying unit 12, and the imaging unit 50 of the crown portion R20a arrives at each detection sound. Turn in the direction.

That is, the control unit 10 drives the drive unit 40 to operate the interlocking head R20 to detect the image pickup unit 50 of the crown R20a in the direction of arrival of the detection sound (front of the head and rear of the head, in the figure). Point in the direction of the arrow shown). The imaging unit 50 images the radio device Q (detection target) and the television screen (detection target) of the television device T, which are located in the direction of arrival of the detection sound. The imaging unit 50 transmits the imaged data acquired by imaging to the human image detection unit 13.

The human image detection unit 13 detects whether or not a human image exists in each image data based on the information on the arrival direction of each detected sound from the detection sound direction specifying unit 12 and each image data from the imaging unit 50. In this example, the human image detection unit 13 outputs that the human image has not been detected from the image data including the radio device Q and the image data including the television image of the television device T. The human image detection unit 13 transmits information that the human image has not been detected to the detection sound acquisition valid / invalid determination unit 14.

The detection acquisition valid / invalid determination unit 14 determines that the acquisition of both detected sounds (sound of the television device and the sound of the radio device) is invalid based on the information that the human image has not been detected from the human image detection unit 13.

Here, the drive control unit 16 determines whether or not the angle created by the arrival direction of each detected sound is a specific angle, for example, 150 ° or more. In this example, the radio device Q is arranged in front of the robot body R10, the arrival direction of the detection sound of the radio device Q is set to an angle of 0 ° (reference), and the television device T is arranged behind the robot body R10. The angle of the arrival direction of the detected sound of the television device T is set to 180 °. That is, when the angle α created by the arrival directions of the detected sounds of the radio device Q and the television device T is, for example, 150 ° or more, the robot body R10 is moved based on the control information from the drive control unit 16. The robot body R10 moves the foot portion R50 by driving the moving portion 41 based on the control information and moves a predetermined distance to the side (FIG. 11 (c)).

In this example, due to the movement of the robot body R10, the angles α formed by the arrival direction axes L and L of the two detection sounds of the radio device Q and the television device T become 150 ° or less, and the detection sound acquisition valid / invalid judgment is made. Based on the information from the human image detection unit 13 that has not detected a person, the unit 14 determines that the acquisition of each detected sound in the arrival direction of the detected sound is invalid (FIG. 11 (d)). The range α in each of the dotted lines L and L in the figure indicates the range in which the detected sound is invalidated.

In this example, when the angle formed by the arrival direction axes L and L of the two detection sounds is 150 ° or less, the detection sound acquisition valid / invalid determination unit 14 uses the human image detection unit 13 as undetected information. Based on this, it was determined that the acquisition of each detected sound in the direction of arrival of the detected sound is invalid, but this angle can be arbitrarily changed, and may be, for example, an angle of 120 °. By narrowing the angle in this way, the effective area can be set wide while the invalid area is narrowed.

As shown in FIG. 11 (e), when each microphone 20 of the dialogue robot R detects the utterance sound of the person P (one-dot chain line shown in the figure) from the front, the detection sound acquisition unit 11 of the control unit 10 moves the person. Acquires the detection sound derived from the utterance of. The detection sound direction specifying unit 12 specifies the arrival direction (front of the head) of the detected sound.

The control unit 10 directs the imaging unit 50 to the arrival direction of the detection sound based on the information of the arrival direction of the detection sound from the detection sound direction identification unit 12. The image pickup unit 50 images a person located in the direction of arrival of the detection sound, and transmits the image pickup data to the human image detection unit 13.

The human image detection unit 13 detects the existence of a human image from the image data, and transmits the human detection information to the detection sound acquisition valid / invalid determination unit 14.

The detection sound acquisition enable / disable determination unit 14 determines that the acquisition of the detection sound (human voice) in the direction of arrival of the detection sound is valid based on the information of the person detection. The detection sound acquisition unit 11 of the control unit 10 continues to acquire the detection sound.

The processing flow of the interactive robot will be described with reference to FIG. FIG. 12 is a flowchart showing an example of the processing flow of the interactive robot.

In the processing flow shown in FIG. 12, the following processing can be performed depending on whether or not a person is present in the direction of each detected sound.
(1) When a person is detected in both the arrival directions of the first and second detected sounds, voice recognition regarding the detected sounds in the first and second voice arrival directions is disabled.
(2) When a person is detected in only one direction of the arrival direction of the detected sound, the voice acquisition from the arrival direction determined that there is no person is invalidated.
(3) When a person is not detected in each of the two detection sound arrival directions, the angle created by the arrival direction axis of each detection sound is confirmed.
(4) When the angle created by the arrival direction axis of each detected sound is 150 ° or more, the robot body (speech recognition device) is moved by the walking motion of the foot of the driving unit.
(5) When the angle created by the arrival direction axis of each detected sound is 150 ° or less, voice acquisition is invalidated in the area surrounded by each arrival direction axis.

Hereinafter, the flowchart shown in FIG. 12 will be described in detail.

It goes into a standby state until the plurality of microphones 20 detect sound (S11). When a sound arrives (S12), a plurality of microphones 20 detect the sound, the detected sound acquisition unit 11 separately acquires the detected sound, and the detected sound direction specifying unit 12 arrives from the detected sound to the sound of the sound source. When the direction is specified and the arrival direction information of the detected sound is acquired (S13, YES) and the number of detected sounds is one (S14, YES), the detected sound from the detected sound direction specifying unit 12 Based on the arrival direction information, the drive unit 40 is operated to direct the image pickup unit 50 of the crown R20a of the head R20 in the arrival direction of the sound. The imaging unit 50 images the direction of arrival of the detected sound. (S15). The image data is transmitted to the human image detection unit 13.

When the human image detection unit 13 detects the presence of a human image from the image data (S16, YES), the detection sound acquisition valid / invalid determination unit 14 detects the detection sound in the generation direction of the detection sound based on the human image detection information (S16, YES). After determining that the acquisition of human voice) is valid, the control unit 10 (detection sound acquisition unit 11) acquires voice data to be detected sound derived from a person (S17), and returns to the sound arrival process (S12). ..

Here, when there is not one detected sound (S14, NO), the driving unit 40 is operated based on the arrival direction information of each detected sound from the detected sound direction specifying unit 12, and the head portion R20a of the head R20 is operated. The imaging unit 50 is directed in the direction of arrival of the detection sound. The imaging unit 50 images the radio device Q (detection target) in the direction of arrival of one detection sound and the television device T (detection target) in the direction of arrival of the other detection sound, respectively. The image data including the radio device Q and the image data including the television device T are transmitted to the human image detection unit 13 (S21).

The human image detection unit 13 detects the presence of a human image from the image data in the first sound arrival direction (S22, YES) and detects the presence of a human image from the image data in the second sound arrival direction (S23). , YES), the detection sound acquisition valid / invalid determination unit 14 determines that the acquisition of both detection sounds in the arrival directions of the first and second detection sounds is invalid based on the information of the human image detection from the human image detection unit 13. The control unit 10 (detection sound acquisition unit 11) does not acquire the sound data (S24), and returns to the sound arrival process in order to acquire the sound again (S12).

Further, in the processing of S22, when the human image detection unit 13 cannot detect the presence of the human image from the image data including the radio device Q in the first detection sound arrival direction (S22, NO), the human image detection unit 13 is in the second detection sound arrival direction. This is a case where the presence of a human image cannot be detected from the image data including the television device T (S25, NO).

When the drive control unit 16 determines that the angle created by the arrival direction of each detection sound is a specific angle, for example, 150 ° or more (S25', YES), the drive control unit 16 is a moving unit of the robot body R10. The control information for controlling the 41 is generated and transmitted to the moving unit 41. The robot body R10 moves a predetermined distance to the side by driving the foot portion R50 of the moving portion 41 based on the control information (S28).

In the process of S25', when it is determined that the angle created by the arrival direction of each detected sound is not a specific angle, for example, 150 ° or more (S25', NO), the detection sound acquisition valid / invalid determination unit 14 does not have a human image. Based on the detection information, it is determined to invalidate the acquisition of both detected sounds in the arrival directions of the first and second detected sounds, and the detected sound acquisition unit 11 of the control unit 10 does not acquire voice data (S27). , The process returns to the sound arrival process in order to acquire the voice again (S12).

When the presence of a human image is detected from the image data in the second detection sound arrival direction (S25, YES), the detection sound acquisition valid / invalid determination unit 14 is based on the human image detection information, and the first detection sound arrival direction. The control unit 10 (detection sound acquisition unit 11) does not acquire the voice data (S26), and returns to the sound arrival process (S12) after determining that the acquisition of the detected sound is invalid.

When the human image detection unit 13 cannot detect the presence of a human image from the image data (S16, NO), the human image detection unit 13 cannot detect the presence of a human image from the image data in the second detection sound arrival direction (S23, NO). In the case of), the detection sound acquisition valid / invalid determination unit 14 determines that the acquisition of the detection sound in the arrival direction of the detected sound is invalid based on the information that the human image detection unit 13 has not detected the human image, and the control unit 10 (detection sound acquisition unit 11) does not acquire voice data (S26), and returns to the sound arrival process (S12).

According to the above processing, in the dialogue robot R, the area formed by the arrival direction axis of the detected sounds from the plurality of sound sources is the acquisition of the detected sounds from the plurality of sound sources and clearly not human. When the area is widened, the robot body R10 moves in a direction that narrows the area formed by the arrival direction axis of the detection sound, forms the area again, and invalidates the voice acquisition of that area. After that, since the voice of the person detected from the area other than the invalidated area can be effectively acquired, the recognition accuracy of the voice spoken by the person can be improved.

The dialogue robot R may be equipped with a drive unit 40 including a moving unit 41 that drives the image pickup unit 50 and the motion sensor unit 60 according to the direction of arrival of the voice, and is not necessarily the humanoid structure shown in FIG. There is no need to prepare. For example, other structures of the interactive robot may be a cylindrical structure, a triangular prism structure, a cubic structure, a rectangular parallelepiped structure, a spherical structure, or the like.

In the above example, the noise source is the TV device T or the radio device Q, but when the sound arrives, the noise source is a radio cassette player, a CD player, an air conditioner, a fixed (installed) device such as a fixed telephone, a sound source, etc. It doesn't matter.

[Modification example]
In each of the above embodiments, the control unit 10 is integrally configured with the storage unit 70, the microphone 20, the imaging unit 50, the motion sensor unit 60, and the speaker 30 in the dialogue robot R, but the control unit 10 and the storage unit are integrated. The 70, the microphone 20, the image pickup unit 50, the motion sensor unit 60, and the speaker 30 are separate devices. For example, at least the microphone 20, the image pickup unit 50, the motion sensor unit 60, and the drive unit are provided in the same device. The control unit 10 and the storage unit 70 may be provided on an external server or the like, and the device and the server may be connected by wired or wireless communication.

For example, the dialogue robot R may include a microphone 20 and a speaker 30. Further, a server different from the dialogue robot R may include a control unit 10 and a storage unit 70. In this case, the dialogue robot R may transmit the detection sound of the microphone 20 to the server, and may receive instruction control related to the stop and start of the detection of the sound of the microphone 20 and the output of the speaker 30 from the server.

Further, the present disclosure may be applied to other than the dialogue robot R. For example, various configurations according to the present disclosure may be realized in smartphones, home appliances, personal computers, and the like.

Further, the dialogue robot R may indicate the response by a method other than voice output. For example, information that specifies a predetermined operation (gesture or the like) of the dialogue robot R as a response may be registered in advance in the response statement table. Then, the control unit may show the operation, that is, the response to the user by controlling the motor or the like of the interactive robot R. Alternatively, the dialogue robot R may be equipped with a display device such as a liquid crystal panel to display a sentence in response to the display device.

[Example of realization by software]
The control block of the control unit 10 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software using a CPU (Central Processing Unit).

In the latter case, the control unit 10 is a CPU that executes instructions of a program that is software that realizes each function, a ROM (Read Only Memory) or a ROM (Read Only Memory) in which the above program and various data are readablely recorded by a computer (or CPU). It is equipped with a storage device (referred to as a "recording medium"), a RAM (Random Access Memory) for developing the above program, and the like.

Then, the object of the present invention is achieved by the computer (or CPU) reading the program from the recording medium and executing the program. As the recording medium, a "non-temporary tangible medium", for example, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used.

Further, the program may be supplied to the computer via an arbitrary transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program.

It should be noted that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the above program is embodied by electronic transmission.

[Summary]
The voice recognition device (control unit 10) according to the first aspect of the present invention is a detected sound direction specifying unit that specifies a sound generation direction of a sound source from a plurality of detected sounds acquired from a plurality of microphones (mics 20, 20, 20). A human image that detects a human image based on (detected sound direction specifying unit 12) and imaging data acquired by imaging the sound generation direction of the sound source and / or a sensor signal acquired by sensing the sound generation direction of the sound source. Based on the information acquired from the detection unit (human image detection unit 13) and the human image detection unit, it is determined that the acquisition of the detection sound is valid when the human image can be confirmed, or the acquisition of the detection sound is invalid when the human image cannot be confirmed. It is provided with a detection sound acquisition valid / invalid determination unit (detection sound acquisition valid / invalid determination unit 14).

According to the above configuration, in the voice recognition device, the detection sound direction specifying unit identifies the sound generation direction of the sound source from the detected sound, and images the sound generation direction of the sound source based on the sound generation direction information of the detected sound. Or / and the sound generation direction of the sound source can be sensed. The human image detection unit detects the human image based on the imaged data acquired by imaging and / and the sensing signal acquired by sensing, and the detection sound acquisition valid / invalid determination unit detects the human image. Based on the information acquired from the department, it is judged that the acquisition of the detected sound is valid or invalid. Therefore, among the detected sounds in the direction of arrival, the detected sounds other than those derived from humans are not acquired, while the detected sounds derived from humans are acquired. Therefore, the sound in the direction of arrival of the detected sound can be recognized with high accuracy as the voice spoken by a person.

In the first aspect, the voice recognition device according to the second aspect of the present invention includes a sound source detection unit (sound source detection unit 15) that detects the presence of a sound source based on human image information from the human image detection unit and imaging data. May be provided.

Since the sound source detection unit detects the existence of the sound source based on the information of the human image from the human image detection unit and the imaging data, it reliably discriminates between a person and a non-human sound source in the direction of arrival of the detected sound, and is derived from a person. Since it is possible to acquire the detected sound derived from a human while not acquiring the detected sound other than the above, the accuracy of voice recognition can be improved, and the malfunction of erroneously responding to the sound other than the human can be prevented.

The voice recognition device according to the third aspect of the present invention includes a drive control unit (drive control unit 16) that controls the drive unit (drive unit 40) based on the information from the sound source detection unit in the second aspect. You may do so.

According to the above configuration, even when the sound source and the person are in the same direction, it is possible to control the drive unit to be changed and driven in the designated direction based on the sound source detection information from the sound source detection unit.

In the first aspect, the voice recognition device according to the fourth aspect of the present invention includes a drive control unit (drive control unit 16) that controls the drive unit (moving unit 41) in the sound generation direction of the sound source. You may.

According to the above configuration, it is possible to control the movement of the drive unit in the sound generation direction of the sound source.

The electronic device (dialogue robot R) according to the fifth aspect of the present invention has a detection sound direction specifying unit that specifies a sound generation direction of a sound source from a plurality of detected sounds acquired from a plurality of microphones, and a sound generation direction of the sound source. Based on the imaged data acquired by imaging and / and the sensor signal acquired by sensing the sound generation direction of the sound source, the human image is generated based on the human image detection unit that detects the human image and the information acquired from the human image detection unit. A voice recognition device (control unit 10) having a detection sound acquisition valid / invalid judgment unit that determines that the acquisition of the detected sound is valid when it can be confirmed or that the acquisition of the detected sound is invalid when the human image cannot be confirmed. A drive unit (drive unit 40) that drives the image pickup unit in the sound generation direction of the sound source is provided.

According to the above configuration, the drive unit can be operated to accurately direct the image pickup unit in the direction of arrival of the sound (detection sound) of the sound source, and the same effect as that of the voice recognition device according to the first aspect can be obtained. ..

The control method of the voice recognition device according to the sixth aspect of the present invention includes a detection sound direction specifying step (S3) for specifying a sound generation direction of a sound source from a plurality of detected sounds acquired from a plurality of microphones, and a sound generation of the sound source. Based on the imaged data acquired by imaging the direction and / or the sensor signal acquired by sensing the sound generation direction of the sound source, the human image detection step (S5) for detecting the human image and the information acquired from the human image detection step are obtained. Based on this, the detection sound acquisition valid / invalid judgment step (which determines that the acquisition of the human-derived detection sound is valid when the human image can be confirmed, or determines that the acquisition of the non-human-derived detection sound is invalid when the human image cannot be confirmed. S6, S7) and. According to the above processing, the same effect as that of the voice recognition device according to the first aspect is obtained.

The voice recognition device according to each aspect of the present invention may be realized by a computer. In this case, the voice recognition device is made into a computer by operating the computer as each part (software element) included in the voice recognition device. The control program of the voice recognition device and the computer-readable recording medium on which the control program is recorded are also included in the scope of the present invention.

The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present invention. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment.

R Dialogue robot (electronic device)
R10 Robot body R20 Head R20a Head R20b Frontal R20c Back of the head R21 Eyes R30 Body R40 Arms R50 Foot 10 Control (voice recognition device)
11 Detection sound acquisition unit 12 Detection sound direction identification unit 13 Human image detection unit 14 Detection sound acquisition valid / invalid judgment unit 15 Sound source detection unit 16 Drive control unit 18 Detection control unit 19 Output control unit 20 Microphone 30 Speaker 40 Drive unit 41 Moving unit 50 Imaging unit 60 Human sensor unit 70 Storage unit P person Q Radio device T TV device

Claims (7)

A detection sound direction identification unit that specifies the sound generation direction of the sound source from multiple detection sounds acquired from multiple microphones,
A human image detection unit that detects a human image based on the imaging data acquired by imaging the sound generation direction of the sound source and / or the sensor signal acquired by sensing the sound generation direction of the sound source.
Based on the information acquired from the human image detection unit, it is determined that the acquisition of the plurality of detected sounds is valid when the human image can be confirmed, or the acquisition of the plurality of detected sounds is invalid when the human image cannot be confirmed. A voice recognition device including a detection sound acquisition valid / invalid judgment unit. A sound source detection unit that detects the presence of the sound source based on the information acquired from the human image detection unit and the imaging data is provided.
The voice recognition device according to claim 1. The voice recognition device according to claim 2, further comprising a drive control unit that controls the drive unit based on information from the sound source detection unit. A drive control unit that controls the drive unit is provided in the sound generation direction of the sound source.
The voice recognition device according to claim 1. With multiple microphones
Imaging unit and
With the drive unit
With the sensor
Has a control unit
The control unit identifies the sound generation direction of the sound source from multiple detected sounds acquired from multiple microphones.
A human image is detected based on the imaging data acquired by imaging the sound generation direction of the sound source and / or the signal from the sensor acquired by sensing the sound generation direction of the sound source.
Based on the information acquired from the human image detection unit, it is determined that the acquisition of the plurality of detected sounds is valid when the human image can be confirmed, or the acquisition of the plurality of detected sounds is invalid when the human image cannot be confirmed. To determine whether the detection sound acquisition is valid / invalid,
Electronics. A detection sound direction identification step that identifies the sound generation direction of the sound source from multiple detection sounds acquired from multiple microphones,
A human image detection step for detecting a human image based on the imaging data acquired by imaging the sound generation direction of the sound source and / or the sensor signal acquired by sensing the sound generation direction of the sound source.
Based on the information acquired from the human image detection step, it is determined that the acquisition of the plurality of detected sounds is valid when the human image can be confirmed, or it is determined that the acquisition of the plurality of detected sounds is invalid when the human image cannot be confirmed. Detection sound acquisition valid / invalid judgment step and
A method of controlling a voice recognition device, including. The control program for operating a computer as the voice recognition device according to claim 1, wherein the computer functions as the detection sound direction specifying unit, the human image detecting unit, and the detected sound acquisition valid / invalid determination unit. Control program.

Images