JPWO2014132533A1 - Voice input device and image display device provided with the voice input device - Google Patents

Abstract

Provided is a voice input device capable of reducing misrecognition of a user voice when there is a sound source that is not intended by a recording person, such as a person who is speaking at a different distance in the same direction as a user who is a voice acquisition target. The audio input device arranges the camera and the microphone array so that the reference point (41) of the camera and the reference point (22) of the microphone array are separated by a predetermined distance L, and the camera is used as a reference based on an image input by the camera. The camera reference user angle θ and the camera reference user distance D are calculated. Then, the voice input device calculates a microphone array reference user angle α with respect to the microphone array based on the camera reference user angle θ and the camera reference user distance D, so that the microphone array reference user angle α becomes the microphone array reference user angle α. Control the pointing angle.

Description

  The present invention relates to a voice input device including a microphone array having a plurality of microphones and capable of inputting voice from a specific direction, and an image display device including the voice input device.

  A method is known in which a microphone array having a plurality of microphones is used to change the directivity angle of voice input, emphasize voice information from a specific direction, and suppress voice information from other directions. By mounting this in a voice input device and directing the directivity angle of the microphone array in the direction in which voice is generated, the recognition accuracy of voice recognition can be improved. When performing voice input by such a method, the direction of the voice uttered by the user is specified, and the directivity angle of the microphone array is directed to the direction of the spoken voice.

  However, since the direction is specified only from the voice information, the directivity angle is also directed to the generation direction of the voice information not intended by the user, such as voice information of a person other than the user, and the voice information not intended by the user May be entered and recognized.

  In view of this, there has been proposed a technique for reducing unintended voice input by detecting a user's direction from image information captured by a camera and directing the direction angle of voice input only in that direction (patent) Reference 1). In the technique described in Patent Literature 1, a speaker direction that dynamically changes is specified using a captured image captured by a camera, and the microphone directivity direction is controlled to be the speaker direction, thereby realizing speech recognition. Improves accuracy.

JP 2009-225379 A

However, the technique described in Patent Document 1 has the following problems.
If there is a sound source not intended by the recording person in the same direction as the user detected by the camera, such as when another speaker is behind the user whose sound is to be acquired with respect to the camera, the directivity angle of the microphone array Even when facing the user's direction, voice unintended by the user may be input, and erroneous recognition may occur in the voice recognition processing. In other words, this technology picks up and recognizes sounds that are not the user's voice from the same direction as the user, such as the voice of a person who is at a different distance from the target user, causing erroneous recognition of the user's voice.

  The present invention has been made in view of the above situation, and its purpose is when there is a sound source that is not intended by the recording person, such as a person who is speaking at a different distance in the same direction as the user who is a voice acquisition target. Another object of the present invention is to provide a voice input device capable of reducing erroneous recognition of a user voice and an image display device including the voice input device.

  In order to solve the above-mentioned problem, a first technical means of the present invention includes a microphone array having a plurality of microphones for acquiring sound, a camera having an image sensor and inputting an image by photographing, and being input by the camera. And a directivity angle control unit that controls a directivity angle of the microphone array based on the image, wherein the camera and the microphone array are arranged apart from each other by a predetermined distance, An apparatus includes: a camera reference user angle calculation unit that calculates a camera reference user angle that is an angle of a user position of a voice acquisition target user based on the camera based on the image; and the camera based on the image A camera reference user distance calculating unit that calculates a camera reference user distance that is a distance to the user position as a reference; the predetermined distance; and the camera reference user A microphone array reference user angle calculation unit that calculates a microphone array reference user angle that is an angle of a user position with respect to the microphone array based on the angle and the camera reference user distance; The unit controls the directivity angle of the microphone array so as to be the microphone array reference user angle.

  According to a second technical means of the present invention, in the first technical means, the microphone array reference user angle calculation unit includes the predetermined distance, the camera reference user distance, the camera reference user angle, and the camera reference user. The microphone array reference user angle range is calculated based on an angle calculation error range corresponding to the angle, and the directivity angle control unit controls the directivity angle of the microphone array so as to be within the microphone array reference user angle range. It is characterized by that.

  According to a third technical means of the present invention, in the first technical means, the microphone array reference user angle calculation unit includes the predetermined distance, the camera reference user angle, the camera reference user distance, and the camera reference user. The microphone array reference user angle range is calculated based on an angle and / or a distance calculation error range corresponding to the camera reference user distance, and the directivity angle control unit is configured to be in the microphone array reference user angle range. The directional angle of the microphone array is controlled.

  According to a fourth technical means of the present invention, in any one of the first to third technical means, the face area of the subject in the image is detected, and the position of the face area is set as the user position, so that the camera reference user The angle and / or the camera reference user distance is calculated.

  According to a fifth technical means of the present invention, in the image display device, the voice input device according to any one of the first to fourth technical means is provided.

  According to the present invention, in the voice input device capable of controlling the directivity angle, when there is a sound source not intended by the recording person, such as a person who is speaking at a different distance in the same direction as the user who is the target of voice acquisition, the distance is determined. Then, by inputting only the voice of the user at the target distance, the misrecognition of the user voice can be reduced, and the user voice can be acquired with low noise.

It is a block diagram which shows the example of 1 structure of the audio | voice input apparatus which concerns on Embodiment 1 of this invention. FIG. 2 is an overhead view showing a configuration example of a microphone array in the voice input device of FIG. 1. It is an overhead view which shows the other structural example of the microphone array in the audio | voice input apparatus of FIG. FIG. 2 is an overhead view showing an example of a positional relationship between a camera and a microphone array in the voice input device of FIG. 1. It is a figure of the perspective projection camera model for demonstrating an example of the camera reference | standard user angle calculation method in the audio | voice input apparatus of FIG. It is a figure for demonstrating the calculation method of the microphone array reference | standard user angle in the audio | voice input apparatus of FIG. 1, and is an overhead view which shows an example in case a user is located in the middle of a camera and a microphone array. It is a figure for demonstrating the calculation method of the microphone array reference | standard user angle in the audio | voice input apparatus of FIG. 1, and is an overhead view which shows an example when a user is located in the same direction seeing from a camera and a microphone array and a place near a microphone array It is. It is a figure for demonstrating the calculation method of the microphone array reference | standard user angle in the audio | voice input apparatus of FIG. 1, Comprising: It is an overhead view which shows an example when a user is located in the same direction and the place near a camera seeing from a camera and a microphone array. is there. It is a figure which shows one structural example of the microphone array in the audio | voice input apparatus which concerns on Embodiment 2 of this invention. It is a figure for demonstrating an example of the calculation method of the microphone array reference | standard user angle range in the audio | voice input apparatus which concerns on Embodiment 2 of this invention, and is an overhead view which shows an example when a calculation error range is large. It is a figure for demonstrating an example of the calculation method of the microphone array reference | standard user angle range in the audio | voice input apparatus which concerns on Embodiment 2 of this invention, and is an overhead view which shows an example when a calculation error range is small. It is a block diagram which shows one structural example of the speech recognition apparatus which concerns on Embodiment 4 of this invention. It is a block diagram which shows one structural example of the image display apparatus which concerns on Embodiment 5 of this invention.

(Embodiment 1)
Hereinafter, Embodiment 1 of the present invention will be specifically described with reference to FIGS.
1 is a block diagram showing a configuration example of a voice input device according to the present embodiment, FIG. 2 is an overhead view showing a configuration example of a microphone array in the voice input device of FIG. 1, and FIG. It is an overhead view which shows the other structural example of the microphone array in 1 audio | voice input apparatus.

  As shown in FIG. 1, the audio input device 1 according to the present embodiment includes a microphone array 11, a camera 12, an in-image user coordinate detection unit 13, a camera reference user angle calculation unit 14, a camera reference user distance calculation unit 15, and a microphone array. A reference user angle calculation unit 16 and a directivity angle control unit 17 are provided.

  The microphone array 11 has a plurality of microphones that acquire sound. Each microphone may be an audio input element, but FIG. 2 shows an example in which the microphone array 11 is configured by arranging a plurality of super-directional microphones 21. Each super-directional microphone 21 is a microphone having a sharp directivity, such as a gun microphone, and is arranged so that the sound collection direction 23 of each super-directional microphone 21 is substantially radial around an arbitrary point in space. The In the present embodiment, the center point of the sound collection direction 23 is referred to as a microphone array reference point 22. However, the microphone array reference point 22 does not need to be exactly one point, and the sound collection directions may intersect with each other in substantially the same region.

  Further, in the present embodiment, the reference direction of the directivity angle of the microphone array 11 is determined as a central direction in which the microphone array 11 is arranged (a direction connecting the array center and the microphone array reference point 22). Is called the microphone array front direction 24. A virtual straight line passing through the microphone array reference point 22 and extending in the microphone array front direction 24 is referred to as a microphone array axis 25. That is, the straight line connecting the center of the array and the microphone array reference point is the microphone array axis 25. By configuring the microphone array 11 in this way, the directivity angle of the input voice information can be controlled by the directivity angle control unit 17.

  In the present embodiment, the directivity angle is controlled by the configuration shown in FIG. 2, but the method for controlling the directivity angle is not limited to this. For example, a plurality of microphones having low directivity may be used as the microphone array 11, and a microphone array that controls the directivity angle by calculating a difference in time and volume of sound reaching each microphone may be used. The arrangement of the microphones illustrated in FIG. 3 is an arrangement example in which the microphone array 11 is realized by controlling the directivity angle by calculating the time difference between the voices reaching each microphone 31 (difference in the voice arrival time). . The microphone 31 is a microphone having weak directivity or a microphone having no directivity. When the directivity direction 32 (directivity angle) is determined, there is a difference in the arrival time of sound to each microphone 31. Only the sound components from the directivity direction 32 can be emphasized by superimposing the sounds of the microphones 31 while shifting the difference between the arrival times. A microphone array reference point 22 is defined as a reference point for determining the directivity direction 32, and the microphone array front direction 24 and the microphone array axis 25 are defined in the same manner as in the configuration of FIG. With the microphone array 11 having such a configuration, the directivity angle of the input audio information can be controlled by the directivity angle control unit 17.

  The camera 12 is an image input unit that has an image sensor and inputs an image or shooting information about the image (for example, including a focal length in the image) by shooting. Examples of the imaging device include solid-state imaging devices such as a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. The camera 12 includes an optical component such as a lens in addition to the image sensor, and can be mounted on the voice input device 1 as a camera module. As the camera 12, it is preferable to use a camera with a wide shooting angle of view, such as a camera equipped with a super-wide-angle lens or a fisheye lens, because a voice acquisition target user (hereinafter simply referred to as “user”) can be photographed over a wide range. .

  Then, the camera 12 in the present embodiment transmits the captured image (and the captured information) to the in-image user coordinate detection unit 13. The camera 12 does not have to be dedicated to still image shooting, and may be dedicated to moving image shooting or may be used for both moving image and still image. Further, the image used by the in-image user coordinate detection unit 13 is basically a still image, and when a moving image is used, a frame immediately before the sound acquisition may be used.

  FIG. 4 is an overhead view showing an example of the arrangement relationship between the camera 12 and the microphone array 11 in the present embodiment. As shown in FIG. 4, the camera 12 and the microphone array 11 are arranged such that their reference points 41 and 22 are separated by a predetermined distance L. Since the change in the user position is greater in the horizontal direction, which is the left-right direction, than in the vertical direction, which is the vertical direction, the camera 12 and the microphone array 11 are moved in the horizontal direction so that the predetermined distance L is in the horizontal direction. It is good to arrange them apart. This is because the desired sound with reduced noise can be output as the input sound by controlling the directivity direction as described later with respect to the separated direction. The camera reference point 41 is a point used as a reference when calculating an angle (direction) where the user exists with respect to the camera 12 within the shooting range 40 of the camera 12. In this embodiment, an example in which the camera focus is the camera reference point 41 is given, but the present invention is not limited to this. Of course, the camera 12 and the microphone array 11 can be arranged apart from each other in the direction other than the horizontal direction, and the directivity direction can be controlled accordingly.

  The camera 12 and the microphone array 11 are arranged so that the camera reference point 41 and the microphone array reference point 22 are not at the same position in the horizontal direction. This is because the change in the user position is larger in the horizontal direction than in the vertical direction as described above.

  Further, the positional relationship between the camera 12 and the microphone array 11 is known at the stage of design or manufacture when both are built in the voice input device 1. On the other hand, since the positional relationship between the camera 12 and the microphone array 11 can be changed independently, the positional relationship between the camera 12 and the microphone array 11 is measured in advance. Fix it so that it will not change later. In the present embodiment, the camera optical axis 42 and the microphone array axis 25 are substantially parallel, the camera shooting direction 43 and the microphone array front direction 24 are the same direction, and pass through the camera reference point 41 with respect to the camera optical axis 42. The microphone array 11 and the camera 12 are installed in such a positional relationship that the microphone array reference point 22 exists on a substantially vertical straight line 44. Further, the microphone array 11 is installed so that the straight line 44 exists on the same plane in which the sound collection direction 23 (FIG. 2) extends radially.

  The in-image user coordinate detection unit 13 detects user position information from an image (input image) captured by the camera 12. The user position can be detected by detecting a face area from the image information, for example. The two-dimensional coordinates of the user's face in the image and the size of the user's face in the image are calculated. As a technique for detecting a face, a generally used method can be used. For example, there is a method in which an image obtained by photographing a standard human face is stored in advance as reference data, and the face is detected from a correlation value with the reference data. Here, as human face images, standard male and female facial images may be stored as reference data, and generational standard male and female facial images. May be held as reference data. Although a specific user's face can be held, such an example will be described later as a third embodiment.

  The in-image user coordinate detection unit 13 transmits the two-dimensional coordinates where the user's face in the image exists to the camera reference user angle calculation unit 14, and the size of the user's face in the image as the camera reference user distance calculation unit 15. To communicate. The detection of the face area is preferable as a process for detecting the user position because there is a mouth that speaks. Of course, the mouth area in the face may be detected instead.

  When there are a plurality of users in the image, the in-image user coordinate detection unit 13 transmits the two-dimensional coordinates where the faces of the respective users in the image exist to the camera reference user angle calculation unit 14, and each in the image The size of the user's face is transmitted to the camera reference user distance calculation unit 15. At this time, since a large amount of data is detected when a large number of users are detected, a specific number of detection results may be transmitted according to the two-dimensional coordinate position of the detected face and the size of the detected face. Good.

  The camera reference user angle calculation unit 14 calculates a camera reference user angle based on the input image. Here, the camera reference user angle is an angle of the user position of the voice acquisition target user with reference to the camera 12.

  The camera reference user angle calculation unit 14 in the present embodiment includes a user who uses the camera reference point 41 as a reference from the in-image user face coordinates that are two-dimensional coordinates in the image transmitted from the in-image user coordinate detection unit 13. The camera reference user angle, which is an angle indicating the direction to perform, is calculated. The camera reference user angle is calculated by using a perspective projection camera model.

  This calculation method will be described with reference to FIG. FIG. 5 is a diagram of a perspective projection camera model for explaining an example of a camera reference user angle calculation method in the voice input device 1. This perspective projection camera model is a model of the camera 12 viewed from a direction perpendicular to the camera optical axis 42 and the straight line 44. If the focal length of the camera 12, the pixel pitch of the sensor (imaging device), and the coordinates of the intersection of the sensor pixel and the optical axis 42 are known, the position of the projection plane 51 can be calculated. The angle indicated by the direction 53 of the user face coordinates 52 in the image on the projection plane 51 from the camera reference point 41 is the camera reference user angle θ. In this way, the camera reference user angle θ can be calculated. The camera reference user angle calculation unit 14 transmits the calculated camera reference user angle θ to the microphone array reference user angle calculation unit 16.

  When the camera 12 is a fish-eye camera, the projection plane 51 becomes a projection camera model that is a sphere centered on the camera reference point 41, and the angle of view and image circle radius of the fish-eye camera, sensor If the coordinates of the intersection of the pixel and the optical axis 42 are known, the position of the projection plane 51 can be calculated, so that the camera reference user angle θ can be calculated by the same method.

  The camera reference user distance calculation unit 15 calculates a camera reference user distance based on the input image. Here, the camera reference user distance is a distance to the user position based on the camera 12.

The camera reference user distance calculation unit 15 in this embodiment calculates the camera reference user distance from the camera reference point to the user from the size of the user's face in the image transmitted from the in-image user coordinate detection unit 13. . The camera standard user distance is obtained by photographing an image of a person's face used as reference data by the in-image user coordinate detection unit 13 using the camera 12 and measuring the distance at the time of photographing. It can be calculated by the following equation (1).
D = d · (t / t ′) (1)

  Each variable of Expression (1) is a camera standard user distance calculated by D, d is a camera standard user distance at the time of capturing reference data, t is the number of vertical pixels of the reference data, and t ′ is a user coordinate detection unit 13 in the image. This is the number of vertical pixels in the extraction range sent from.

  Here, since the size of the face varies depending on age and gender, the user coordinate detection unit 13 in the image does not hold the reference data for each gender or generation (reference data with different face sizes). It is preferable to estimate the age and sex from the face to be detected and calculate the camera reference user distance D because the error in distance calculation can be reduced. This can be realized by setting the relationship between the face size and distance of the reference age and increasing or decreasing the distance according to the estimated age. It can also be realized by setting the relationship between the face size and distance for each age. For example, when faces of the same size are detected, the distance between the child and the adult is different, and the child is present closer than the adult. Therefore, for example, when the detected face is a child, the calculated distance may be increased by a predetermined ratio. The camera reference user distance calculation unit 15 transmits the calculated camera reference user distance D to the microphone array reference user angle calculation unit 16.

  In the present embodiment, the distance D from the camera reference point 41 to the user is calculated from the size of the user face in the image, but this method is used as the method for calculating the distance from the camera reference point 41 to the user. Without being limited thereto, various types of shooting information acquired when the image is acquired by the camera 12 can also be used. For example, if the camera 12 has an autofocus function, the distance of the user position is calculated by using the focus distance when the face area is detected and the detected face area is focused as the user position distance. can do. Also, using a multi-view camera as the TOF (Time Of Flight) using infrared rays or the camera 12, parallax information of the user face area is obtained as shooting information, and the distance from the camera reference point 41 to the user from the parallax information May be calculated. Further, the various distance calculation methods described above can be applied in combination.

  The in-image user coordinate detection unit 13 in the present embodiment detects a face area of a subject (mainly a person but may be an animal) in the input image, and the camera reference user angle calculation unit 14 and the camera reference user distance. The calculation unit 15 calculates the camera reference user angle θ and the camera reference user distance D using the position of the face area as the user position. However, only one of the camera reference user angle θ and the camera reference user distance D may be calculated based on the detection of the face area. Alternatively, either the camera reference user angle θ or the camera reference user distance D can be calculated by detecting the user position without using the face area detection. This point is the same in other embodiments.

  In addition, although the configuration example in which the in-image user coordinate detection unit 13 is provided separately from the camera reference user angle calculation unit 14 and the camera reference user distance calculation unit 15 is described, this configuration example is the same as the in-image user coordinate detection unit 13. This is synonymous with providing both the camera reference user angle calculation unit 14 and the camera reference user distance calculation unit 15 with a function for specifying the user position.

  The microphone array reference user angle calculation unit 16 is based on the predetermined distance L, the camera reference user angle θ transmitted from the camera reference user angle calculation unit 14, and the camera reference user distance D transmitted from the camera reference user distance calculation unit 15. Then, the microphone array reference user angle is calculated. Here, the microphone array reference user angle is an angle of the user position with respect to the microphone array 11, and in this example, a user angle with respect to the microphone array reference point 22.

  FIG. 6 is a view for explaining the method of calculating the microphone array reference user angle, and is an overhead view showing an example in the case where the user is positioned between the camera 12 and the microphone array 11. If the camera reference user angle θ and the camera reference user distance D are calculated, the position of the user 61 relative to the microphone array reference point 22 can be calculated by geometric calculation. Specifically, since the positional relationship between the camera reference point 41 and the microphone array reference point 22 is set in advance including the predetermined distance L when designing the audio input device 1, the microphone array reference point 22 is used as a reference. The user's microphone array reference user angle α can be calculated. For example, even when another speaker 62 is speaking behind the user as viewed from the camera reference point 41 and another speaker 62 cannot be detected by the camera, the directivity angle of the microphone array 11 is directed only to the user 61. Thus, the voice of the user 61 can be input appropriately. The microphone array reference user angle α calculated by the microphone array reference user angle calculation unit 16 is transmitted to the directivity angle control unit 17.

  Moreover, although FIG. 6 demonstrated the case where the user 61 was located between the camera 12 and the microphone array 11, even when a user exists in the same direction from each reference point, it is applicable similarly. This point will be described with reference to FIGS. FIG. 7 is an overhead view showing an example of the case where the user is located in the same direction as viewed from the camera 12 and the microphone array 11 and close to the microphone array 11. FIG. 8 is the same as viewed from the camera 12 and the microphone array 11. It is a bird's-eye view which shows an example in case a user is located in the direction and the place near the camera 12. FIG.

  Even when the user 61 is positioned on the right side when viewed from the reference points 22 and 41 as illustrated in FIG. 7, the user 61 is positioned on the left side when viewed from the reference points 22 and 41 as illustrated in FIG. However, as in the case of FIG. 6, the position of the user 61 relative to the microphone array reference point 22 can be calculated from the camera reference user angle θ and the camera reference user distance D by geometric calculation.

  In this manner, the microphone array reference user angle α of the user 61 with respect to the microphone array reference point 22 can be calculated using the positional relationship between the camera reference point 41 and the microphone array reference point 22. However, when the camera 12 and the microphone array 11 are arranged so that the camera reference point 41 and the microphone array reference point 22 are at the same position, when the other speaker 62 is behind the user 61, the same as viewed from the camera. When the user is at an angle, the user's voice cannot be separated and input independently by directivity control. In addition, the user can not detect overlapping images taken by the camera 12. Therefore, the camera 12 and the microphone array 11 are arranged in a state in which the angle calculated by the camera reference user angle calculation unit 14 is changed by a predetermined distance L. As a result, when another speaker 62 is at substantially the same angle as the user 61 when viewed from the camera 12, it is possible to independently input the voice of each speaker. The predetermined distance L may be any value.

  The directivity angle control unit 17 controls the directivity angle of the microphone array 11 based on the image input by the camera 12. As a main feature of the present invention, the directivity angle control unit 17 controls the microphone array 11 to have the microphone array reference user angle α described above. Control the pointing angle. That is, the directivity angle control unit 17 controls the directivity angle of the microphone array 11 toward the microphone array reference user angle α transmitted from the microphone array reference user angle calculation unit 16 and outputs the sound transmitted from the microphone array 11. To do. 6 to 8, for the sake of convenience, the angles θ and α are expressed as angles formed by the axes 42 and 25, regardless of whether they are positive or negative. The direction may be determined arbitrarily.

  In the present embodiment, by selecting the sound corresponding to the microphone array reference user angle α from the sound information of the superdirectional microphone 21 (or the microphone 31 in FIG. 3) sent from the microphone array 11, Control can be realized. Here, when the microphone array 11 uses a microphone with low directivity, it is possible to acquire sound from a specific angle as sound information by taking into account the difference in sound arrival time.

  In the present embodiment, the angle θ and the angle α are each described as one value. However, since there are errors and widths in the calculated values, a predetermined allowable range can be set for each angle. . For example, the allowable range of the calculated angle can be set in advance as a certain range such as ± 5 °. Note that the allowable range does not need to be set as an angle in advance or does not need to be a certain range, and such an example will be described later as a second embodiment.

  As described above, the camera 12 and the microphone array 11 are disposed in a state where the camera reference user angle θ is changed and is separated by a predetermined distance L, and is separated from the camera 12 by a target distance (the above distance D). By directing the directivity angle of the microphone array 11 to the selected position, when another user (another speaker) 62 is at substantially the same angle as the user 61 when viewed from the camera 12, the voice of each user is independently input. It is possible to reduce misrecognition of the user voice by inputting only the voice of the user who is the target of voice acquisition separately from the voice of another user. Note that by applying a moving image as an input image, the directivity angle is sequentially switched in a direction that matches the position of the user, so that it is possible to reduce erroneous recognition even if the user moves.

  As described above, according to the present invention, in a voice input device capable of controlling the directivity angle, a recording person (speech input device such as a person who is speaking at a different distance in the same direction as the speaker (user) as a voice acquisition target. When there is an unintended sound source of a user who is basically a person who is different from the above speaker), the user's voice is determined by determining the distance and inputting only the voice of the user at the target distance. Misrecognition can be reduced, and user voice can be acquired with low noise.

(Embodiment 2)
Hereinafter, Embodiment 2 of the present invention will be described in detail with reference to FIGS.
FIG. 9 is a diagram illustrating a configuration example of a microphone array in the voice input device according to the present embodiment. 10 and 11 are diagrams for explaining an example of a method of calculating the microphone array reference user angle range in the voice input device according to the present embodiment. FIG. 10 shows an example when the calculation error range is large, and FIG. 11 shows an example when the calculation error range is small.

  A block diagram illustrating a configuration example of the voice input device according to the present embodiment is the same as that illustrated in FIG. 1 according to the first embodiment, and detailed description of each unit is omitted. The operations of the microphone array 11 and the camera 12 are the same as those shown in the first embodiment.

  The directivity angle control unit 17 in the present embodiment is configured to be able to control the directivity range (the range indicated by the directivity angle) of the microphone array 11 of FIG. 1, and the microphone array 11 can also perform such control. It consists of multiple microphones. In the present embodiment, as illustrated in FIG. 9, it is desirable to use a plurality of microphones 31 having low or no directivity as the microphone array 11, and such an example is given. However, even a plurality of microphones having strong directivity can be used.

  The directivity angle control unit 17 emphasizes the sound in each directivity direction (directivity angle) within the directivity range (directivity area) 91 in which the sound is to be acquired by the same method as in the first embodiment, and the sound at other angles. Suppress. Here, as in the example of FIG. 3, the directivity range is controlled by using the difference in the voice arrival times to the plurality of microphones 31. Thereby, the directivity angle control unit 17 can control the directivity range of the microphone array 11. Hereinafter, this control will be specifically described. Note that even a plurality of microphones having high directivity can be controlled by driving one or a plurality of microphones that are directed to the directivity range indicated by the directivity angle.

  The camera 12 transmits the captured image (and the captured information) to the in-image user coordinate detection unit 13 as in the first embodiment. Here, for example, when a fisheye camera is used as the camera 12, the user's face can be detected over a wide range because the shooting angle of view is wide, but since the image is circular, the resolution between the center of the image and the periphery of the image is low. Different. Therefore, when the angle value of the camera reference user angle (angle θ in FIG. 6) with respect to the camera optical axis is small and large, the angle calculation accuracy differs because the resolution of the captured image of the camera 12 is different. Therefore, in this embodiment, the camera reference user angle calculation unit 14 operates in consideration of the angle calculation error range so that a desired effect can be appropriately obtained even if the angle calculation accuracy is different.

  The camera reference user angle calculation unit 14 can reduce malfunction due to an angle calculation error by setting an allowable range when calculating the camera reference user angle θ from the user face coordinates in the image. Therefore, the camera reference user angle calculation unit 14 also calculates an angle calculation error range at the calculated angle θ when calculating the camera reference user angle θ from the user face coordinates in the image.

  The angle calculation error range is calculated, for example, by calculating camera reference user angles θa and θb (see FIG. 10) when the user face coordinates in the image are shifted by one pixel left and right in the camera 12, and the camera reference user angles θa, A range with θb as a boundary is an angle calculation error range 101. At this time, when the resolution of the captured image is high and the angle calculation error is small at a certain camera reference user angle, the change in angle when the user face coordinates in the image are shifted by one pixel is small. On the other hand, when the resolution of the captured image is low and the angle calculation error is large, the angle change is large when the user face coordinates in the image are shifted by one pixel. That is, the angle calculation error range varies depending on the calculated camera reference user angle θ.

  For comparison, a case will be described in which an error tolerance range is set assuming that the error range is constant regardless of the calculated camera reference user angle θ as described in the first embodiment. In such a setting, if the angle calculation error range of the area where the resolution at the center of the input image is high is set as a reference, the resolution is low in the peripheral area (edge side) of the input image. There is a possibility of being excluded. On the other hand, when the angle calculation error range of the low resolution area located around the input image is set as a reference, an excessive tolerance range is provided in the center area of the image, although the resolution is high and the angle calculation error is small. There is a possibility that noise is included in the desired voice.

  In order to prevent such a situation, in this embodiment, the angle calculation error range is different depending on the calculated camera reference user angle θ, and an allowable range of error is set depending on the angle θ, and the input image The angle calculation error range is reduced at the center of the input image, and the angle calculation error range is increased in the peripheral region of the input image. As a result, it is possible to appropriately perform noise reduction and desired voice acquisition.

  Here, the angle calculation error range has been described as a range in which one pixel is shifted to the left and right, but the angle calculation error range can be appropriately set according to shooting conditions such as camera resolution. In addition, the angle calculation error range can be calculated every time from the calculated angle θ, but an LUT (Look Up Table) that defines the angle calculation error range for each calculated angle θ may be held.

  The camera reference user angle calculation unit 14 in this embodiment transmits the calculated camera reference user angle θ and the angle calculation error range 101 to the microphone array reference user angle calculation unit 16.

  The camera reference user distance calculation unit 15 sets an allowable range when calculating the camera reference user distance D from the size of the face of the user in the image transmitted from the user coordinate detection unit 13 in the image, It is possible to reduce malfunctions due to distance calculation errors. Therefore, the camera reference user distance calculation unit 15 also calculates a distance calculation error range when calculating the camera reference user distance D from the in-image user face coordinates.

  In the present embodiment, the distance calculation error range includes distances Da and Db when the size of the user's face in the image is increased by one pixel on the left and right and by one pixel on the left and right (see FIG. 10). And a range with the calculated distances Da and Db as a boundary is set as a distance calculation error range 102. At this time, when the camera reference user distance D is short, the change in the user distance when the size of the user face in the image is shifted by one pixel is small. On the other hand, when the distance D is long, the user distance change becomes large when the size of the user face in the image is shifted by one pixel. That is, the distance calculation error varies depending on the calculated camera reference user distance D.

  For comparison, a case will be described in which an allowable error range is set assuming that the distance calculation error range is constant regardless of the calculated camera reference user distance D. In such a setting, if the distance calculation error range when the distance D is short is set as a reference, the distance calculation error increases when the distance D is long, and a desired user voice may be excluded. On the other hand, if the distance calculation error range when the distance D is long is set as a reference, an excessive allowable range is provided even though the distance calculation error is small when the distance D is short. There is a possibility of becoming.

  The distance calculation error range 102 also changes depending on the camera reference user angle θ at the time of calculation. For example, when a camera equipped with a fisheye lens is used as the camera 12, the resolution of the face area decreases as the camera reference user angle θ increases. When the resolution of the captured image is high and the distance calculation error is small, the change in the user distance when the size of the user face in the image is shifted by one pixel is small. On the other hand, when the resolution of the captured image is low and the distance calculation error is large, the change in the user distance is large when the size of the user face in the image is shifted by one pixel. That is, the distance calculation error varies depending on the calculated camera reference user angle θ.

  For comparison, a case will be described in which an allowable error range is set assuming that the distance calculation error range is constant regardless of the calculated camera reference user angle θ. In such a setting, if the distance calculation error range of the area where the resolution at the center of the input image is high (corresponding to the area where the angle θ is small) is set as a reference, the distance calculation error is small because the resolution is low in the peripheral area of the input image There is a possibility that a desired user voice is excluded due to the increase. On the other hand, if the distance calculation error range of the low resolution area (corresponding to the area where the angle θ is large) located around the input image is set as a reference, the area at the center of the image has high resolution and small distance calculation error. Since an excessive allowable range is provided, there is a possibility that noise is included in a desired user voice.

  In order to prevent such a situation, in this embodiment, the distance calculation error range is different depending on the camera reference user distance D and the camera reference user angle θ, and the allowable range of error depends on the distance D and the angle θ. Set. The distances Da and Db are calculated when the size of the user's face in the image is increased by one pixel on the left and right and by one pixel on the left and right, and the distance between the calculated distances Da and Db is calculated as the boundary. By setting the error range 102, the distance calculation error range 102 is small at the center of the input image (corresponding to a region where the angle θ is small), and at the peripheral region of the input image (corresponding to a region on the edge side where the angle θ is large) When the distance calculation error range 102 is large and the distance D is short, the distance calculation error range 102 is small, and when the distance D is long, the distance calculation error range 102 is large. As a result, noise reduction and desired voice acquisition can be appropriately performed as a result, which is preferable.

  Here, the distance calculation error range has been described as a range in which the size of the user's face in the image is shifted by one pixel left and right, but can be set as appropriate depending on shooting conditions such as camera resolution. In addition, the distance calculation error range can be calculated every time from the calculated angle θ and the calculated distance D, but an LUT that defines the distance calculation error range for each calculated angle θ and the calculated distance D is held. May be.

  In the present embodiment, the distance calculation error range is different depending on the distance D and the angle θ, and an example in which the allowable range of error is set depending on the distance D and the angle θ is given. However, the distance calculation error range may be different depending on only one of the distance D and the angle θ, and the allowable range of error may be set depending on the distance D or the angle θ. Also in this case, a LUT that defines a distance calculation error range for each calculated distance D or a LUT that defines a distance calculation error range for each angle θ at the time of calculation is held, and a distance calculation error is referred to by referring to the LUT. You may ask for.

  The camera reference user distance calculation unit 15 of the present embodiment transmits the camera reference user distance D, which is the calculated distance from the camera reference point 41 to the user, and the distance calculation error range 102 to the microphone array reference user angle calculation unit 16. .

  The microphone array reference user angle calculation unit 16 uses the microphone array reference point 22 as a reference based on the predetermined distance L, the camera reference user angle θ, the angle calculation error range 101, the camera reference user distance D, and the distance calculation error range 102. The microphone array reference user angle range that is the range of the user angle is calculated.

  A method for calculating the microphone array reference user angle range will be described with reference to FIGS. 10 and 11. The microphone array reference user angle calculation unit 16 receives the predetermined distance L, the camera reference user angle θ and the angle calculation error range 101 transmitted from the camera reference user angle calculation unit 14, and the camera reference user distance calculation unit 15. From the camera reference user distance D and the distance calculation error range 102, a range 103 that can be estimated that the user 61 exists is calculated by geometric calculation. Here, the range 103 is a common range that is a common part between the angle calculation error range 101 and the distance calculation error range 102.

  Specifically, since the positional relationship between the camera reference point 41 and the microphone array reference point 22 is set in advance including the predetermined distance L when designing the voice input device 1, a range in which it can be estimated that the user 61 exists. A microphone array reference user angle range 104 with reference to the microphone array reference point 22, which includes 103, can be calculated. At this time, a large range 103 is detected with respect to the size of the face of the user 61 as shown in FIG. 10, or a small range 103 is detected with respect to the size of the face of the user 61 as shown in FIG. Sometimes. However, since the range 103 is calculated in consideration of an appropriate error based on the distance D and the angle θ, the range 103 can be said to be an appropriate range in consideration of the error, and the microphone array reference user angle range obtained from the range 103 104 is also an appropriate range.

  The microphone array reference user angle range 104 calculated by the microphone array reference user angle calculation unit 16 is transmitted to the directivity angle control unit 17. At this time, the microphone array reference user angle (the median angle of the range 104) may be transmitted at the same time.

  The directivity angle control unit 17 in the present embodiment controls the directivity range 91 of the microphone array 11 so that the sound in the microphone array reference user angle range 104 transmitted from the microphone array reference user angle calculation unit 16 can be acquired. That is, the directivity angle control unit 17 in the present embodiment controls the directivity range 91 that is the directivity angle of the microphone array 11 to be the microphone array reference user angle range 104. Thereby, the directivity range of the microphone array 11 can be appropriately controlled according to the microphone array reference user angle range 104.

  According to the voice input device 1 of the present embodiment, an appropriate microphone array reference user angle range 104 can be calculated according to the resolution for each camera reference user angle θ of the camera 12 and the resolution for each camera reference user distance D. The directivity range of the microphone array can be set appropriately. Further, the microphone array reference user angle (the angle of the median value in the range 104) is also transmitted, or the microphone array reference user angle (the angle α described above) is obtained from the microphone array reference user angle range 104 as the angle of the median value. It is also possible to perform control such as increasing the voice input level in the directivity direction indicated by and lowering the voice input level as it approaches the end of the range 104.

  As described above, the microphone array reference user angle range 104 is set with respect to the result of calculating the angle calculation error range 101 and the distance calculation error range 102. However, only the angle calculation error range 101 or the distance calculation error range 102 is given. It is possible to perform such setting only by using only, and in any case, an effect of realizing noise reduction and desired voice acquisition can be obtained. When such a setting is performed only in the angle calculation error range 101, such a setting is performed only in the distance calculation error range 102 so as to sandwich the arc that is the distance D and the angle calculation error range 101. The microphone array reference user angle range 104 is set so as to sandwich the line segment that is the angle θ and the distance calculation error range 102.

  That is, the microphone array reference user angle calculation unit 16 may calculate the microphone array reference user angle range based on the predetermined distance, the camera reference user distance, the camera reference user angle, and the angle calculation error range. Alternatively, the microphone array reference user angle calculation unit 16 may calculate the microphone array reference user angle range based on a predetermined distance, a camera reference user angle, a camera reference user distance, and a distance calculation error range.

  In addition, examples of calculating the angle calculation error range and the distance calculation error range are such that the edge side of the input image is often out of focus compared to the center part, and therefore the camera reference user angle and camera reference user Since it can be said that the distance calculation error often increases, it is useful even when the user position is detected by a method other than the detection of the face area.

  As described above, in the present embodiment, the microphone array reference user angle range is set according to the camera reference user angle and / or the camera reference user distance, more specifically, the resolution and / or the camera reference user angle. Calculation is performed according to the resolution for each camera reference user distance, and thus the directivity range of the microphone array can be appropriately controlled.

(Embodiment 3)
Hereinafter, Embodiment 3 of the present invention will be specifically described.
A block diagram showing a configuration example of the voice input device according to the present embodiment is the same as that shown in FIG. The operations of the microphone array 11 and the camera 12 are the same as those shown in the first embodiment. However, the features of the present embodiment can also be applied to the second embodiment.

  The in-image user coordinate detection unit 13 in this embodiment identifies an individual user from the image captured by the camera 12 and detects position information of the user. For example, the user can be specified by performing template matching on an image obtained by photographing a user's face (hereinafter referred to as a user's face) stored in advance, using an image whose magnification is changed in several stages as a template image.

  More specifically, the range of the same number of pixels as the template image is extracted as a window image from the image for detecting the user face, and the sum of the pixel value differences from the corresponding template image pixels for all the pixels in the window image Calculate the value. Differences are calculated for all window images that can be extracted from the user face detection image, and a range in which a window image with the smallest difference is extracted is searched. The two-dimensional coordinates for the image that detects the user face in the extraction range where the difference is the smallest and the difference is less than or equal to the threshold value are the two-dimensional coordinates of the user's face, and the extraction range where the difference is the smallest and the difference is less than or equal to the threshold value For example, the number of vertical pixels is set as the size of the user face. When there is no extraction range where the difference is the smallest and the difference is equal to or less than the threshold, it is determined that there is no user face in the image.

  The in-image user coordinate detecting unit 13 transmits the in-image user face coordinates and the user specifying information in the image to the camera reference user angle calculating unit 14, and the size and user specifying information for each user face in the image. (ID indicating the user or name) is transmitted to the camera reference user distance calculation unit 15.

  The camera reference user angle calculation unit 14 calculates the camera reference user angle from the user face coordinates in the image of the specified user, and associates the user with the user specification information indicating which user the camera reference user angle belongs to. The calculation of the camera reference user angle can be realized by the same method as in the first embodiment. The camera reference user angle calculation unit 14 transmits the camera reference user angle to which the user specifying information is given to the microphone array reference user angle calculation unit 16.

  The camera reference user distance calculation unit 15 calculates the camera reference user distance from the camera reference point to each user from the size of the user's face in the image transmitted from the in-image user coordinate detection unit 13, Which user the camera reference user distance belongs to is associated with user identification information. The camera reference user distance is implemented by photographing an image of a user face used as a template image by the in-image user coordinate detection unit 13 using the camera 12 and measuring the distance at the time of photographing. It can be calculated by the calculation of equation (1) as in the first mode.

  In the present embodiment, each variable of the expression (1) is a camera reference user distance calculated by D, d is a camera reference user distance at the time of photographing a template image, t is the number of vertical pixels of the template image, and t ′ is an in-image user. The number of vertical pixels in the extraction range sent from the coordinate detection unit 13. Here, since the size of the face varies depending on the user, the size of the user's face specified by the in-image user coordinate detection unit 13 is calculated by calculating the camera reference user distance using the distance at the time of shooting the template image of the user. The camera reference user distance reflecting the distance is calculated, and an error in distance calculation can be reduced. The camera reference user distance calculation unit 15 transmits the camera reference user distance given the user identification information to the microphone array reference user angle calculation unit 16.

  The microphone array reference user angle calculation unit 16 is based on the predetermined distance L, the camera reference user angle θ transmitted from the camera reference user angle calculation unit 14, and the camera reference user distance D transmitted from the camera reference user distance calculation unit 15. Thus, the microphone array reference user angle that is the user angle with respect to the microphone array reference point 22 is calculated. The microphone array reference user angle can be calculated from the camera reference user angle θ and the camera reference user distance D to which the same user identification information is given in addition to the predetermined distance L by the same method as in the first embodiment.

  The microphone array reference user angle calculation unit 16 uses the user identification information transmitted from the camera reference user angle calculation unit 14 and the user identification information transmitted from the camera reference user distance calculation unit 15 as the microphone array reference user angle α. Give. The microphone array reference user angle calculation unit 16 transmits the microphone array reference user angle α to which the user specifying information and the user specifying information are given to the directivity angle control unit 17.

  The directivity angle control unit 17 performs directivity angle control of the microphone array 11 toward the microphone array reference user angle α transmitted from the microphone array reference user angle calculation unit 16, and outputs the sound transmitted from the microphone array 11. Thereby, the distance calculation in consideration of the actual face size of the user can be performed, and the directivity angle control in the directivity angle control unit 17 can be performed with high accuracy.

  Here, in this embodiment, when photographing a user face in advance, the face photographing data and the distance are associated with each other as reference data. However, the face attribute detected by the in-image user coordinate detection unit 13 is used as user identification information. The camera reference user distance may be corrected. The face attribute can be set by sex, age, etc., such that males are larger than females and adults are larger than children. A conventional method can be used to determine each attribute, and it can be estimated from the similarity of reference data. Thus, the directivity angle control in the directivity angle control unit 17 can be accurately performed by calculating the user distance in consideration of the face attribute.

(Embodiment 4)
Hereinafter, Embodiment 4 of the present invention will be specifically described with reference to FIG.
FIG. 12 is a block diagram illustrating a configuration example of a speech recognition apparatus according to Embodiment 4 of the present invention. As illustrated in FIG. 12, the speech recognition apparatus according to the present embodiment is a speech recognition apparatus including the speech input device 1 and the speech recognition unit 121 described in Embodiment 3. However, this embodiment can be applied to both the first and second embodiments.

  The voice recognition unit 121 analyzes the voice input result transmitted from the voice input device 1 and outputs the analysis result. As the analysis method, a conventional method can be used. For example, the speech is analyzed and recognized by comparing with reference data. The output result is converted into text data or used for controlling ON / OFF of the electric device. Since the speech recognition apparatus according to the present embodiment includes the speech input device 1 that can appropriately acquire desired speech with low noise, it is possible to recognize speech with high accuracy.

  Furthermore, the directivity angle control unit 17 according to the present embodiment can add the user identification information given to the microphone array reference user angle α used for directivity angle control to the voice transmitted from the microphone array 11 and output the voice. it can. Thereby, in speech recognition, it is possible to use a speech recognition method with high speech recognition accuracy in accordance with the user, which is preferable because erroneous recognition of speech recognition can be reduced. For example, the voice recognition reference data is updated from the user identification information and the voice input result, and voice recognition is performed using the reference data for each user. As a result, it is possible to realize speech recognition in consideration of user-specific speech features. As the feature amount, a general speech parameter can be targeted, and there are intonation, frequency, and the like. Therefore, by providing the user identification information to the voice input, the reference data for voice recognition can be changed according to the user, and erroneous recognition can be reduced.

(Embodiment 5)
Hereinafter, Embodiment 5 of the present invention will be specifically described with reference to FIG.
FIG. 13 is a block diagram illustrating a configuration example of an image display apparatus according to the fifth embodiment of the present invention. As illustrated in FIG. 13, the image display device according to the present embodiment is an image display device including the voice recognition device, the control unit 131, and the image display unit 132 described in the fourth embodiment. Although the description will be made on the assumption that the image display device according to the present embodiment includes the voice recognition unit 121, the voice recognition unit 121 may be configured as a separate device, and the image display device according to the present embodiment is What is necessary is just to provide the voice input device 1 as described in the first to third embodiments.

  The control unit 131 controls display contents of the image display unit 132, power ON / OFF, volume, and the like based on the voice recognition result for each user transmitted from the voice recognition unit 121. As a result, the image display apparatus according to the present embodiment can be operated by voice. The voice input device 1 is preferably mounted on a remote controller of the image display device.

  Examples of the contents of the operation include a television broadcast channel change, a program table call, and a program designation. The image display apparatus according to the present embodiment includes the voice input device 1 that can appropriately obtain desired voice with low noise, and can recognize voice with high accuracy. Therefore, more accurate operation using voice can be performed. It can be carried out.

  The timing of shooting by the camera 12 may be performed when such an operation is performed. In any of the first to fifth embodiments, the timing of shooting by the camera 12 in the voice input device 1 may be during voice input, immediately before or after voice input, and a predetermined operation (for example, It may be when the voice input button is pressed or a shutter button different from the voice input operation is pressed.

  In addition to the present embodiment, if the voice input device 1 includes an image display unit, the user specifies the user position by touching the user's face while the captured image is displayed by the camera 12. The camera reference user angle and the camera reference user distance can also be calculated based on the designated position.

  Thus, according to the present embodiment, an image display device including a voice input device that can appropriately control the directivity range of the microphone array 11 can be provided. Furthermore, since the image display apparatus according to the present embodiment has user position detection functions such as the camera 12 and the in-image user coordinate detection unit 13, it is possible to detect in-image user coordinates of a plurality of users. In addition, the image of the camera 12 can be displayed on the image display unit 132. In addition, in this embodiment, the voice input device 1 can separately input a plurality of user voices with low noise. Therefore, if the image display apparatus according to the present embodiment is used, for example, a video conference system that can cut out and display images of a plurality of users, and distribute the images and sounds of each user separately for each user. Can be built. In addition, examples of the image display device in the present embodiment include a mobile phone (including a so-called smartphone), a tablet, and the like.

(Other embodiments)
For example, each component of the apparatus according to the present invention, such as a portion excluding the microphone array 11, the camera 12, and the image display unit 132 illustrated in FIGS. 1, 12, and 13, is, for example, a microprocessor (or DSP: Digital Signal Processor). It can be realized by hardware such as peripheral devices such as a memory, a bus, an interface, and a remote controller, and software that can be executed on these hardware. A part of the hardware can be mounted as an integrated circuit / IC (Integrated Circuit) chip set. In this case, the software may be stored in the memory. In addition, all the components of the present invention may be configured by hardware, and in that case, a part of the hardware can also be mounted as an integrated circuit / IC chip set.

  In addition, a recording medium on which a program code of software for realizing the functions in the various configuration examples described above is recorded is a voice input device or a device (may be a general-purpose computer) including the same, and a microphone array 11 and the camera 12 are supplied to a built-in or connected device, and the program code is executed by a microprocessor or a DSP in the device, and the object of the present invention is achieved. In this case, the software program code itself realizes the functions of the above-described various configuration examples. Even if the program code itself or a recording medium (external recording medium or internal storage device) on which the program code is recorded is used. The present invention can be configured by the control side reading and executing the code. Examples of the external recording medium include various media such as an optical disk such as a CD-ROM or a DVD-ROM and a non-volatile semiconductor memory such as a memory card. Examples of the internal storage device include various devices such as a hard disk and a semiconductor memory. The program code can be downloaded from the Internet and executed, or received from a broadcast wave and executed.

  As described above, the voice input device according to the present invention and the voice recognition device and the image display device including the voice input device have been described. As described above, the present invention has the arrangement as described above. A voice input direction changing method in a voice input device including a microphone array and a camera can also be adopted. In this audio input direction changing method, a camera reference user angle calculation unit calculates a camera reference user angle that is an angle of a user position of an audio acquisition target user with reference to the camera, based on an image input by the camera; A camera reference user distance calculating unit calculating a camera reference user distance, which is a distance to the user position based on the camera, based on the image; and a microphone array reference user angle calculating unit Calculating a microphone array reference user angle, which is an angle of a user position with respect to the microphone array, based on the camera reference user angle and the camera reference user distance; And controlling the directivity angle of the microphone array so as to be at an angle. Other application examples are the same as those described for the voice input device, and a description thereof will be omitted.

  The program code itself is a program for causing a computer to execute the voice input direction changing method. That is, the program calculates a camera reference user angle, which is an angle of a user position of a voice acquisition target user based on the camera, based on an image input to the computer by the camera, and based on the image, Calculating a camera reference user distance, which is a distance from the camera to the user position, and a user based on the microphone array based on the predetermined distance, the camera reference user angle, and the camera reference user distance A step of calculating a microphone array reference user angle, which is a position angle, and a step of controlling the directivity angle of the microphone array so as to be the microphone array reference user angle are executed. Other application examples are the same as those described for the voice input device, and a description thereof will be omitted.

  As described above, the audio input device according to the present invention is based on a microphone array having a plurality of microphones for acquiring audio, a camera having an image sensor and inputting an image by photographing, and the image input by the camera. And a directivity angle control unit that controls the directivity angle of the microphone array, wherein the camera and the microphone array are spaced apart from each other by a predetermined distance, and the speech input device A camera reference user angle calculation unit that calculates a camera reference user angle that is an angle of a user position of a voice acquisition target user based on the camera, and the user based on the camera based on the image A camera reference user distance calculation unit for calculating a camera reference user distance that is a distance to the position; the predetermined distance; the camera reference user angle; A microphone array reference user angle calculation unit that calculates a microphone array reference user angle that is an angle of a user position with reference to the microphone array based on the camera reference user distance, and the directivity angle control unit includes: The directivity angle of the microphone array is controlled to be the microphone array reference user angle.

  Thereby, in the voice input device capable of controlling the directivity angle, a recording person (a user who uses the voice input device, such as a person who is speaking at a different distance in the same direction as the speaker (user) to be voice acquisition target, Basically, when there is an unintended sound source of a person who is different from the above-mentioned speaker), it is possible to reduce the misrecognition of the user voice by determining the distance and inputting only the voice of the user at the target distance, User voice can be acquired with low noise.

  Further, the microphone array reference user angle calculation unit is configured to generate the microphone based on the predetermined distance, the camera reference user distance, the camera reference user angle, and an angle calculation error range corresponding to the camera reference user angle. An array reference user angle range may be calculated, and the directivity angle control unit may control the directivity angle of the microphone array so as to be within the microphone array reference user angle range. Since the microphone array reference user angle range is calculated according to the camera reference user angle, the directivity range of the microphone array can be appropriately controlled.

  Alternatively, the microphone array reference user angle calculation unit may calculate a distance calculation error according to the predetermined distance, the camera reference user angle, the camera reference user distance, the camera reference user angle, and / or the camera reference user distance. The microphone array reference user angle range may be calculated based on the range, and the directivity angle control unit may control the directivity angle of the microphone array so as to be within the microphone array reference user angle range. Since the microphone array reference user angle range is calculated according to the camera reference user distance, the directivity range of the microphone array can be appropriately controlled.

  Further, the face area of the subject in the image may be detected, and the camera reference user angle and / or the camera reference user distance may be calculated using the position of the face area as the user position. The detection of the face area is preferable as a process for detecting the user position because there is a mouth that speaks.

  An image display apparatus according to the present invention includes the above-described voice input device. Thereby, the image display apparatus provided with the audio | voice input apparatus which can control the directivity range of a microphone array appropriately can be provided.

D: Camera reference user distance, L: Distance between camera and microphone array (predetermined distance), θ ... Camera reference user angle, α ... Direction direction, 1 ... Audio input device, 11 ... Microphone array, 12 ... Camera, 13 ... In-image user coordinate detection unit, 14 ... camera reference user angle calculation unit, 15 ... camera reference user distance calculation unit, 16 ... microphone array reference user angle calculation unit, 17 ... directivity angle control unit, 21 ... superdirective microphone, 22 Reference numeral 23: Microphone array reference point, 23: Sound collection direction, 24: Microphone array front direction, 25 ... Microphone array axis, 31 ... Microphone, 32 ... Camera reference point, 40 ... Shooting range, 41 ... Camera reference point, 42 ... Camera light Axis, 43 ... Camera shooting direction, 44 ... Straight line, 51 ... Projection plane of camera, 52 ... User face coordinates in image, 61 ... User, 62 ... Another speaker, 91 ... Directional range Surrounding (directing area), 101 ... Angle calculation error range, 102 ... Distance calculation error range, 103 ... Common range, 104 ... Microphone array reference user angle range, 121 ... Voice recognition unit, 131 ... Control unit, 132 ... Image display unit .

In order to solve the above-mentioned problem, a first technical means of the present invention includes a microphone array having a plurality of microphones for acquiring sound, a camera having an image sensor and inputting an image by photographing, and being input by the camera. And a directivity angle control unit that controls a directivity angle of the microphone array based on the image, wherein the camera and the microphone array are arranged apart from each other by a predetermined distance, An apparatus includes: a camera reference user angle calculation unit that calculates a camera reference user angle that is an angle of a user position of a voice acquisition target user based on the camera based on the image; and the camera based on the image A camera reference user distance calculating unit that calculates a camera reference user distance that is a distance to the user position as a reference; the predetermined distance; and the camera reference user Angle, and on the basis of the camera reference user distance, and the microphone array reference user angle calculator for calculating a microphone array reference user angle is the angle of the user position with respect to the microphone array, further wherein the microphone array reference The user angle calculation unit is configured to determine a range of the microphone array reference user angle based on the predetermined distance, the camera reference user distance, the camera reference user angle, and an angle calculation error range according to the camera reference user angle. And the directivity angle control unit controls the directivity angle of the microphone array so as to be within the range of the microphone array reference user angle.

According to a second technical means of the present invention, in the first technical means, the microphone array reference user angle calculation unit calculates the range of the microphone array reference user angle using the camera reference user angle and / or the camera. It is characterized by being executed using a distance calculation error range corresponding to the reference user distance .

According to a third technical means of the present invention, there is provided a microphone array having a plurality of microphones for acquiring sound, a camera having an image sensor and inputting an image by photographing, and the microphone array based on the image input by the camera. A directivity angle control unit that controls the directivity angle of the camera, wherein the camera and the microphone array are spaced apart by a predetermined distance, and the speech input device is based on the image, A camera reference user angle calculation unit that calculates a camera reference user angle that is an angle of a user position of a voice acquisition target user with respect to the camera, and a distance to the user position with reference to the camera based on the image A camera reference user distance calculation unit that calculates a camera reference user distance, the predetermined distance, the camera reference user angle, and the camera reference Based on over The distance, and the microphone array reference user angle calculator for calculating a microphone array reference user angle is the angle of the user position with respect to the microphone array, further wherein the microphone array reference user angle calculation unit , said predetermined distance, said camera reference user angle, and the camera reference user distance, based on the distance calculation error range in accordance with the camera reference user angular and before Symbol camera reference user distance, the microphone array reference calculating a range of user angles, the directional angle controller is obtained by control means controls the directivity angle of the microphone array so that the range of the microphone array reference user angle.

Claims (5)

A microphone array having a plurality of microphones for acquiring sound;
A camera that has an image sensor and inputs an image by shooting;
A directivity angle control unit that controls the directivity angle of the microphone array based on the image input by the camera;
The camera and the microphone array are spaced apart by a predetermined distance;
The voice input device includes:
A camera reference user angle calculation unit that calculates a camera reference user angle that is an angle of a user position of a voice acquisition target user based on the camera based on the image;
A camera reference user distance calculation unit that calculates a camera reference user distance that is a distance to the user position with respect to the camera based on the image;
A microphone array reference user angle calculation unit that calculates a microphone array reference user angle that is an angle of a user position with respect to the microphone array, based on the predetermined distance, the camera reference user angle, and the camera reference user distance; Further comprising
The voice input device, wherein the directivity angle control unit controls the directivity angle of the microphone array so as to be the microphone array reference user angle. The microphone array reference user angle calculation unit is configured to calculate the microphone array reference based on the predetermined distance, the camera reference user distance, the camera reference user angle, and an angle calculation error range corresponding to the camera reference user angle. Calculate the user angle range,
The voice input device according to claim 1, wherein the directivity angle control unit controls the directivity angle of the microphone array so as to be in the microphone array reference user angle range. The microphone array reference user angle calculation unit includes the predetermined distance, the camera reference user angle, the camera reference user distance, the camera reference user angle and / or a distance calculation error range according to the camera reference user distance. Based on the microphone array reference user angle range,
The voice input device according to claim 1, wherein the directivity angle control unit controls the directivity angle of the microphone array so as to be in the microphone array reference user angle range.   4. The camera reference user angle and / or the camera reference user distance is calculated by detecting a face area of a subject in the image and using the position of the face area as the user position. The voice input device according to claim 1.   The image display apparatus provided with the audio | voice input apparatus of any one of Claims 1-4.

Images