JP2012029209A - Audio processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an audio processing system capable of automatically tracking a positional change of an audio source which needs attention; and continuously performing selectively needed audio processing on the audio source which has been paid attention.SOLUTION: An audio processing system includes a face movement detector 3 which detects a face of a figure from images of external world monitoring cameras 2-2; a binocular vision detector 7 which detects a visual line direction of a wearer from images of a left-eye monitoring camera 4 and a right-eye monitoring camera 5; a visual line target detector 8 which detects a direction and distance of a target object from the visual line direction; a positional information updating unit 9 which obtains positional information of the face from pieces of information of the face movement detector 3 and the visual line target detector 8; directivity adjusters 11-11which calculate a parameter to augment sound at a position corresponding to the positional information by beam-forming; and stereo correspondence directivity computing units 13-13which adjust left side gain and right side gain of input signals from an microphone array 12 in accordance with a relative position of a target audio source by using the result calculated by the directivity adjusters 11-11, and output the adjusted signals as left sound and right sound.

Description

  The present invention relates to a processing technique for sound source information, and more particularly, to a technique effective for processing audio or the like using a video processing technique.

  There are systems that perform beamforming using a plurality of microphones to emphasize and process a specific range of sounds. As this type of speech processing system, for example, only the sound source localized at a certain angle is extracted or adjusted (for example, refer to Patent Document 1), the direction of the sound source is stored and estimated, A device that performs processing based on estimation (see, for example, Patent Document 2) is known.

  In addition, as a device that emphasizes the sound source of the part of interest as an image, in a video camera having a line-of-sight position AF (Auto Focus) function that detects the line-of-sight position and focuses on the subject at the line-of-sight position, One having a function of matching the directivity of the sound collecting means is known (for example, see Patent Document 3).

JP 2009-10992 A JP 2005-354223 A JP-A-8-298609

  However, the present inventors have found that the above-described voice processing technology has the following problems.

  In the above-described technique, there is a problem in selectively following the position of a sound source that needs attention and selectively performing sound processing. For example, in the technique disclosed in Patent Document 3, it is necessary for a human to keep an eye on a video to be noticed at all times, and it becomes difficult to adjust directivity when it is out of the angle of view of the video camera. There's a problem.

  Further, in the technique disclosed in Patent Document 2, it is difficult to accurately track the sound source position because estimation is included, and tracking is extremely difficult, particularly when the silent state of the object continues for a long time. End up.

  Furthermore, in the case of the technique disclosed in Patent Document 1, although it is an effective technique for selectively performing speech processing, there is a problem that it is impossible to follow the sound source position.

  An object of the present invention is to provide a technique capable of automatically following a change in the position of a sound source that needs attention and selectively continuing necessary sound processing for the sound source of interest.

  The above and other objects and novel features of the present invention will be apparent from the description of this specification and the accompanying drawings.

  Of the inventions disclosed in the present application, the outline of typical ones will be briefly described as follows.

  The present invention includes a sound source acquisition unit that acquires sound from a sound source of an object, a line-of-sight monitoring detection unit that detects the object by monitoring the user's line of sight, and detects the direction and distance of the object, A movement detection unit that detects an object from video captured by the camera, detects movement of the object from continuity of image position change, and outputs the movement information, and the movement detection And a position information detection unit that detects the position information by tracking the object from the detection results detected by the line-of-sight monitoring detection unit, and the position information detected by the position information detection unit, A directional sound source adjustment unit that outputs a sound signal by adjusting directivity of the sound source acquisition unit so that sound at a position corresponding to the position information is enhanced, and a sound signal output from the directional sound source adjustment unit. With a sound output section to output Than is.

  Further, according to the present invention, the line-of-sight monitoring and detecting unit monitors a user's line of sight, and from the monitoring result of the monitoring camera unit, the line of sight of each of the left eye and the right eye from the black eye portion and the cornea reflection image A line-of-sight detection unit that detects a direction, and obtains an intermediate point coordinate of a position where a straight line corresponding to the line of sight closest in space from the line-of-sight information detected by the line-of-sight detection unit, and the direction of the object the user is looking at A line-of-sight object detection unit that detects a distance, and the position information detection unit detects the object when the line-of-sight remains for an arbitrary time or longer from the detection results of the movement detection unit and the line-of-sight detection unit. The position information is output as a position to be noted, and based on the movement information detected by the movement detection unit, the position information update unit that updates the position information of the object, and the position information output from the position information update unit Storing location information It is obtained and a storage unit.

  Furthermore, according to the present invention, when the position information update unit detects position information and determines that the object is separated by an arbitrary distance or more, the position information update unit excludes the object from the object.

  Further, according to the present invention, the movement detection unit detects a person's face from the video captured by the camera, detects a face feature amount, determines the same person based on the feature amount, and determines an image position. The movement of the face is detected from the continuity of the change, and the sound source acquisition unit acquires sound based on the control of the directional sound source adjustment unit.

  Furthermore, in the present invention, the sound source acquisition unit includes a plurality of microphones.

  According to the present invention, the sound output unit includes a noise canceling unit that generates a signal for canceling noise and performs noise cancellation.

  Furthermore, the outline | summary of the other invention of this application is shown briefly.

  An image in a direction in which a sound source can exist is monitored, and a portion that may be a gaze target is marked as a candidate point. When an event occurs for a portion marked as a candidate point or at the marked portion, the position is stored to handle the mark as a target sound source. The movement of the image corresponding to the position is monitored by image processing, and the position information is updated as appropriate. The beam forming characteristics of a plurality of microphones are adjusted in accordance with this position information, and necessary processing is performed on the sound collected and extracted by beam forming.

  The present invention is also input from a display unit that displays video, a sound output unit that outputs sound, a camera that monitors the surroundings, a sound source acquisition unit that acquires sound from a sound source of an object, and the camera A face movement detection unit that extracts a human face from an image, detects the position and distance of the extracted face, tracks the face of the person, and outputs it as position information; an image of the camera; and the face movement detection Based on the positional information output by the unit, the eye part is extracted from the detected face image, and the specific display that is the place where the direction in which the person is viewed is displayed on the display unit is to be performed A gaze state detection unit that determines whether the person is a part, a position information update unit that acquires position information of a corresponding person when the gaze state detection unit determines that the person is looking at the specific display unit, The position acquired by the position information update unit Based on the information, the sound source acquisition unit calculates a parameter for enhancing the sound corresponding to the position of the person by beam forming, and based on the calculation result, the sound at the position corresponding to the position information is enhanced. A directivity sound source adjustment unit that adjusts the directivity of the sound and outputs a sound signal, and recognizes the voice included in the sound signal output from the directivity sound source adjustment unit, and based on the recognition result, the target of bidirectional communication When the target person is determined, the voice recognition display control unit that displays information on the display unit based on the position information corresponding to the person and the voice recognition display control unit output the information. Based on the position information, a directivity calculation process is performed to provide directivity so that sound is transmitted only to the target person, and sound is output from the sound output unit based on the calculation result Those having a tropism adjustment computing unit.

  Further, according to the present invention, the position information update unit changes the position information of the person's face whenever the gaze state detection unit determines that the person is gazing at the specific display unit. The location information is updated and tracked.

  Further, according to the present invention, the voice recognition display control unit displays small screen video information on the display unit based on position information of a person to be interactively communicated.

  Further, according to the present invention, the position information update unit acquires position information of a plurality of persons, and tracks and updates the position information of the plurality of persons.

  Among the inventions disclosed in the present application, effects obtained by typical ones will be briefly described as follows.

  (1) It is possible to selectively amplify the voice of a person who needs to be heard in an environment where the mutual positional relationship between the user and the sound source changes every moment.

  (2) According to the above (1), it is possible to assist optimal conversation even in a situation of noise.

  (3) A digital signage device capable of two-way communication can be realized by combining a beam-forming speaker.

  (4) According to the above (3), in a situation where a conversation is performed by a plurality of people, it is possible to pick up the voice of the person who has paid attention even when zooming on a specific person.

It is a block diagram which shows an example of a structure in the speech processing system by Embodiment 1 of this invention. It is explanatory drawing which shows an example of the external appearance of the speech processing system of FIG. It is explanatory drawing which shows an example of the monitoring area | region by the external appearance monitoring camera shown to the audio | voice processing system of FIG. It is a block diagram which shows an example of a structure in the digital signage system by Embodiment 2 of this invention. It is explanatory drawing which shows an example of the external appearance of the digital signage system of FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that components having the same function are denoted by the same reference symbols throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted.

(Embodiment 1)
1 is a block diagram showing an example of the configuration of a voice processing system according to Embodiment 1 of the present invention, FIG. 2 is an explanatory diagram showing an example of the appearance of the voice processing system of FIG. 1, and FIG. 3 is a diagram of FIG. It is explanatory drawing which shows an example of the monitoring area | region by the external appearance monitoring camera shown to the audio | voice processing system.

  In the first embodiment, the voice processing system 1 is a system that selectively emphasizes the voice of the conversation partner during the conversation and transmits the voice to the wearer of the voice processing system 1.

As shown in FIG. 1, an audio processing system 1 which is a sound processing system includes an external monitoring camera 2 ₁ to 2 ₈ , a face movement detector 3, a left eye monitoring camera 4, a right eye monitoring camera 5, and a near infrared LED 6 (see FIG. 1). 2), a binocular gaze detector 7, a gaze target detector 8, a position information updater 9, a position information storage unit 10, directivity adjusters 11 _{1 to} 11 ₈ , and a microphone array 12 composed of a plurality of microphones. Stereo-compatible directivity calculators 13 _{1 to} 13 ₈ , adders 14 and 15, left ear earphone 16, right ear earphone 17, left ear microphone 18, right ear microphone 19, left noise cancellation device 20, and right noise cancellation device 21.

  The left eye monitoring camera 4, right eye monitoring camera 5, binocular line of sight detector 7, and line of sight target detector 8 constitute a line of sight monitoring detector, and a position information updater 9 and a position information storage unit 10 The position information detection unit is configured by.

Further, the directivity adjusters 11 _{1 to} 11 ₈ and the stereo-compatible directivity calculators 13 _{1 to} 13 ₈ constitute a directivity sound source adjustment unit, and the left ear microphone 18, the right ear microphone 19, and the left noise cancellation device. 20 and the right noise canceling device 21 constitute a noise canceling unit.

The external monitoring cameras 2 _{1 to} 2 ₈ are cameras for monitoring the surroundings, and are a group of cameras for monitoring at which position in the surroundings there is an object that can be a sound source and where the focused sound source is moving. .

Images taken by the external monitoring cameras 2 _{1 to} 2 ₈ are output to the face movement detector 3 to detect a human face. The face movement detector 3 has a function of detecting a person's face from the input video and a function of detecting a feature amount of the detected face. The face movement detector 3 determines the same person based on the feature amount and changes the image position. The movement of the face is detected from the continuity of.

  Description of the face detection algorithm is omitted because it is already implemented in many digital cameras and the like, but, for example, a method using an AdaBoost algorithm using Haar type feature values is widely known.

Further, by photographing the same face with a plurality of external monitoring cameras 2 ₁ to 2 _8, the feature amount, and using the positional relationship of the face image in the captured image in the external monitoring camera 2 ₁ to 2 _8, a plurality of external monitoring by associating the same face image between the images captured by the camera 2 ₁ to 2 _8, the face motion detector 3, the distance to the face, not only in the direction of the face can also be detected.

  The left-eye monitoring camera 4 that is the monitoring camera unit is a near-infrared camera that monitors the line of sight of the left eye of the wearer of the audio processing system 1, and the right-eye monitoring camera 5 that is the monitoring camera unit is the right of the device wearer. This is a near-infrared camera that monitors the line of sight of the eyes.

  Further, near-infrared LEDs 6 (shown in FIG. 2) are provided in the vicinity of the left-eye monitoring camera 4 and the right-eye monitoring camera 5. Images taken by the left eye monitoring camera 4 and the right eye monitoring camera 5 are binarized by the binocular gaze detector 7 which is a gaze detection unit, and the left eye of the wearer is detected from the black eye portion and the cornea reflection image. The line-of-sight directions of the eyes and right eye are detected.

  The line-of-sight target detector 8 serving as the line-of-sight target detection unit obtains the intermediate point coordinates of the position where the straight line corresponding to the line of sight of both eyes is closest in space based on the line-of-sight information detected by the binocular line-of-sight detector 7. Detect the direction and distance of the object the wearer is looking at.

  The position information update unit 9 serving as a position information update unit detects the position of the face image when the line of sight stays at a place where the face exists for an arbitrary time or longer from the outputs of the face movement detector 3 and the line-of-sight target detector 8. Information is stored in the position information storage unit 10 serving as a position information update unit as a position to be noted.

  At this time, when the number of attentions is larger than the number that can be stored in the position information storage unit 10, the position information of the face image that has been noticed most recently by the line of sight is erased and replaced with the position information of the newly noticed face image. .

  When a face image is discriminated as a position to be noticed, depending on the application, there may be a method of using a distance to the face as a target of attention when the distance is not more than a certain distance. For example, if it is an application centered on normal conversation, it is possible to target about 2 meters or 3 meters or less.

  The position information updater 9 also has a function of updating the position information of the target of interest stored in the position information storage 10 based on the face movement information detected by the face movement detector 3. With this function, not only the movement of the focused face image but also the relative position information of the focused face image with the wearer can be correctly updated even when the wearer moves or moves the face. .

Each position information corresponding to the face image of interest stored in the position information storage 10 is assigned to _{one of the} directivity adjusters 11 _{1 to} 11 ₈ . Of the directivity adjusters 11 _{1 to} 11 ₈ , the directivity adjuster to which position information is assigned is for enhancing the sound at the position corresponding to the position information by beam forming using the microphone array 12 serving as a sound source acquisition unit. Calculate the necessary digital filter parameters.

  As a method for obtaining these coefficients, there is a method of delaying the input signal from each microphone so as to compensate for the difference between the position of the sound source and the distance to each microphone constituting the microphone array 12, and calculating the total. is there.

Among the stereo-compatible directivity calculators 13 _{1 to} 13 ₈ , the stereo-compatible directivity calculator corresponding to the directivity adjuster that has calculated the digital filter parameters is based on the calculation result of the directivity adjuster, and the microphone array 12. For example, a digital audio signal input by A / D (Analog / Digital) conversion is filtered from each microphone constituting the.

In addition, in order to reproduce the sense of localization according to the direction of the sound source, that is, the face image, to the wearer, the stereo correspondence directivity calculators 13 _{1 to} 13 ₈ are respectively provided on the left side and the right side according to the relative position of the target sound source. The gain is adjusted and output as left audio and right audio.

Audio output signals from the stereo-compatible directivity calculators 13 _{1 to} 13 ₈ are added by the adders 14 and 15 respectively to the left side and the right side, and transmitted to the wearer's ear through the left ear earphone 16 and the right ear earphone 17.

  The left ear microphone 18 is a microphone for canceling the noise of the left side sound output from the left ear earphone 16, and the right ear microphone 19 is a microphone for canceling the noise of the right side sound output from the right ear earphone 17.

  The left ear microphone 18 inputs a sound that directly reaches the left ear from the outside, generates a signal (basically an antiphase signal) that cancels this sound by the left noise canceling device 20, and reproduces it by the left ear earphone 16. The gain is adjusted to the sound and added by the adder 14. Similar processing is performed on the right side. By this processing, noise that directly reaches the ear from the outside can be suppressed, and the sound generated inside the device can be easily heard even under noise.

  Note that the left noise cancellation device 20 and the right noise cancellation device 21 do not generate a signal to cancel so as not to cancel a sound in a specific frequency band in order to prevent a necessary external sound from being completely blocked. In some cases. In a state where there is no notable sound source, the operation of the left noise canceling device 20 and the right noise canceling device 21 is stopped, or a signal in the same phase as the left ear microphone 18 and the right ear microphone 19 is generated. It is also possible to change and reproduce a state close to non-wearing.

  FIG. 2 is an explanatory diagram showing an example of an external configuration in the voice processing system 1.

As shown in the figure, the sound processing system 1 is composed of, for example, a glasses-type device provided in the glasses 22. Microphone array 12 is comprised of a microphone 12 ₁ to 12 _10. The microphones 12 _{1 to} 12 ₃ are provided at equal intervals from the ear fitting part side to the lens side in the left temple part of the glasses 22.

The microphones 12 ₄ and 12 ₅ are provided at equal intervals above the left lens or above the left lens edge, and the microphones 12 ₆ and 12 ₇ are respectively above the right lens or above the right lens edge. It is provided at equal intervals.

The microphones 12 _{8 to} 12 ₁₀ are provided at equal intervals from the lens side to the earrest side of the temple portion on the right side of the glasses 22. The microphones 12 _{1 to} 12 ₃ mainly collect the left sound. The microphones 12 _{4 to} 12 ₇ mainly collect the sound on the front surface, and the microphones 12 _{8 to} 12 ₁₀ mainly collect the sound on the right side. The left and front microphones 12 _{1 to} 12 ₅ are used to collect left front sound, and the right and front microphones 12 _{6 to} 12 ₁₀ are used to collect right front sound.

However, the microphone that is in the shadow of the wearer when viewed from the sound source is not used, such as the right microphone signal is not used to collect the left sound source. This control can be realized by generating parameters according to the positions to which the directivity adjusters 11 _{1 to} 11 ₈ are paying attention, and operating the stereo correspondence directivity calculators 13 _{1 to} 13 ₈ according to the parameters.

  That is, by generating a parameter that makes the gain from the unused microphone zero, the input from the unused microphone is masked. By not using the microphone that is in the shadow of the wearer, the sound input reflected mainly from the sound reflected from the outside world, the device itself, and the sound transmitted through the device user is filtered rather than the direct sound from the sound source. It can be excluded from processing, and adverse effects on directivity calculation processing can be suppressed.

External monitoring camera 2 ₁ is provided on the earmuffs portion side of the temple part of the left eyeglass 22, external monitoring camera 2 ₂ are provided on the lens side of the temple part of the left eyeglass 22.

The outside world monitoring camera ₂₅ is provided on the ear contact side of the right temple part of the glasses 22, and the outside world monitoring camera ₂₆ is provided on the lens side of the right temple part of the glasses 22.

External monitoring camera 2 ₇ is provided at an end portion of the earmuff portion side of the temple part of the left eyeglass 22, external monitoring camera 2 _8, the end portions of the earmuff portion side in the right temple part of the eyeglass 22 Is provided.

The external world monitoring cameras 2 ₇ and 2 ₈ are arranged with their arms slightly extended so that they are provided behind the earpiece so that they are located slightly behind the wearer. By arranging in this way, it is possible to avoid the influence of shadows due to the back of the head of the wearer as much as possible, and monitoring similar to the front can be made possible.

External monitoring camera 2 _3, upper left of the left lens or is provided on the upper left side of the left lens edge, external monitoring camera 2 ₄ provided on the upper right or upper right of the right lens edge, the right lens It has been.

The outside world monitoring cameras 2 ₁ and 2 ₃ are image monitoring in the left direction, the outside world monitoring cameras 2 ₂ and 2 ₅ are image monitoring in the front direction, the outside world monitoring cameras 2 ₇ and 2 ₈ are image monitoring in the backward direction, and the outside world monitoring camera. 2 ₄ and 2 ₆ perform image monitoring in the right direction. By monitoring with two cameras in each direction, the distance to the object can also be detected.

  The left ear earphone 16 is connected to the left temple portion of the glasses 22 via a wire, and the right ear earphone 17 is connected to the right temple portion of the glasses 22 with a wire. The left ear earphone 16 has a structure including a left ear microphone 18 for noise canceling, and the right ear earphone 17 has a structure including a right ear microphone 19 for noise canceling.

  The left eye monitoring camera 4 is provided below the left lens of the glasses 22 or below the left lens edge, and the right eye monitoring camera 5 is provided below the right lens of the glasses 22 or below the right lens edge. Yes. Near the left-eye monitoring camera 4 and the right-eye monitoring camera 5 are provided near-infrared LEDs 6 serving as photographing light sources.

In the glasses 22, a left storage box 23 is provided on the lens side of the left temple, and a right storage box 24 is provided on the lens side of the right temple. The left storage box 23 and the right storage box 24 include a face movement detector 3, a binocular gaze detector 7, a gaze target detector 8, a position information updater 9, a position information storage unit 10, and a directivity adjuster 11 _1. To 11 ₈ , stereo-compatible directivity calculators 13 _{1 to} 13 ₈ , adders 14 and 15, left noise canceling device 20, right noise canceling device 21, and other circuit blocks and a battery serving as a power source for operating the sound processing system 1. Etc. are stored.

  Further, the left storage box 23 or the right storage box 24 may have a configuration in which, for example, a wireless communication circuit is mounted and a part of processing is performed by a wireless connection.

FIG. 3 is an explanatory diagram showing an example of a region that can be monitored by the external monitoring cameras 2 ₁ to 2 ₈ in the voice processing system 1 of FIG. In FIG. 3, the front half region is mainly shown, and the rear half is symmetrical with the front half, and therefore the illustration of the region is partially omitted.

In FIG. 3, an area A is an area that cannot be monitored by any of the left-side external monitoring cameras 2 ₁ and 2 ₃ , and an area B is monitored by either of the right-side external monitoring cameras 2 ₄ and 2 _6. The area C cannot be monitored by any of the forward external monitoring cameras 2 ₂ and 2 ₅ , and the area D cannot be monitored by any of the external monitoring cameras 2 ₇ and 2 _{8 in} the backward direction. Show.

The region E is monitored by the external monitoring camera 2 _3, region F is outside the monitoring camera 2 ₂ According monitored region G is monitored by external monitoring camera 2 _5, and region H is monitored by external monitoring camera 2 ₄ This is a region that can only be monitored and is a region that cannot be monitored by two external monitoring cameras.

The area I is an area that can be monitored by two cameras, the external monitoring cameras 2 ₂ and 2 ₃ , and information including the distance can be acquired from the monitoring image using cameras having different directions. The area J is an area that can be monitored by two cameras, the external monitoring cameras 2 ₄ and 2 ₅ , and similarly, information including the distance can be acquired from the monitoring image. In other words, the direction and distance of the object may be detected using the signals of both the front external monitoring cameras 2 ₃ and 2 ₄ and the left or right external monitoring cameras 2 ₂ and 2 ₅ .

Furthermore, the region K is capable of prior external monitoring camera 2 ₂ for directions, and monitoring with two cameras of external monitoring camera 2 ₃ for the left direction, region L is outside the monitoring camera before a direction 2 _5, and it is possible to monitor with two cameras of external monitoring camera 2 ₄ for the right direction.

  That is, the region K and the region L are used as connection regions in the left and right directions and the front direction, both face detection is performed simultaneously, the same person is associated from the positional relationship and the feature amount, and the camera used for monitoring is appropriately selected. By changing, it is possible to continuously monitor the entire circumference of the wearer including movement.

  Accordingly, in the first embodiment, only the voice of the conversation partner focused on by the wearer can be selectively amplified and transmitted to the wearer with emphasis. However, it is possible to have a conversation without disturbing the conversation.

  In the first embodiment, since it is assumed that the target is human conversation, the face movement detector 3 is used. However, a detector that detects different objects depending on the purpose of the device is used. It can also be used.

  For example, in birdwatching applications, a bird's cry can be selectively emphasized by using a detector that detects bird images. It is also conceivable to use a detector that can detect both human faces and birds.

  In the first embodiment, the human hearing is assisted. However, for example, the video captured by the forward camera can be recorded, and the output of the stereo-compatible directivity calculator is changed to the normal left and right. By adding the sound captured by the microphone and recording, even if the sound is not recorded as a video, if it is recorded as a target of attention once, the sound at that position is recorded with a loud sound It is possible.

In this case, in order to allow play stereoscopic image at the time of reproduction, and records eye images taken at ambient monitoring camera 2 ₂ image, the image photographed at ambient surveillance camera 2 ₅ as the right-eye image. However, when it is desired to record only one of the images in order to suppress the recording amount, the device user can select the recording target from either the left eye image or the right eye image. With this function, the wearer can select an image to be recorded in accordance with his / her effectiveness.

Furthermore, in the first embodiment, a configuration in which _eight external monitoring cameras 2 ₁ to 2 ₈ , directivity adjusters 11 _{1 to} 11 ₈ , and _eight stereo-compatible directivity calculators 13 _{1 to} 13 ₈ are provided. However, the number of these circuit blocks is not limited to this.

(Embodiment 2)
FIG. 4 is a block diagram showing an example of the configuration of the digital signage system according to Embodiment 2 of the present invention, and FIG. 5 is an explanatory diagram showing an example of the appearance of the digital signage system of FIG.

  In the first embodiment, the digital signage system 25 is an electronic signboard that displays video and information on a display installed in a public facility or store.

As shown in FIG. 4, the digital signage system 25 includes an external monitoring camera 2 a, 2 b, a face movement detector 3, a position information updater 9, a position information storage device 10, a directivity adjuster 11 _{1 to} 11 ₈ , a microphone 12. ₁ to microphones 12 ₁₄ (shown in FIG. 5), the same circuit block as that of FIG. 1 of the first embodiment including the microphone array 12 includes directivity calculators 26 _{1 to} 26 ₈ , speech recognizers 27 ₁ to 27 ₈ , overall control device 28, display control device 29, display device 30, pronunciation directivity adjustment calculators 31 _{1 to} 31 ₈ , speech synthesizers 32 _{1 to} 32 ₈ , a plurality of speakers 33 _{1 to} 33 ₁₈ (see FIG. 5) In this configuration, a speaker array 33, a gaze state detector 34, and an adder 35 are provided.

  The external monitoring cameras 2a and 2b are cameras that monitor the surroundings of the digital signage system 25. The cameras are used to monitor the position around the digital signage system 25 that can be a sound source and to track the person who is paying attention. It is.

  The face movement detector 3 extracts a person's face from each image input from the external monitoring cameras 2a and 2b, and detects the feature amount. The face portions extracted from the images captured by the external monitoring cameras 2a and 2b are associated with each other based on the feature amount and the position information in each image.

  By using the result of the association, not only the position of the person's face but also the distance to the face is detected from the difference in position in each of the plurality of images. Further, it is possible to track the movement of the face once detected until it moves out of the monitorable area based on the motion information of the image and the feature amount of the face image.

  The monitoring image of the external monitoring camera 2a (or external monitoring camera 2b), and the face image position information extracted from the monitoring image of the external monitoring camera 2a (or external monitoring camera 2b) by the face movement detector 3 and its identification number Is sent to the gaze state detector 34.

  The gaze state detector 34 extracts and analyzes the eye portion from the detected face image, and detects the direction in which the person is looking. When the portion being viewed is a specific image presented on the display device 30 of the digital signage system 25, that is, a portion recorded so as to be processed in the case where attention is paid in advance in the image The identification number of the person's face is sent to the position information updater 9.

  When the position information updater 9 receives the identification number of the face being watched from the gaze state detector 34, if the position information corresponding to the identification number is still unstored, it stores it in the position information storage unit 10. .

  Of the position information stored in the position information storage device 10, the face of the person who has become difficult to be tracked by the external monitoring cameras 2 a and 2 b is erased from the memory, the gaze state is released, and The location information is also deleted.

  When the position information becomes larger than the number that can be stored in the position information storage device 10, the position information on the face of the person whose gaze state is quickly released is deleted, and the position information of the face of the person who is newly focused Replace with.

  However, if it is difficult to delete the position information, the addition of new position information is temporarily stopped and not performed. This is because the position information of the person who is already responding to the digital signage system 25 is not lost.

  Depending on the use of the digital signage, the distance to the face determined to be paying attention to the digital signage system 25 can be limited to a certain distance or less. That is, based on the distance information to the face image obtained by the face movement detector 3, it is determined whether or not the digital signage system 25 is in a corresponding range. Position information corresponding to the gaze state detector 34 is sent, and a person who is paying attention to the digital signage system 25 is extracted.

  Further, the position information updater 9 updates the position information stored in the position information storage unit 10 based on the face movement information detected by the face movement detector 3. Thereby, even when the face of a person who is paying attention to the digital signage system 25 moves, the position information of the face can be correctly updated.

Position information corresponding to the face image of interest stored in the position information storage 10 is assigned to _{one of the} directivity adjusters 11 _{1 to} 11 ₈ . Among the directivity adjusters 11 _{1 to} 11 ₈ , the directivity adjuster to which position information is input is a digital filter necessary for enhancing the sound at the position corresponding to the position information by beam forming using the microphone array 12. These parameters are calculated using the same method as in the first embodiment.

  However, since the arrangement and number of microphones constituting the microphone array 12 are different from those of the first embodiment, the specific calculation method needs to be changed accordingly.

The calculation result of the directivity adjuster that has calculated the parameter is input to the subsequent directivity adjuster corresponding to the directivity adjuster that has calculated the parameter among the directivity adjusters 26 _{1 to} 26 ₈ . Of the directivity adjusters 26 _{1 to} 26 ₈ , the directivity adjuster to which the calculation result of the parameter is input is a digital voice that is A / D converted and input from each microphone constituting the microphone array 12 using the parameter. Perform signal filtering.

The voice output signal from the directivity calculator that has been subjected to the filter processing is sent to any one of the voice recognizers 27 _{1 to} 27 ₈ . The speech recognizer to which the filtered speech signal is input recognizes speech included in the speech signal.

Depending on the application, the functions and performances of all the speech recognizers 27 _{1 to} 27 ₈ do not have to be targets. If the digital signage system 25 can simultaneously communicate with k people, k out of n Only a single speech recognizer is considered to have a high function and high performance, and other speech recognizers may have a low function configuration that can only detect the first talk to the digital signage system.

  Since the position information storage 10 manages which position information stored in the digital signage system currently corresponds to the position information of the face of the person that is the target of two-way communication, the position information is highly functional and high performance. Control is performed so as to be sent to the directivity adjuster of the directivity calculator sent to the speech recognizer.

  The result recognized by the speech recognizer is sent to the overall control device 28. The overall control device 28 detects a word corresponding to the first talk, for example, "I'm sorry", "Tell me", or the like, while the digital signage system 25 has a margin for new two-way communication. In the image displayed on the screen 30, when attention is paid in advance, if a gaze is detected for a part that is recorded so that some processing should be performed, a two-way communication state with the person is entered. .

  When the two-way communication state is entered, a flag indicating that it is a target for two-way communication is set for the position information corresponding to the person recorded in the position information storage unit 10.

  If necessary for two-way communication, the overall control device 28 sets the display control device 29 and assigns an arbitrary child to the display device 30 in order to appropriately assign the child screens 30a, 30b, 30c (shown in FIG. 5). One of the screens 30a, 30b, and 30c is displayed. Information is displayed for the two-way communication target using this sub-screen.

Position information flag as a two-way communication target is set, is sent to the sound directivity adjustment calculator 31 _{1-31 8.} These sound directivity adjustment calculator 31 _{1-31 8,} the speech synthesized in the speech synthesizer 321 _to 323 ₈ to be heard well only in certain locations, directivity adjustment for driving the speaker array 33 Perform the operation.

  That is, from the positional relationship between the position of each speaker to be driven in the speaker array 33 and the face of the person who is the object of bidirectional communication, the delay is adjusted so that the generated sound is in phase near the position of the person who is the object of bidirectional communication, An operation for reducing the gain is performed in consideration of being driven by a plurality of speakers.

Pronunciation directivity adjustment calculator 31 _{1-31 8} operation result of the computed directivity adjustment by are summed by the adder 35, and output speaker array 33.

  Note that the speaker array 33 may be controlled by other methods as long as a sound that can be heard well only near the face of the bidirectional target person can be generated. In addition, a method of using a super-directional speaker that uses ultrasonic waves as a carrier wave instead of the speaker array and mechanically adjusting the direction of the speaker may be considered.

  FIG. 5 is an explanatory diagram showing an example of the appearance of the digital signage system 25.

The digital signage system 25 is provided with a display device 30 at the center, and microphones 12 _{1 to} 12 ₃ constituting the microphone array 12 are provided on the left side of the display device 30 from below to above.

Further, microphones 12 _{4 to} 12 ₁₁ are provided on the upper portion of the display device 30 from left to right, respectively, and microphones 12 _{12 to} 12 ₁₄ are provided on the right side of the display device 30 from the upper side to the lower side, respectively. It has been.

Below the display device 30, speakers 33 _{1 to} 33 ₁₈ constituting the speaker array 33 are arranged in two upper and lower stages. External monitor cameras 2a and 2b are provided between the microphone 12 ₅ and the microphone 12 ₆ and between the microphone 12 ₉ and the microphone 12 ₁₀ , respectively.

  When the actual installation of the digital signage system 25 is taken into consideration, the positional relationship between the digital signage system 25 and the person changes only by a height difference of the person in the vertical direction, but in the horizontal direction, the position of the person who is the object of bidirectional communication is changed. Because it depends on the position, more microphones are arranged in the horizontal direction than in the vertical direction so that the directivity in the horizontal direction can be further improved.

  In addition, the outside world surveillance cameras 2a and 2b are arranged on the upper part of the digital signage system 25 to perform monitoring from a high position as much as possible, and another person crosses in front of the person who is the object of bidirectional communication. Even in this case, the image tracking necessary for two-way communication is not hindered.

  However, even if it is actually a shadow of another person and it becomes difficult to track the image with the two external monitoring cameras 2a and 2b, if one external monitoring camera 2 can track the image, that camera If both the external monitoring cameras 2a and 2b become unable to track, the movement prediction is temporarily performed. When the image is captured again, the movement prediction information and the image characteristics are recorded. Based on the amount, an attempt is made to associate with a position recorded in the position information storage unit 10, and if association is possible, tracking is resumed. If the association fails, the two-way communication corresponding to the person is stopped.

The speakers 33 _{1 to} 33 ₁₈ constituting the speaker array 33 are configured in a two-stage configuration in the vertical direction and a large number in the horizontal direction in consideration of the positional relationship between the digital signage system 25 and the person, like the microphones 12 _{1 to} 12 _14. The speaker is arranged.

  Further, the display device 30 can display a small screen 30a, a small screen 30b, and a small screen 30c. These sub-screens 30a, 30b, and 30c are used for presenting information that needs to be individually handled to a two-way communication person.

  The sub-screens 30a, 30b, and 30c may be deleted when they are unnecessary, or may be always displayed, and other information may be presented when not used for two-way communication. If you have any questions, please look at this part and speak. "

  As a result, in the second embodiment, the digital signage system 25 extracts sounds in the vicinity of the target person for two-way communication and outputs sounds that can be heard well only in the vicinity of the target person. Even in many places, it is possible not only to simply display an image but also to provide more active guidance to a person who has focused on the image and to respond to requests and questions from the person. Also, a plurality of two-way communications can be realized with one digital signage system 25.

  As mentioned above, the invention made by the present inventor has been specifically described based on the embodiment. However, the present invention is not limited to the embodiment, and various modifications can be made without departing from the scope of the invention. Needless to say.

  The present invention is suitable for a voice processing technique that emphasizes the voice of a conversation partner in selection.

1 speech processing system 2 ₁ to 2 ₈ external surveillance cameras 2a external surveillance camera 2b external monitoring camera 3 faces motion detector 4 the left-eye monitoring camera 5 right-eye monitoring camera 6 near infrared LED
7 binocular gaze detector 8 gaze target detector 9 position information updater 10 position information storage unit 11 _{1 to} 11 ₁₈ directivity adjuster 12 microphone array 12 _{1 to} 12 ₁₄ microphone 13 _{1 to} 13 ₈ stereo-compatible directivity calculator 14, 15 Adder 16 Left ear earphone 17 Right ear earphone 18 Left ear microphone 19 Right ear microphone 20 Left noise cancellation device 21 Right noise cancellation device 22 Glasses 23 Left storage box 24 Right storage box 25 Digital signage system 26 _{1 to} 26 ₈ directional calculator 27 _{1-27 8} speech recognizer 28 total control unit 29 display control unit 30 display unit 30a small picture 30b small picture 30c small pictures 31 _{1-31 8} pronunciation directivity adjustment operator 32 _{1-32 8} speech synthesis Device 33 speaker array 33 _{1 to} 33 ₁₈ speaker 34 gaze state detector 35 adder

Claims (10)

A sound source acquisition unit for acquiring sound from the sound source of the object;
A line-of-sight monitoring detection unit that detects the object by monitoring the user's line of sight, and detects the direction and distance of the object;
A camera that monitors the surroundings,
A movement detection unit that detects an object from video captured by the camera, detects movement of the object from continuity of changes in image position, and outputs movement information;
From the respective detection results detected by the movement detection unit and the line-of-sight monitoring detection unit, a position information detection unit that detects the position information by tracking the object;
A directional sound source adjustment unit that adjusts the directivity of the sound source acquisition unit and outputs a sound signal based on the position information detected by the position information detection unit so that sound at a position corresponding to the position information is enhanced. When,
A sound processing system comprising: a sound output unit that outputs a sound signal output from the directional sound source adjustment unit. The sound processing system according to claim 1,
The line-of-sight monitoring detector
A surveillance camera unit for monitoring the user's line of sight;
From the monitoring result of the monitoring camera unit, a gaze detection unit that detects the gaze direction of the left eye and the right eye from the black eye part and the cornea reflection image, and
A line-of-sight target detection unit for detecting a direction point and a distance of an object viewed by a user, by obtaining an intermediate point coordinate of a position where a straight line corresponding to the line of sight closest in space from the line-of-sight information detected by the line-of-sight detection unit; With
The position information detector
From the detection results of the movement detection unit and the line-of-sight detection unit, when the line of sight stays for an arbitrary time or longer, the position information of the object is output as a position to be noticed, and the movement detection unit detects A position information update unit that updates the position information of the object based on movement information;
A sound processing system comprising: a position information storage unit that stores position information output from the position information update unit. The sound processing system according to claim 1 or 2,
The location information update unit
When the position information is detected, if it is determined that the object is more than an arbitrary distance, the sound processing system is excluded from the object. The sound processing system according to any one of claims 1 to 3,
The movement detector is
Detecting the face of a person from the video taken by the camera to detect the feature amount of the face, determining the same person based on the feature amount, detecting the movement of the face from the continuity of image position changes,
The sound source acquisition unit
A sound processing system that acquires sound based on control of the directional sound source adjustment unit. The sound processing system according to any one of claims 1 to 4,
The sound source acquisition unit
A sound processing system comprising a plurality of microphones. The sound processing system according to any one of claims 1 to 5,
The sound output unit is
A sound processing system comprising a noise canceling unit that generates a signal that cancels noise and cancels noise. A display unit for displaying images;
A sound output unit for outputting sound;
A camera that monitors the surroundings,
A sound source acquisition unit for acquiring sound;
A face movement detection unit that extracts the face of a person from an image input from the camera, detects the position and distance of the extracted face, tracks the face of the person, and outputs the position information;
Based on the image of the camera and the position information output by the face movement detection unit, an eye part is extracted from the detected face image, and the viewing direction of the person is displayed on the display unit. A gaze state detection unit that determines whether the specific display unit is a place to perform some processing;
A position information update unit that acquires position information of a corresponding person when the gaze state detection unit determines that the person is looking at the specific display unit;
Based on the position information acquired by the position information update unit, a parameter for enhancing the sound corresponding to the position of the person by beam forming is calculated, and based on the calculation result, the sound at the position corresponding to the position information is calculated. A directional sound source adjustment unit that outputs a sound signal by adjusting the directivity of the sound source acquisition unit to be enhanced;
The voice included in the voice signal output from the directional sound source adjustment unit is recognized, and based on the recognition result, it is determined whether or not the person is a two-way communication target. A voice recognition display control unit that displays information on the display unit based on position information;
Based on the position information output from the voice recognition display control unit, directivity calculation processing is performed to provide directivity so that sound is transmitted only to the target person, and based on the calculation result, the sound output unit A sound processing system comprising a pronunciation directivity adjustment calculation unit that outputs sound from The sound processing system according to claim 7,
The location information update unit
When the gaze state detection unit determines that the person is gazing at the specific display unit, the gaze state detection unit updates and tracks the position information each time the position information of the person's face changes. Sound processing system. The sound processing system according to claim 7,
The voice recognition display control unit
A sound processing system that displays video information of a small screen on the display unit based on position information of a person who is an object of bidirectional communication. The sound processing system according to claim 8, wherein
The position information acquired by the position information update unit is a plurality of persons, and the position information of the plurality of persons is tracked and updated.

Images