JP2005316704A - Surrounding state notification device and method for notifying surrounding state - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enhance accuracy, safety, etc. about surrounding state notification based on the surrounding sound of an automobile. <P>SOLUTION: Sound (surrounding sound) around the outside of an automobile is collected and sound information obtained by sound collection is reproduced in the automobile as sound by which orientation is given. Namely the surrounding sound of the automobile is stereophonically reproduced in the automobile. Moreover a notice direction is determined and a surrounding sound to be reproduced in the automobile is controlled so that a sound in the orientation corresponding to the determined notice direction is emphasized by any acoustic change. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an ambient condition notifying apparatus and method for notifying a driver of an ambient condition of a vehicle such as an automobile.

In automobiles and the like, it is often performed to provide an ambient condition notification device for notifying a driver of an ambient condition outside the vehicle for safety.
As such an ambient condition notification device, for example, a device in which a predetermined number of camera devices are attached to a predetermined part of an automobile and an image around the automobile imaged by the camera apparatus is displayed on a display device in the automobile is known. It has been.
In addition, as information on the surrounding situation, information using ambient sounds obtained by collecting sounds outside the vehicle with a microphone has been proposed. For example, it recognizes the traveling sound of other vehicles from the collected ambient sounds, and automatically determines the presence or absence of an approaching vehicle based on the recognition result.

JP-A-8-202999

  As the above-mentioned surrounding state notification device, the method of displaying the surrounding image of the automobile imaged by the camera on the display device in the vehicle has a great advantage in that the driver can visually recognize the surrounding state. Although there are obstacles in the direction in which the camera captures images or when the surroundings are very dark at night or the like, it is difficult to turn around and recognize the surroundings. In addition, since the driver has to look at the display screen of the display device in order to confirm the surrounding situation, the driver's line of sight is deviated from the traveling direction, for example.

Considering such a situation, it is more possible for the ambient condition notification device to make the situation notification based on the ambient sound as complete as possible rather than relying on the surrounding captured image for the notification of the ambient condition. It can be said that this is one form for providing high safety.
From this point of view, for example, when looking at the configuration described in Patent Document 1, the surrounding situation such as the presence or absence of an approaching vehicle is grasped based on the ambient sound. It can be said that the problem of use at night or at night can be avoided. However, since the driver himself / herself needs to visually recognize the image displayed on the display device, the movement of the line of sight is inevitably involved. In this respect, it is difficult to say that the situation notification based on the ambient sound is sufficiently performed.

Further, in Patent Document 1, a warning sound is output in response to the determination that the approaching vehicle is present based on the ambient sound. The warning sound has a fixed sound pattern. . Therefore, for example, while outputting a warning sound only for the presence of an approaching vehicle, the warning sound only needs one pattern, but in order to enhance the situation notification based on the ambient sound, the situation judgment items are diversified, When aiming at accuracy, it is necessary to prepare and output various patterns of warning sounds.
In this case, the driver has to hear several complicated warning sounds during driving. This can be annoying to the driver and also imposes the burden of attention that they must listen carefully, understand and respond to these various patterns of warning sounds each time. It is easily imagined that it will interfere with safe driving.
In this way, at present, there is still room for enhancing the accuracy, safety, etc. of the surrounding situation notification based on the ambient sound of the vehicle.

Accordingly, the present invention is configured as follows as an ambient condition notifying device in consideration of the above-described problems.
That is, sound collecting means that is attached to a required position of the vehicle so that ambient sounds that are ambient sounds outside the vehicle are picked up, and voice information obtained by collecting the sound by the sound collecting means are The attention direction, which is the direction in which the attention level is particularly high in the direction of the surroundings of the vehicle, based on the reproduction means that reproduces the sound with a sense of localization and the current behavior state of the driver and / or the operation state of the vehicle And a control means for controlling the reproduction means so that the localization sound corresponding to the attention direction determined by the determination means is emphasized more than the localization sound in a direction other than the attention direction around the vehicle. It was decided to prepare.

  Further, as a surrounding situation notification method, a sound collection procedure that is attached to a required position of the vehicle so that ambient sounds, which are ambient sounds outside the vehicle, are collected, and the sound is collected by the sound collection procedure. Based on a reproduction procedure for reproducing sound information as a sound having a sense of orientation in the vehicle and a predetermined behavior state of the current driver and / or an operation state of the vehicle, the attention level is particularly high in the vehicle surrounding direction. A direction determination procedure for determining a direction of attention, which is a direction to be generated, and a sound that is localized according to the direction of attention determined by the determination procedure of the sound reproduced by the reproduction procedure is the attention direction around the vehicle And a control procedure for performing control so that the sound is more emphasized than a localized sound in any other direction.

  According to each of the above-described configurations, the present invention collects ambient sounds outside the vehicle (ambient sounds) and reproduces the sound information obtained by the collected sounds as sounds with localization in the vehicle. Like to do. This means that sounds around the vehicle can be reproduced in the vehicle. As a result, the driver can recognize the situation around the vehicle by listening to the reproduced sound. However, since the reproduced sound has a localization corresponding to the ambient sound, it is included in the ambient sound. The driver can also recognize the direction of arrival of a specific sound (for example, the traveling sound of another vehicle). In addition, the ambient sound reproduced in the vehicle is controlled so that the localized sound corresponding to the determined direction of attention is emphasized by some acoustic change, so that the driver should be careful in the direction to listen carefully. An environment where it is easy to hear sound will be obtained.

Thus, according to the present invention, a surrounding sound with a sense of localization can be heard while being in the vehicle, so that the situation around the vehicle can be accurately recognized. In other words, it provides an environment that notifies the surrounding situation with high reliability by sound. In this respect, for example, when the surrounding situation is notified by displaying an image of the surrounding situation. Since there is no movement of the driver's line of sight, high safety can be obtained.
In addition, the accuracy of the notification of the surrounding situation is further enhanced by emphasizing the reproduced ambient sound so that the sound in the direction in which the driver should be careful is easily heard.
In this way, in the present invention, as a surrounding situation notification based on information obtained by collecting ambient sounds, the surrounding situation is more accurately notified to the driver than before. Therefore, the safety is improved accordingly.

  The best mode for carrying out the present invention (hereinafter referred to as an embodiment) will be described below. In the present embodiment, an ambient situation notification system mounted on an automobile as a vehicle is taken as an example.

  The ambient condition notification system according to the present embodiment uses audio information obtained by collecting ambient sounds outside the automobile (ambient sounds) using a microphone. For this reason, in the present embodiment, a microphone is attached to the automobile so that sounds around the automobile are collected. An example of attaching a microphone as the present embodiment is shown in FIGS.

First, FIG. 1 will be described.
FIG. 1 is a plan view of an automobile 1 on which the surrounding state notification system of the present embodiment is mounted.
As an example shown in FIG. 1, a total of eight microphones MR <b> 1 to MR <b> 8 are attached to the periphery of the edge of the roof of the automobile 1 in the arrangement state shown in the figure.
The mounting positions of these microphones MR1 to MR8 correspond to the direction of ambient sound to be collected.
The microphone MR1 is attached to the front center of the roof, and collects ambient sounds in the direction of the front center of the automobile 1.
The microphone MR2 is attached to the right front side of the roof, and collects ambient sounds in the direction of the right front side of the automobile 1.
The microphone MR3 is attached to the right center of the roof, and collects ambient sounds in the direction of the right center of the automobile 1.
The microphone MR4 is attached to the right rear side of the roof, and collects ambient sounds in the direction of the right rear side of the automobile 1.
The microphone MR5 is attached to the center of the back of the roof, and collects ambient sound in the direction of the center of the back of the automobile 1.
The microphone MR6 is attached to the left rear side of the roof, and collects ambient sounds in the direction of the left rear side of the automobile 1.
The microphone MR7 is attached to the left center of the roof, and collects ambient sounds in the direction of the left center of the automobile 1.
The microphone MR8 is attached to the left front side of the roof, and collects ambient sounds in the direction of the left front of the automobile 1.
According to such an arrangement of the microphones MR1 to MR8, it is possible to collect sound around 360 ° around the automobile 1 so as to be divided into approximately eight directions (45 °).

FIG. 2 shows an example in which a microphone is attached to the body side instead of the roof of the automobile 1.
Also in this figure, a total of eight microphones MB1 to MB8 are attached. The relationship between the mounting positions of these microphones MB1 to MB8 with respect to the automobile and the direction of ambient sound to be collected is the same as that of the microphones MR1 to MR8 in FIG.

As the surrounding state notification system of the present embodiment, at least one of the microphone attachment forms shown in FIGS. 1 and 2 is adopted. Regardless of which one of FIGS. 1 and 2 is adopted, the ambient sound around 360 ° around the automobile 1 can be collected with the same eight-direction resolution.
Further, focusing on the fact that the microphone mounting position is the roof of the automobile 1 in FIG. 1 and the body in FIG. 2, the difference is the height. That is, in the form of FIG. 1, a relatively high position corresponding to the height of the roof is preferentially collected as ambient sound, and in the form of FIG. 2, a position lower than the roof corresponding to the height of the body is collected. Sound is collected with emphasis.
Considering that there is a difference in the height of the sound collection position, combining the microphone mounting examples in FIGS. 1 and 2 can cover a wide sound collection range even in the height direction. It can be said that ambient sound information is more accurate and preferable. As a matter of course, the present embodiment can adopt a configuration in which the microphone mounting examples shown in FIGS. 1 and 2 are combined. However, in the following description, it is convenient to make the description simple and easy to understand. The above description is based on a surrounding situation notification system that employs the configuration shown in FIG.
In addition, the microphones MR1 to MR8 and MB1 to MB8 shown in FIG. 1 or FIG. 2 are preferably those having strong directivity characteristics such as unidirectionality, for example.

FIG. 3 schematically shows the indoor environment of the automobile 1 when the surrounding state notification system is mainly used.
The driver U shown in this figure is assumed to be sitting in the driver's seat. A left speaker SPL and a right speaker SPR are embedded in the headrest 2 that supports the driver's head behind the seat in the seat. The left speaker SPL and the right speaker SPR are in positions corresponding to the left and right ears of the driver U, respectively, in a state where the driver U is normally sitting in the driver's seat for driving. It is attached as follows.

The left speaker SPL and the right speaker SPR reproduce and output sound signals obtained by collecting ambient sounds with the microphones MR1 to MR8 as shown in FIG. 1 in the ambient condition notification system of the embodiment. It is provided to do.
In this case, the sound obtained by collecting the ambient sound has 8 channels independently corresponding to the microphones MR1 to MR8, whereas the speakers for reproducing the sound are the left speaker SPL and the right speaker SPR. For two channels.
However, the ambient condition notification system according to the present embodiment performs virtual surround processing in the audio signal processing system as described later. In this virtual surround processing, when the driver's seat is in the listening position, the sound field formed by the sound output from the left speaker SPL and the right speaker SPR is as if the sound collected by the microphones MR1 to MR8. Are perceived as sounds as if they were output from the virtual speakers SPV1 to SPV8 shown in FIG. Each position of the virtual speakers SPV1 to SPV8 corresponds to the direction of the ambient sound collected by the microphones MR1 to MR8.
As the sound field formed in this way, for example, the sound that actually arrives from the right front outside the automobile 1 can be heard as it is from the right front by the driver U as it is. Become. In other words, the sound field formed by the sound output from the left speaker SPL and the right speaker SPR is almost reproduced for the driver U in the interior of the automobile 1 with respect to the direction of arrival of the ambient 360 ° sound outside the automobile 1. It will sound like it is.

  In addition, a camera 3 for taking an image of the face of the driver U is provided in the interior of the automobile 1. The camera 3 is an image pickup apparatus including an image pickup device such as a CCD. For example, when the driver U is sitting in the driver's seat and driving normally, the face of the driver U is more than necessary from the front side. The image is taken according to the occupancy ratio in the screen and is provided at a position that does not hinder driving. For example, it is attached near the upper side of the windshield on the driver's seat side.

The block diagram of FIG. 4 shows a configuration example of a part mainly serving as an audio signal processing system in the ambient state notification system of the present embodiment.
The sound signal obtained by collecting the sound with the microphone MR1 is actually converted into a digital signal and input to the band-pass filter (BPF) 11-1. The band-pass filter 11-1 removes predetermined low / high frequency band components that may be regarded as unnecessary noise such as wind noise and vibration noise of the automobile 1 itself, and passes the input audio signal. , Input to the unsteady sound detector 12-1.

The unsteady sound detection unit 12-1 performs frequency analysis processing by FFT, for example, at regular intervals for the input audio signal, and converts the time waveform into information in the frequency domain. Further, the memory 13-1 is connected to the unsteady sound detection unit 12-1. The unsteady sound detection unit 12-1 generates averaged data obtained by averaging audio signals input in a past fixed time, and writes and stores the averaged data in the memory 13-1. In addition, the unsteady sound detection unit 12-1 subtracts, in the frequency domain, the audio signal input as described above from the averaged data held in the memory 13-1. This subtraction result is the difference between the ambient sound of the past fixed time averaged and held in the memory 13-1, and the current ambient sound. If the ambient sound is treated as a stationary sound, it is a frequency domain component of an unsteady sound. Specifically, for example, if another vehicle approaches as the state of the ambient sound of a car, and the traveling sound of the other vehicle becomes noticeable or an alarm sound such as a horn sounds, A frequency domain component corresponding to a smooth running sound or horn sound is obtained as the subtraction result. That is, the unsteady sound detection unit 12-1 of the present embodiment detects the unsteady sound by obtaining the subtraction result. Further, according to the specific example described above, the unsteady sound detected by the unsteady sound detection unit 12-1 is a sound that should be paid attention when the driver is driving as the ambient sound of the automobile 1. It can be said that.
In the non-stationary sound detection unit 12-1, the frequency domain (band) component as the non-stationary sound detected as described above is converted into a time waveform (audio signal waveform) by performing inverse FFT, and the subsequent stage is detected. The data is output to the delay processing unit 15-1 in the direction processing unit 14.

  Similarly, the audio signals obtained by collecting the remaining seven microphones MR2 to MR7 pass through the bandpass filters 11-2 to 11-8, respectively, and have the memories 13-2 to 13-8. A component as an unsteady sound is detected by being input to the steady sound detection units 12-2 to 12-8. Then, the audio signal as the unsteady sound is output to the direction processing unit 14.

The direction processing unit 14 emphasizes the sound in the direction determined as the attention direction for the sound reproduced and output from the left speaker SPL and the right speaker SPR as virtual surround sound in accordance with the control of the direction processing control unit 21 described later. This is a part for executing signal processing (direction processing) for making the above.
In the direction processing unit 14, [delay processing unit 15-1 → sign inversion processing unit 16-] corresponding to each non-stationary sound signal output from the non-stationary sound detection units 12-1 to 12-8. 1] to [Delay processing unit 15-8 → Signal inversion processing unit 16-8].
In these signal processing systems, each of the delay processing units 15-1 to 15-8 is individually set with a delay time under the control of the direction processing control unit 21. Then, the input audio signal is delayed and output by the set delay time.
Digital audio signals output from the delay processing units 15-1 to 15-8 are input to the sign inversion processing units 16-1 to 16-8, respectively. The sign inversion processing units 16-1 to 16-8 individually output the digital values as the input audio signals as non-inverted as controlled by the direction processing control unit 21 or function as inverters. To set whether to invert and output. The audio signals output from the sign inversion processing units 16-1 to 16-8 are input to the selection / addition processing unit 17.

The selection / addition processing unit 17 performs signal selection and addition processing using a predetermined pattern for the audio signals input from the sign inversion processing units 16-1 to 16-8 under the control of the direction processing control unit 21. To do. Then, each of a plurality of systems of signals obtained by selection and addition is output to a required portion of the sound image localization processing units 18-1 to 18-8 provided in the subsequent stage of the direction processing unit 14. Is done.
A specific operation example of the direction processing unit 14 according to the control of the direction processing control unit 21 will be described later.

Each of the sound image localization processing units 18-1 to 18-8 is basically provided corresponding to each system of audio signals collected by the microphones MR1 to MR8. Therefore, as the operation, for example, the sound image localization processing unit 18-1 is a signal processing system (BPF 11-1 → unsteady sound detection unit 12-1 → delay processing unit 15-1 → sign inversion processing unit) corresponding to the microphone MR1. 16-1 → selection / addition processing unit 17) generates audio signals for two channels to be supplied to the speakers SPL and SPR in FIG. The two-channel audio signals are subjected to signal processing based on a predetermined transfer function or the like so as to be heard from the position of the virtual speaker SPV1 when reproduced and output from the speakers SPL and SPR in FIG. It is a signal obtained by this.
Similarly, the remaining sound image localization processing units 18-2 to 18-8 perform signal processing on the audio signals input through the signal processing systems corresponding to the microphones MR2 to MR8 and reproduce them from the speakers SPL and SPR. When output, two-channel audio signals are generated and output so as to be heard from the positions of the virtual speakers SPV2 to SPV8.

Of the two-channel audio signals generated by each of the sound image localization processing units 18-1 to 18-8, the left-channel audio signal to be output from the left speaker SPL is input to the addition processing unit 19L. .
Of the two-channel audio signals generated by each of the sound image localization processing units 18-1 to 18-8, the right-channel audio signal to be output from the right speaker SPL is input to the addition processing unit 19R. The
The addition processing units 19L and 19R perform addition synthesis processing on a maximum of eight audio signals respectively input. Then, the audio signal obtained by adding and synthesizing by the addition processing unit 19L is input to the left speaker SPL as a speaker drive signal. Similarly, the audio signal obtained by addition synthesis by the addition processing unit 19R is input as a speaker drive signal to the right speaker SPR.
With such a configuration, the sound reproduced and output from the speakers SPL and SPR is based on the sound collected by the microphones MR1 to MR8 as described above with reference to FIG. A sound field that is a sound image localized at the position of ~ SPV8 is formed.

The camera 3 shown in FIG. 4 is to be provided in the interior of the automobile 1 in FIG. 3, and as described above, the driver U is normally sitting in the driver's seat. The driver's face is imaged.
An imaging signal (video signal) obtained by imaging by the camera 3 is input to the face image analysis processing unit 20.

The face image analysis processing unit 20 recognizes the image portion of the driver's face that is visible when the image signal is displayed as an image, using the input image signal data. Then, an analysis process is performed on the image portion of the recognized face, and in this case, the direction in which the recognized face is facing is determined. In this case, the determination is made with resolution in four directions, front, rear, left and right. That is, as the direction of the face of the driver U, if it recognizes that it is almost facing the front, it is the front direction, if it recognizes that it is facing the left, it is the left direction, and if it recognizes that it is facing the right, it is the right direction. If it is recognized that it is facing backward, it is determined that it is the backward direction.
It should be noted that as a face image recognition technique employed for such face orientation determination, a technique known so far or to be known in the future may be appropriately employed. Here, there is no particular limitation.

  The direction processing control unit 21 captures information on the determination result of the face orientation in the front, rear, left, and right sections by the face image analysis processing unit 20 and also captures predetermined vehicle operation status information from the vehicle 1. Here, the information on the setting state of the transmission and the operation state information of the direction indicator are taken in as the vehicle operation state information.

  In the direction processing control unit 21, based on the information of the face orientation determination result captured (input) as described above and the vehicle operation status information, the driver U should be particularly careful at that time. The direction of attention, which is the direction at, is determined. And based on the determination result, control with respect to the direction processing part 14 demonstrated previously is performed. Hereinafter, this point will be described.

  FIG. 5 shows a procedure of processing for determining the attention direction executed by the direction processing control unit 21. Note that the processing shown in this figure includes a processing procedure for obtaining the determination result of the face orientation by the face image analysis processing unit 20 described above.

In step S101, the direction processing control unit 21 first captures transmission state information as vehicle operation status information, and determines whether or not the transmission is in the back state based on the transmission state information at this time. . Here, if a positive determination result is obtained because the transmission is in the back state, the process proceeds to step S111. Depending on step S111, it is determined that the attention direction is backward.
On the other hand, if a negative determination result is obtained in step S101 that the transmission is not in the back state, the process proceeds to step S102.

  In step S102, the operation status information of the direction indicator is taken in as the vehicle operation status information, and it is determined whether or not the direction indicator is being operated. If a positive determination result is obtained that the direction indicator is being operated, the process proceeds to step S107 to determine whether the direction indicated by the direction indicator is left or right according to the operation. .

If it is determined in step S107 that the direction indicated by the direction indicator is left, the process proceeds to step S109.
In step S109, the attention direction is determined to be the left direction.
If it is determined in step S107 that the direction indicated by the direction indicator is right, the process proceeds to step S110.
In step S110, the attention direction is determined to be the right direction.

  On the other hand, if a negative determination result is obtained in step S102 that the direction indicator is not operated, the process proceeds to step S103 and subsequent steps. The processing of step S103 to step S106 is processing as the face image analysis processing unit 20.

  In step S103, a signal (video signal) of an image captured by the camera 3 is captured, and in the next step S104, a facial image analysis process is executed using the captured signal. In this case, as described above, first, face image data is obtained by extracting data of an image portion where a face is displayed when a video signal is displayed as an image. Then, by analyzing the face image data according to a predetermined algorithm, it is possible to recognize which direction the driver's face is facing. And based on this recognition result, the process of subsequent step S105, S106 is performed.

In step S105, it is determined whether or not a state in which the face is directed in a specific direction has been obtained for a predetermined time or more. In this case, for example, if the driver cannot normally determine the direction in which the driver is facing for some reason, a negative result is obtained in step S105.
If a negative result is obtained in step S105, the face orientation is indefinite (unknown). Therefore, in this case, the process proceeds to step S113, and the attention direction is determined as unknown.
On the other hand, when a positive determination result is obtained in step S105 that the face is directed in a specific direction over a predetermined time set in advance, the process proceeds to step S106.

In step S106, it is determined whether the specific direction in which the face is directed for a predetermined time or more in step S105 is the front, back, left, or right. Then, when the determination result that the direction facing the face is the front is obtained, the process proceeds to step S108, and the attention direction is determined to be the front.
If it is determined that the direction in which the face is directed is left, the process proceeds to step S109, and the attention direction is determined to be left.
If it is determined that the direction in which the face is directed is right, the process proceeds to step S110, and the attention direction is determined to be right.
If it is determined that the direction in which the face is directed is backward, the process proceeds to step S111, and the attention direction is determined to be backward.
In this way, in the present embodiment, information indicating the driver's face orientation obtained by the face image analysis processing unit 20 and vehicle operation status information (transmission status information, direction indicator operation status information). ) Can be used to determine the direction of attention divided into front, rear, left and right (and unknown).
Then, when the attention direction is determined through any of the processes of steps S108 to S112, the process corresponding to the determined attention direction is executed by the process as step S113. The control for the direction processing unit 14 is executed.

6 to 9 show the direction processing set in the direction processing unit 14 in accordance with the control of the direction processing control unit 21 as step S113.
First, FIG. 6 shows direction processing corresponding to a case where the attention direction is determined to be forward or unknown after step S108 or step S112.
In this figure, inputs 1 to 8 are audio signals as non-stationary sounds output from the non-stationary sound detectors 12-1 to 12-8, and microphones MR1 to MR8 are used as sound collection sources, respectively. Corresponding to
The outputs 1 to 8 are signals to be input to the sound image localization processing units 18-1 to 18-8, respectively.

In this case, first, for the delay processing units 15-1 to 15-8, all the delay time values VLd are set to VLd = 0.
Further, the non-inverted mode is set for all of the sign inversion processing units 16-1 to 16-8 to which the outputs of the delay processing units 15-1 to 15-8 are input.
Also in the selection / addition processing unit 17 to which the outputs of the sign inversion processing units 16-1 to 16-8 are input, the sign inversion processing units 16-1 to 16-8 are not particularly performed without performing signal selection and addition processing. The output signal from is output as it is.
In such a direction processing setting in the direction processing unit 14, the sound signals of the unsteady sound collected by the microphones MR1 to MR8 are not particularly processed as they are, and the sound image localization processing units 18-1 to 18- 8 respectively. In this case, the unsteady sound reproduced by the virtual surround from the speakers SPL and SPR is not such that the sound in a specific direction is acoustically emphasized, but a sound field corresponding to the actual balance of ambient sounds. Will be obtained.

FIG. 7 shows direction processing corresponding to the case where the attention direction is determined to be right after step S110.
In this case, the delay time value VLd = 0 is set for the delay processing units 15-1 to 15-5 corresponding to the microphones MR1 to MR5.
On the other hand, for the delay processing unit 15-6 corresponding to the microphone MR6, | a, b | is the absolute value of the physical distance between a and b, and C is the speed of sound.
VLd = | MR4, MR6 | / C (Formula 1)
A delay time value VLd represented by In this case, | MR4, MR6 | indicates the absolute value of the physical distance between the left rear microphone MR6 and the right rear microphone MR4. Therefore, the delay time value VLd according to the above equation 1 is the microphone MR6. It shows the arrival time of sound from to the microphone MR4.

Further, regarding the delay processing unit 15-7 corresponding to the microphone MR7,
VLd = | MR3, MR7 | / C (Formula 2)
A delay time value VLd represented by That is, the arrival time of the sound from the left center microphone MR7 to the right center microphone MR3 is set as the delay time value VLd.

In addition, regarding the delay processing unit 15-8 corresponding to the microphone MR8,
VLd = | MR2, MR8 | / C (Formula 3)
A delay time value VLd represented by That is, the arrival time of sound from the left front microphone MR8 to the right front microphone MR2 is set as the delay time value VLd.

The sign inversion processing units 16-1 to 16-5 corresponding to the delay processing units 15-1 to 15-5 in which the delay time VLd = 0 is set are set to the non-inversion mode.
On the other hand, the sign inversion processing units 16-6 to 16- corresponding to the delay processing units 15-6 to 15-8 in which the delay time values VLd represented by the above (formula 1) to (formula 3) are set. Inversion modes are set in 8 respectively.

Further, in the selection / addition processing unit 17, as shown in the figure, the output of the sign inversion processing unit 16-6 is output as the output of the sign inversion processing unit 16-4 as the input signal selection / addition processing. To be added to.
Further, the output of the sign inversion processing unit 16-7 is added to the output 3 which is the output of the sign inversion processing unit 16-3. The output of the sign inversion processing unit 16-8 is added to the output 2 that is the output of the sign inversion processing unit 16-2.

The output 4 is originally an unsteady sound signal using the right rear microphone MR4 as a sound collection source. Further, the output of the sign inversion processing unit 16-6 is an unsteady sound signal using the microphone MR6, which is normally the output 6, as a sound collection source, and this sound signal is transmitted from the microphone MR6 to the microphone MR4. After delaying the sound arrival time and further inverting the value, the output 4 is added.
The output 4 obtained in this way is obtained by picking up sound from a sound signal obtained by picking up the microphone MR4 with the left rear as the sound pickup position, by the microphone MR6 located on the opposite left side. The same audio signal component as that obtained is reduced and canceled.

Similarly, the output 3 using the right center microphone MR3 as the sound collection source is the sound signal obtained by picking up the sound from the microphone MR3 to the left center microphone MR7 located on the opposite side in the left-right direction. Thus, the same audio signal component obtained by collecting the sound is reduced and canceled.
The output 2 using the right front microphone MR2 as the sound collection source is picked up by the left front microphone MR7 located on the opposite side in the left-right direction from the sound signal obtained by the sound picked up by the microphone MR2. The same audio signal component as obtained in this way is reduced and canceled.

As for output 1 and output 5, input 1 and input 5 are output as they are. That is, the audio signal having the sound collection sound sources of the microphones MR1 and MR5 at the center front and center rear is output as it is without being processed in the direction processing control unit 14.
In this case, the output 6 to the output 8 are no signal.

  As a result of the subsequent sound image localization processing units 18-1 to 18-5 and the addition processing units 19L and 19R executing the processing using the outputs 1 to 5 output in this way, they are output from the speakers SPL and SPR. As the sound to be played, the sound that is obtained by picking up the microphones MR6, MR7, MR8 with the left side as the sound pickup direction is canceled or suppressed on the right side that is the opposite direction. That is, the sound that arrives from the left direction as the ambient sound to be reproduced is weakened, and an acoustic state in which the sound that arrives from the right direction determined as the attention direction is emphasized relatively is obtained.

FIG. 8 shows direction processing corresponding to the case where the attention direction is determined to be left after step S109.
In this case, the delay time value VLd = 0 is set for the microphone MR1 and the delay processing units 15-1 and 15-5 to 15-8 corresponding to the microphones MR5 to MR8.
On the other hand, for the delay processing unit 15-2, the delay time value VLd (= | MR2, MR8 | / C) represented by the above (formula 3) is set. That is, the arrival time of sound from the microphone MR2 to the microphone MR8 is set.
For the delay processing unit 15-3, the delay time value VLd (= | MR3, MR7 | / C) represented by the above (formula 2) is set. That is, the arrival time of sound from the microphone MR3 to the microphone MR7 is set.
Further, for the delay processing unit 15-4, the delay time value VLd (= | MR4, MR6 | / C) represented by the above (formula 1) is set. That is, the arrival time of sound from the microphone MR4 to the microphone MR6 is set.

Also in this case, the delay processing unit 15-1 in which the delay time VLd = 0 is set, and the code inversion processing unit 16-1 and the code inversion processing unit 16 corresponding to the delay processing units 15-5 to 15-8. A non-inversion mode is set for each of −5 to 16-8.
Also, the sign inversion processing unit 16-corresponding to the delay processing units 15-2, 15-3 and 15-4 to which the delay time value VLd represented by the above (Equation 3), (Equation 2) and (Equation 1) is set. Inversion modes are set in 2, 16-3 and 16-4, respectively.

In the selection / addition processing unit 17, the outputs of the sign inversion processing units 16-1 and 16-5 are directly input to the sound image localization processing units 18-1 and 18-5 as output 1 and output 5.
On the other hand, for the output 6, the output of the sign inversion processing unit 16-4 is added to the output of the sign inversion processing unit 16-6. For output 7, the output of the sign inversion processing unit 16-3 is added to the output of the sign inversion processing unit 16-7. For the output 8, the output of the sign inversion processing unit 16-2 is added to the output of the sign inversion processing unit 16-8.

By such signal addition, signals as output 1, output 5, output 6, output 7, and output 8 are obtained, and output 2, output 3, and output 4 are no signals.
As outputs 1 and 5, unsteady sound signals using the microphones MR1 (front center) and MR5 (back center) as sound collection sources are output as they are.
On the other hand, the output 6 was obtained by picking up sound from the right rear microphone MR4 located on the opposite side in the left-right direction from the sound signal obtained by picking up the sound with the left rear microphone MR6. The components of the same audio signal as in the above are reduced and canceled.
The output 7 is the same as that obtained by picking up sound from the sound signal obtained by the left center microphone MR7 and picked up by the right center microphone MR3 located on the opposite side in the left-right direction. The audio signal component is reduced or canceled.
The output 8 is the same audio signal as obtained by collecting the sound from the sound signal obtained by the left front microphone MR8 and picked up by the right front microphone MR2 located on the opposite side in the left-right direction. These components are reduced and canceled.

  Sounds output from the speakers SPL and SPR by the audio signals as the output 1 and the output 5 to the output 8 are collected by the microphones MR2, MR3, and MR4 with the sound collecting direction on the right side. The obtained voice is canceled or suppressed in the left direction which is the opposite direction. That is, as a result, an acoustic state is obtained in which the sound that arrives from the left direction determined as the attention direction in this case is emphasized.

FIG. 9 shows direction processing corresponding to the case where the attention direction is determined to be left after step S109.
In this case, the delay time values VLd = 0 are set for the delay processing units 15-3 to 15-7 corresponding to the microphones MR3 to MR7.
About the delay processing unit 15-1,
VLd = | MR1, MR5 | / C (Formula 4)
A delay time value VLd represented by This (Equation 4) indicates the arrival time of sound from the front center microphone MR1 to the rear center microphone MR5.
For the delay processing unit 15-2,
VLd = | MR2, MR4 | / C (Formula 5)
A delay time value VLd represented by This (Equation 5) shows the arrival time of sound from the right front microphone MR2 to the right rear microphone MR4.
The delay processing unit 15-8 is
VLd = | MR8, MR6 | / C (Formula 6)
A delay time value VLd represented by This (Equation 6) indicates the arrival time of sound from the left front microphone MR8 to the left rear microphone MR6.

In this case, the sign inversion processing units 16-3 to 16-7 corresponding to the delay processing units 15-3 to 15-7 in which the delay time VLd = 0 is set are set to the non-inversion mode. .
Also, the sign inversion processing unit 16-corresponding to the delay processing units 15-1, 15-2, and 15-8 in which the delay time value VLd represented by the above (Equation 4), (Equation 5), and (Equation 6) is set. Inversion modes are set in 1, 16-2 and 16-8, respectively.

In the selection / addition processing unit 17, the outputs of the sign inversion processing units 16-3 and 16-7 are directly input to the sound image localization processing units 18-3 and 18-7 as output 3 and output 7. Outputs 3 and 7 use right and center left microphones MR3 and MR7 as sound collection sources.
On the other hand, for the output 4, the output of the sign inversion processing unit 16-2 is added to the output of the sign inversion processing unit 16-4. For output 5, the output of the sign inversion processing unit 16-1 is added to the output of the sign inversion processing unit 16-5. For the output 6, the output of the sign inversion processing unit 16-8 is added to the output of the sign inversion processing unit 16-6.

By such signal addition, signals as output 3, output 4, output 5, output 6, and output 7 are obtained, and output 1, output 2, and output 8 are no signals.
Out of outputs 3 to 7 for signal output, as output 3 and output 7, unsteady sound signals having microphones MR3 (right center) and MR5 (left center) as sound collection sound sources are output as they are, respectively. Will be.
On the other hand, the output 4 was obtained by picking up sound from the right front microphone MR2 located on the opposite side in the front-rear direction from the sound signal obtained by picking up the sound with the right rear microphone MR4. The components of the same audio signal as in the above are reduced and canceled.
Further, the output 5 is the same as that obtained by picking up the sound from the sound signal obtained by the microphone MR5 at the rear center and picked up by the front center microphone MR1 located on the opposite side in the front-rear direction. The audio signal component is reduced or canceled.
The output 6 is the same as that obtained by picking up sound from the sound signal obtained by the left rear microphone MR6 and picked up by the left front microphone MR8 located on the opposite side in the front-rear direction. The audio signal component is reduced or canceled.

  The sound output from the speakers SPL and SPR by the sound signals as the output 3 to output 7 thus obtained is assumed to have been obtained by collecting the sounds with the microphones MR1, MR2 and MR8 having the sound collection direction in the front. Is canceled or suppressed at the rear in the opposite direction. That is, in this case, an acoustic state is obtained in which the sound that arrives from the rear direction determined as the attention direction is emphasized.

The following can be said as the ambient condition notification system of the present embodiment configured as described above.
First, as this embodiment, as shown in FIG. 1 (or FIG. 2), microphones MR1 to MR8 that collect ambient sounds of the vehicle 1 are provided, and these microphones MR1 to MR8 can be obtained as sound collection sources. The audio signal is reproduced in a surround manner in the car interior and output as sound. That is, reproduction is performed in the automobile room so as to give a sense of localization corresponding to the actual ambient sound.
Thus, the driver can clearly understand, for example, what kind of sound comes from which direction by listening to the sound reproduced in the automobile interior. In other words, grasping the surrounding situation that relies on sound is more accurate than ever. As the understanding of the surrounding situation that relies on sound becomes more accurate, the need for notification of the surrounding situation by displaying an image of the surroundings becomes lighter. In other words, when grasping the surrounding situation, it is freed from the inconvenience that the surroundings are dark or an image that grasps the surrounding situation cannot be taken due to obstacles, etc., and the line of sight is moved for confirmation of the surrounding situation It is no longer necessary, and in this respect, it can be said that safety is higher than before.

In addition, as a practical device, in the present embodiment, a past audio signal that is regarded as an unsteady sound by performing unsteady sound detection on the sound signal obtained by collecting each microphone. The sound field of the ambient sound is formed only by the sound signal component of the difference with respect to the average data.
In the present invention, first, it is a requirement that the ambient sound picked up by the microphone is reproduced in the vehicle interior so as to give a sense of localization corresponding to the actual sound of the surrounding sound. Therefore, the sound picked up by the microphone is required. It does not prevent itself from being reproduced and output.
However, as a practical matter, when the driver is driving, it is actually the driving that keeps listening to the sound collected by the microphone as the ambient sound reproduced from the speakers SPL and SPR. It may be a considerable burden on the person. Therefore, if the sound field that reproduces the ambient sound in the vehicle interior is formed only by the sound signal component of the unsteady sound as described above, the unsteady sound is only generated when the unsteady sound is generated. , It will be reproduced so that it can be heard from its actual direction of arrival. That is, it is not necessary for the driver to continue to listen to ambient sounds reproduced from the speaker. However, when an unsteady sound is generated, the sound is accurately reproduced and heard, and a sufficiently effective notification function of the surrounding situation is performed.

Further, in the present embodiment, the direction of attention that is the direction that the driver should be careful of is determined, and the sound coming from the determined direction of attention is emphasized. The sound is weakened so as to contrast it. This means that, for the driver U, an environment is obtained in which it is always easy to hear the sound in the direction that he should be aware of. It is improving the sex and accuracy.
When the determined direction of attention is forward (unknown), the sound that is localized forward is output normally without being emphasized over the sound that is localized backward. This is because the direction in which the driver is paying attention during driving is usually forward, so that the sound of the entire surroundings can be heard evenly rather than emphasizing the front. This is because it is considered to be advantageous from the viewpoint of safety.

  By the way, in the description of the embodiments so far, it is assumed that the microphones MR1 to MR8 are installed on the roof portion of the automobile as shown in FIG. Instead, even when the same eight microphones MB1 to MB8 are installed as shown in FIG. 2, the same signal processing system configuration as described with reference to FIGS. Equivalent safety and accuracy can be obtained for notifications.

In addition, as described above, in the embodiment, the microphone arrangements of FIGS. 1 and 2 are combined, and the microphones MR1 to MR8 on the roof side and the microphones MB1 to MB8 on the body side are provided for one automobile 1. The arrangement may be adopted, and in this case, there is an advantage that the sound collection range can be made wider in the height direction.
As a configuration of the signal processing system in this case, for example, one of the signal processing system and the sound output system corresponding to the microphones MR1 to MR8 on the roof side shown in FIG. It is conceivable to provide a signal processing system and an audio output system corresponding to .about.MB8. In this case, in the headrest 2, two pairs of left and right speakers SPL and SPR corresponding to the microphones MR1 to MR8 on the roof side and left and right speakers SPL and SPR corresponding to the microphones MB1 to MB8 on the body side are provided. Provide a speaker. At this time, the left and right speakers SPL and SPR corresponding to the microphones MR1 to MR8 on the roof side are arranged on the upper side, and the left and right speakers SPL and SPR corresponding to the microphones MB1 to MB8 on the body side are arranged on the lower side. It is possible to reproduce a sense of localization in the height direction according to the difference in height of the sound pickup position between the microphone on the roof side and the microphone on the body side.
Alternatively, the left and right speakers SPL and SPR provided on the headrest 2 are paired as signal processing of the sound image localization processing unit 18 corresponding to the microphones MR1 to MR8 on the roof side and the microphones MB1 to MB8 on the body side. It may be configured to generate an audio signal that reproduces a sense of localization.

Further, in the embodiment, as described above, only the unsteady sound detected by the unsteady sound detection unit 12 is reproduced and output in the automobile room. However, in the present invention, for example, unsteady sound detection is performed. It is also included that the unit 12 (and the memory 13) is omitted and the ambient sound collected by the microphone is reproduced in the automobile room. First, by obtaining such a configuration, the original purpose of being able to accurately grasp the situation around the automobile by sound is achieved.
Further, in practice, as the ambient sound to be reproduced in the automobile room, the operation of the non-stationary sound detection unit 12 is made effective, and the sound field composed of only the non-stationary sound is formed. May be configured to be switchable between a mode in which a sound field composed of an ambient sound picked up by a microphone is formed as is, and a sound field composed of the ambient sound as it is.

  In the above embodiment, the sound collected in eight directions is finally played in virtual surround by the two speakers SPL and SPR. For example, a speaker corresponding to the number of microphones is connected to the microphone. You may make it arrange | position in a motor vehicle corresponding to a sound-collection direction. In this case, for example, without performing the processing as the localization processing units 18-1 to 18-8, for example, by outputting each collected sound obtained by collecting each microphone from the corresponding speaker, The ambient sound collected by the microphone can be reproduced in the automobile room. In addition, in the same manner as in the above embodiment, only non-stationary sound components are extracted from the collected sound signal of each microphone, and signal processing for enhancing the determined attention direction is performed. What is necessary is just to comprise.

  Further, in the above embodiment, the direction processing is performed in the direction processing unit 14 so that the sound localized in the direction corresponding to the determined attention direction is emphasized. In addition to stopping the output of the sound localized at the side opposite to the caution direction, the component of the sound localized at the side opposite to the caution direction from the sound localized according to the caution direction. A technique of canceling (suppressing) is adopted. However, the technique for emphasizing the reproduced sound that is localized corresponding to the attention direction is not limited to this. For example, it may be a simpler change in volume balance, or a method of changing the sound quality so that the sound localized in the required direction is easy to hear or difficult to hear by performing equalizing processing. Conceivable.

  Further, the information used for determining the attention direction is not limited to the information described in the embodiment. For example, as the operation status information of the automobile, it is conceivable to use, for example, information on the rotation angle of the steering wheel in addition to the setting status information of the transmission and the operation status information of the direction indicator. In addition to determining the direction of the driver's face itself, it is also conceivable to detect the driver's line of sight and use it for determining the attention direction.

In the embodiment, the 360 ° direction around the automobile is divided into eight directions, and correspondingly, sound is collected by eight microphones in the plane direction. In other words, sound is collected with a resolution of eight directions and a reproduction sound field is reproduced, but this resolution is not limited to eight directions and may be arbitrary. If the resolution is further increased to 16 directions, for example, it is possible to form a reproduced sound field that is more realistic.
Or conversely, an arbitrary resolution of 8 directions or less may be set. For example, it is considered that sufficiently safe and accurate notification of the surrounding situation is possible even with four directions of resolution in front, rear, left and right. Furthermore, for example, when notification of the surrounding situation only in the forward direction is considered, it is possible to set the resolution in the two directions of the left and right fronts.

  4 may be configured by hardware or realized as software processing executed by a CPU (Central Processing Unit) according to a program, for example. It may be what is done.

It is a figure which shows the example of installation of the microphone with respect to a motor vehicle as the surrounding condition notification system of embodiment of this invention. It is a figure which shows the example of installation of the microphone with respect to a motor vehicle as the surrounding condition notification system of embodiment. It is a figure which shows typically the indoor environment of the motor vehicle at the time of seeing the surrounding condition notification system of embodiment as a main body It is a block diagram which shows the structural example of the signal processing type | system | group in the ambient condition notification system of embodiment. It is a flowchart which shows the processing operation for attention direction determination in the surroundings notification system of embodiment. It is a block diagram which shows the operation example of the direction process part corresponding to the determined attention direction in the surroundings notification system of embodiment. It is a block diagram which shows the operation example of the direction process part corresponding to the determined attention direction in the surrounding condition notification system of embodiment. It is a block diagram which shows the operation example of the direction process part corresponding to the determined attention direction in the surrounding condition notification system of embodiment. It is a block diagram which shows the operation example of the direction process part corresponding to the determined attention direction in the surroundings notification system of embodiment.

Explanation of symbols

  1 automobile, MR1 to MR8, MB1 to MB8 microphone, 2 headrest, SPL, SPR speaker, 3 camera, 11-1 to 11-8 bandpass filter, 12-1 to 12-8 unsteady sound detector, 13-1 -13-8 memory, 14-direction processing unit, 15-1 to 15-8 delay processing unit, 16-1 to 16-8 sign inversion processing unit, 17 selection / addition processing unit, 18-1 to 18-8 sound image localization Processing unit, 19L, 19R addition processing unit, 20 face image analysis processing unit, 21 direction processing unit

Claims (5)

Sound collecting means attached to a required position of the vehicle so that ambient sounds that are ambient sounds outside the vehicle are collected;
Reproducing means for reproducing the sound information obtained by collecting sound by the sound collecting means as sound having a sense of localization in the vehicle;
Determining means for determining a direction of attention, which is a direction in which the degree of attention is particularly high in the surrounding direction of the vehicle, based on a predetermined behavior state of the current driver and / or an operation state of the vehicle;
Control means for controlling the reproduction means so that the localization sound corresponding to the attention direction determined by the determination means is emphasized more than the localization sound in a direction other than the attention direction around the vehicle;
An ambient condition notifying device comprising: The determination means is
Imaging means attached to the vehicle such that the face of the driver driving the vehicle is imaged;
Determination means for determining a direction in which the driver is facing around the vehicle based on information on a driver's face image in a captured image obtained by imaging with the imaging means as the attention direction; ,
The ambient condition notification device according to claim 1, further comprising: The determination means is
The attention direction is determined based on at least one information of a state of a transmission, a state of a direction indicator, and a rotation angle of a handle in the vehicle.
The ambient condition notifying apparatus according to claim 1. The reproducing means is
The difference between the past information about the sound information obtained by collecting the sound by the sound collecting means and the sound information obtained by collecting the sound by the sound collecting means is calculated as a non-stationary sound. With non-stationary sound detection means obtained as audio information,
Reproducing the sound information of the non-stationary sound as a sound having a sense of localization in the vehicle;
The ambient condition notifying apparatus according to claim 1. A sound collection procedure that is attached to a required position of the vehicle so that ambient sounds that are ambient sounds outside the vehicle are collected;
A reproduction procedure for reproducing the sound information obtained by the sound collection procedure as sound having a sense of localization in the vehicle;
A direction determination procedure for determining a direction of caution, which is a direction that is considered to be particularly high in the surrounding direction of the vehicle, based on a predetermined behavior state of the current driver and / or an operation state of the vehicle;
As for the sound reproduced by the reproduction procedure, the localization sound corresponding to the attention direction determined by the determination procedure is more emphasized than the localization sound in a direction other than the attention direction around the vehicle. Control procedures to control;
An ambient condition notification method characterized by being configured to execute

Images