JP6589734B2 - Crew state estimation device - Google Patents

Description

  The present invention relates to an occupant state estimation device that estimates an occupant state.

  Conventionally, there is an occupant state estimation device that estimates the state of a driver's seat occupant, such as the degree of fatigue, consciousness level, and whether or not he / she can concentrate on driving operations, for a passenger seated in the driver's seat (hereinafter, driver's seat occupant) Various proposals have been made. For example, the occupant state estimation device disclosed in Patent Document 1 determines whether or not the driver's seat occupant is in a normal state based on the movement of the driver's occupant's line of sight.

  Specifically, it is as follows. The occupant state estimation device disclosed in Patent Document 1 has a function of detecting the line-of-sight direction of a driver's seat occupant and a function of determining whether an event that should be noted by the driver's occupant has occurred. When an event that requires attention occurs, information for warning the event (hereinafter, alert information) is displayed on the display. The alert information includes the direction in which the driver's seat occupant should look.

  In addition, the driver's seat occupant state estimation device tracks the driver's occupant's line of sight after displaying the alert information on the display, and the driver's seat occupant state is watching the alert information displayed on the display ( Thereafter, the gaze time) and the time from when the line of sight is directed to the display until the line of sight is directed in the direction corresponding to the alert information (hereinafter, reaction time) are measured.

  Then, it is determined whether or not the driver's seat occupant is in a normal state based on whether or not the measured gaze time or reaction time is equal to or less than a predetermined threshold. In addition, the state where the driver's seat occupant is not normal in Patent Document 1 is a state in which the level of consciousness is reduced, fatigued, or concentration is reduced.

JP 2002-25000 A

  In the method disclosed in Patent Document 1, the driver's seat occupant cannot recognize the direction in which the line of sight is directed (in other words, the direction to be careful) unless the display is once viewed. Therefore, in order to estimate the state of the driver's seat occupant, it is necessary to cause the driver's seat occupant to take two actions: a step of visually recognizing the display and a step of visually recognizing the direction displayed on the display. Naturally, the greater the direction in which the driver's seat occupant should look, the greater the burden on the driver's seat occupant.

  The present invention has been made based on this situation, and an object of the present invention is to provide an occupant state estimation device that can estimate the state of the driver occupant without having the driver's sight line look at the display. It is in.

A first invention for achieving the object is used in a vehicle, and outputs a sound signal from each of two or more sound signal output devices to localize a virtual sound source at a desired position in a three-dimensional space. Based on the captured image of the stereoscopic sound image generation unit (F4) that generates the stereoscopic image and the camera provided in the passenger compartment, the visual recognition direction that is the direction that the occupant seated in the predetermined seat is viewing is specified. Whether the occupant saw the localization direction, which is the direction in which the virtual sound source was localized within a predetermined reaction waiting time after the stereoscopic sound image was generated, based on the identification result of the visual recognition direction specifying unit (F5) and the visual recognition direction specifying unit A visual recognition determination unit (F6) that determines whether or not, an occupant state estimation unit (F7) that estimates the state of the occupant based on the determination result of the visual recognition determination unit, and peripheral information that indicates traffic conditions around the vehicle Peripheral information detection device Based on the peripheral information acquired by the peripheral information acquisition unit (F1) that acquires the peripheral information and the peripheral information acquisition unit, the presence of an object that can come into contact with the vehicle is detected and the direction in which the object exists is specified A risk direction specifying unit (F3), and the peripheral information acquiring unit acquires, as peripheral information, a moving body type of a moving body existing around the vehicle, a relative speed with respect to the vehicle, a relative position with respect to the vehicle, and a risk direction. The identifying unit identifies a direction in which an approaching body, which is a moving body whose distance from the vehicle is decreasing, based on the surrounding information acquired by the surrounding information acquisition unit, and collides with each approaching body based on the surrounding information. A risk evaluation unit (F31) that evaluates the risk, and a pseudo sound data storage unit (M1) that stores pseudo sound data for outputting the pseudo sound corresponding to the mobile body type as an acoustic signal for each mobile body type. The surrounding information includes at least a moving body type for each moving body existing around the vehicle, and the three-dimensional sound image generation unit is a direction in which the localization direction that is the direction in which the virtual sound source is located for the occupant is specified by the risk direction specifying unit. By generating a three-dimensional sound image so as to coincide with the occupant, the presence of the approaching body is notified to the occupant, and among the approaching bodies, the risk evaluation unit evaluates that the collision risk is less than a predetermined notification threshold. For the approaching object that does not generate a three-dimensional sound image corresponding to the moving object, there is an approaching object that is evaluated by the risk evaluation unit as having a collision risk equal to or higher than a predetermined notification threshold. , A three-dimensional sound image is generated so that the direction in which the approaching body is present matches the localization direction of the virtual sound source, and there is an approaching body that is evaluated to be equal to or greater than a predetermined notification threshold. In addition, according to the collision risk evaluated by the risk evaluation unit for the approaching object that is to notify the occupant of the existence by generating the three-dimensional sound image with the pseudo sound according to the moving body type of the approaching object , imitative of the types of 3D sound is characterized that you change the blowing mode without changing.
The second invention is used in a vehicle and outputs a sound signal from each of two or more sound signal output devices to generate a three-dimensional sound image in which a virtual sound source is localized at a desired position in a three-dimensional space. A visual direction specifying unit (F5) that specifies a visual direction, which is a direction in which a passenger seated in a predetermined seat is viewing, based on a sound image generating unit (F4) and a captured image of a camera provided in a vehicle interior. And a visual determination that determines whether or not the occupant has viewed the localization direction that is the direction in which the virtual sound source is localized within a predetermined reaction waiting time after the stereoscopic sound image is generated based on the identification result of the visual recognition direction identification unit And an occupant state estimation unit (F7) that estimates the state of the occupant based on the determination result of the visual recognition determination unit, and the occupant state estimation unit sequentially determines the appropriateness of the state as a driver's seat occupant Judgment and less sense of the crew Both the stimulus application unit (F9) that controls the operation of the stimulation device that stimulates any one of them, and the appropriateness degree based on the temporal change of the appropriateness level sequentially determined by the occupant state estimation unit, the driver's seat at the time of automatic driving And a prediction unit (F8) for predicting the timing of shifting to an inappropriate level designed in advance as an occupant state, and the stimulus applying unit is lowered to an inappropriate level within a predetermined remaining time by the prediction unit Then, when it is predicted, the stimulator is operated.
According to a third aspect of the present invention, there is provided a three-dimensional sound image that is used in a vehicle and generates a three-dimensional sound image in which a virtual sound source is localized at a desired position in a three-dimensional space by outputting sound signals from each of two or more sound signal output devices. A visual direction specifying unit (F5) that specifies a visual direction, which is a direction in which a passenger seated in a predetermined seat is viewing, based on a sound image generating unit (F4) and a captured image of a camera provided in a vehicle interior. And a visual determination that determines whether or not the occupant has viewed the localization direction that is the direction in which the virtual sound source is localized within a predetermined reaction waiting time after the stereoscopic sound image is generated based on the identification result of the visual recognition direction identification unit And an occupant state estimation unit (F7) that estimates the state of the occupant based on the determination result of the visual recognition determination unit, and the visual recognition direction specifying unit visually recognizes the occupant seated in each of the plurality of seats. Configure to identify direction The visual recognition determination unit determines, for each occupant, whether or not the virtual sound source has been localized, and the occupant state estimation unit determines whether the state of each occupant is based on the determination result by the visual determination unit. It is characterized in that it is determined whether or not the state of the driver's seat occupant during automatic driving is an appropriate state designed in advance.
A fourth invention is a three-dimensional sound image that is used in a vehicle and generates a three-dimensional sound image in which a virtual sound source is localized at a desired position in a three-dimensional space by outputting sound signals from each of two or more sound signal output devices. A visual direction specifying unit (F5) that specifies a visual direction, which is a direction in which a passenger seated in a predetermined seat is viewing, based on a sound image generating unit (F4) and a captured image of a camera provided in a vehicle interior. And a visual determination that determines whether or not the occupant has viewed the localization direction that is the direction in which the virtual sound source is localized within a predetermined reaction waiting time after the stereoscopic sound image is generated based on the identification result of the visual recognition direction identification unit And an occupant state estimation unit (F7) that estimates the state of the occupant based on the determination result of the visual recognition determination unit, and the visual recognition direction specifying unit visually recognizes the occupant seated in each of the plurality of seats. Configure to identify direction The visual recognition determination unit determines, for each occupant, whether or not the direction in which the virtual sound source is localized is viewed, and the occupant state estimation unit sequentially determines the appropriate degree of the state as the driver's seat occupant, A state for each occupant is estimated based on a determination result by the visual recognition determination unit.

  In the above configuration, the occupant state estimation unit estimates the state of the occupant based on whether or not the occupant has visually recognized the direction in which the stereo sound image is localized. According to such a configuration, it is not necessary for the driver's seat occupant to see the display once in determining the appropriateness of the driver's seat occupant. This is because the direction in which the virtual sound source is localized is the direction in which the line of sight should be directed.

  Therefore, the state of the driver's seat occupant can be estimated even if the driver's seat occupant does not look at the display.

  In addition, the code | symbol in the parenthesis described in the claim shows the correspondence with the specific means as described in embodiment mentioned later as one aspect, Comprising: The technical scope of this invention is limited is not.

1 is a block diagram showing a schematic configuration of an occupant state estimation system 100. FIG. It is a figure which shows an example of the installation position of the speaker. 1 is a block diagram showing a schematic configuration of an occupant state estimation device 10. FIG. It is a figure which shows an example of the determination result data preserve | saved at the result memory | storage part M2. It is a figure for demonstrating the appropriateness evaluation map. It is a figure for demonstrating the action | operation of the appropriateness estimation part F8. It is a flowchart for demonstrating a passenger | crew state management process. It is a figure for demonstrating the action | operation of the appropriateness estimation part F8 in the modification 1. FIG.

  Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating an example of a schematic configuration of an occupant state estimation system 100 according to the present embodiment. The occupant state estimation system 100 of the present embodiment is mounted on a vehicle having an automatic driving function, and the state of an occupant seated in the driver's seat of the vehicle (hereinafter referred to as a driver's seat occupant) at the time of automatic driving. The degree of appropriateness (hereinafter referred to as appropriateness) is sequentially determined.

  The appropriateness of the state of the driver's seat occupant here may be a parameter indicating the degree of fatigue of the body including the brain, or may be a parameter indicating the level of consciousness such as sleepiness. The appropriateness level may be a parameter indicating the degree of stability of the mental state.

  Here, as an example, it is assumed that the appropriateness is a combination of these various factors. The higher the degree of appropriateness, the closer to the appropriate state defined in advance as the state of the driver's seat occupant during automatic driving.

  The automatic driving function here is a function that automatically executes steering, acceleration, and deceleration so that the host vehicle travels along a planned travel route set by a user. Hereinafter, for convenience, a vehicle on which the occupant state estimation system 100 is mounted is also referred to as a host vehicle.

<Configuration of Crew State Estimation System 100>
As shown in FIG. 1, an occupant state estimation system 100 includes an occupant state estimation device 10, a camera unit 21, a millimeter wave radar 22, an inter-vehicle communication unit 23, a wide area communication unit 24, a driver status monitor (hereinafter referred to as DSM: Driver Status). Monitor) 25, speaker 26, display 27, vibrator 28, air conditioner 29, and automatic operation ECU 30.

  The camera unit 21, the millimeter wave radar 22, the inter-vehicle communication unit 23, the wide-area communication unit 24, the DSM 25, the speaker 26, the display 27, the vibrator 28, the air conditioner 29, and the automatic operation ECU 30 are respectively the occupant state estimation device 10 and the vehicle. Are connected to each other via a communication network (hereinafter referred to as LAN: Local Area Network).

  The occupant state estimation device 10 is configured as a computer including a CPU 11, a RAM 12, a ROM 13, an I / O 14, and a bus line that connects these configurations. The CPU 11 performs various arithmetic processes and is realized using, for example, a microprocessor. The RAM 12 is a volatile memory, and the ROM 13 is a non-volatile memory. The I / O 14 is an interface for the occupant state estimation device 10 to input and output data with the camera unit 21, the millimeter wave radar 22, and the like. The I / O 14 may be realized using analog circuit elements, ICs, or the like.

  The ROM 13 stores a program (hereinafter referred to as an occupant state estimation program) for causing a normal computer to function as the occupant state estimation device 10. Note that the above-described occupant state estimation program only needs to be stored in a non-transitory tangible storage medium. The execution of the occupant state estimation program by the CPU 11 corresponds to the execution of a method corresponding to the occupant state estimation program.

  The occupant state estimation device 10 provides various functions to be described later when the CPU 11 executes an occupant state estimation program. Moreover, the passenger | crew state management process mentioned later is performed by making those functions cooperate.

  The outline of the occupant state management process is as follows. The occupant state estimation device 10 may collide with another moving body existing in the vicinity of the own vehicle and the own vehicle based on data input from the camera unit 21 or the millimeter wave radar 22 (in other words, collision risk). ). When there is a moving object having a collision risk equal to or higher than a predetermined threshold, a three-dimensional sound image is generated at a position corresponding to the direction in which the moving object exists.

  Then, the appropriateness of the driver's seat occupant is determined by observing the movement of the line of sight of the driver's seat occupant from the time when the three-dimensional sound image is generated. If the degree of appropriateness determined is at a level corresponding to an inappropriate state, a stimulus is given to the five senses of the driver's seat occupant so as to return to the appropriate state. In addition, the other mobile bodies existing around the own vehicle here include not only other vehicles but also pedestrians and the like.

  The camera unit 21 and the millimeter wave radar 22 detect moving objects such as pedestrians and other vehicles, and still objects such as falling objects on the road, traffic signals, guardrails, curbs, road signs, road markings, and lane markings. It is an autonomous sensor. That is, each of the camera unit 21 and the millimeter wave radar 22 functions as a source device (hereinafter referred to as a surrounding information detection device) of information (hereinafter referred to as surrounding information) indicating the traffic situation and traveling environment around the host vehicle. The traffic situation here represents the positional relationship between the own vehicle and a moving body such as a pedestrian and other vehicles, the number of vehicles existing in the vicinity (that is, the vehicle density), etc. This refers to the positional relationship of the vehicle with respect to stationary objects such as falling objects, traffic signals, guardrails, curbs, road signs, road markings, and lane markings, road gradients, road curvature, and the like.

  For example, the camera unit 21 performs image recognition processing on a camera (hereinafter referred to as an exterior camera) installed so as to capture a predetermined range (for example, the front) outside the vehicle interior and an image captured by the exterior camera. Thus, it may be realized by using an image recognition processing module that detects the above-described various detection objects such as a pedestrian. As the vehicle exterior camera, an optical camera such as a CMOS camera or a CCD camera can be used.

  Of course, an infrared camera may be used as a camera outside the vehicle compartment. Further, the shooting range of the camera outside the vehicle compartment is not limited to the front of the host vehicle, but may be the side of the host vehicle or the rear of the host vehicle. The photographing range of the vehicle exterior camera may be appropriately designed. Further, the vehicle exterior camera may be realized using a plurality of cameras.

  Further, the millimeter wave radar 22 transmits and receives millimeter waves or quasi-millimeter waves, thereby acquiring information about an object existing in a predetermined range in front of the host vehicle (hereinafter referred to as a radar detection area). Specifically, an object existing in the radar detection area is detected, and the direction in which the detected object exists, distance, relative speed, type, and the like are estimated.

  Here, as an example, the millimeter wave radar 22 estimates the direction, distance, relative speed, type, and the like where the detected object exists by analyzing the reception result of the reflected wave, but the present invention is not limited thereto. As another aspect, the millimeter wave radar 22 provides the reception result of the reflected wave to the occupant state estimation device 10, and the occupant state estimation device 10 analyzes the reception result of the reflected wave provided from the millimeter wave radar 22. It is good also as an aspect which estimates the direction, distance, relative speed, classification, etc. in which a detected object exists.

  As another aspect, a laser radar may be employed instead of the millimeter wave radar 22 as a radar device that detects an object existing ahead of the host vehicle. Further, the millimeter wave radar 22 and the laser radar may be used in combination. Each of the camera unit 21 and the millimeter wave radar 22 sequentially provides detected object information related to the detected moving object and stationary object to the occupant state estimation device 10. The detected object information corresponds to the peripheral information described above.

  The inter-vehicle communication unit 23 is a communication module for performing direct wireless communication (so-called inter-vehicle communication) with other vehicles using radio waves in a predetermined frequency band such as a 5.9 GHz band or a 760 MHz band.

  For example, the inter-vehicle communication unit 23 transmits a communication packet including vehicle information indicating the traveling state of the host vehicle, and receives a communication packet including vehicle information of another vehicle. The vehicle information includes the current position, traveling direction, vehicle speed, acceleration, and the like. The communication packet including the vehicle information includes information such as the transmission time of the communication packet and transmission source information in addition to the vehicle information. The transmission source information is an identification number (so-called vehicle ID) assigned to the vehicle corresponding to the transmission source.

  Thus, the inter-vehicle communication unit 23 also functions as a device that acquires the peripheral information. Therefore, the inter-vehicle communication unit 23 also corresponds to a peripheral information detection device.

  The wide area communication unit 24 is a wireless communication module that is connected to a wide area communication network and performs communication with an external device (not shown) via the wide area communication network. For example, the wide area communication unit 24 communicates with a traffic information distribution center that distributes road traffic information corresponding to the current position of the vehicle, a weather information distribution center that distributes weather information, and the like. Thereby, the wide area communication unit 24 acquires traffic information and weather information corresponding to the current position of the host vehicle.

  The traffic information here includes traffic jam information indicating the degree of congestion of the road on which the vehicle is traveling, traffic regulation information, road surface information, construction information, accident information, obstacle information, and the like. The road surface information is information indicating a road surface state such as whether the road surface is wet or frozen. Obstacle information is information indicating the position of a parked vehicle, the position of a fallen object on a road, its type, and the like. The current position of the host vehicle may be specified based on a navigation signal transmitted by a navigation satellite provided in the Global Navigation Satellite System (hereinafter referred to as GNSS).

  Thus, the wide area communication unit 24 also functions as a device that acquires peripheral information. Therefore, the wide area communication unit 24 also corresponds to a peripheral information detection device.

  The DSM 25 is realized using a near-infrared light source, a near-infrared camera, a control unit for controlling them, and the like. The DSM 25 photographs the face of the driver's seat occupant irradiated with the near infrared light from the near infrared light source with the near infrared camera. The DSM 25 sequentially detects the direction of the driver's seat occupant's face, the line-of-sight direction, and the like by performing known image recognition processing on the captured image of the near-infrared camera.

  Information indicating the detected face direction and line-of-sight direction of the driver's seat occupant is sequentially output to the occupant state estimation device 10. Note that the direction of the driver's seat occupant's face, the line-of-sight direction, and the like can be realized by using a known method from the captured image, and therefore detailed description thereof is omitted here.

  The speaker 26 is an acoustic signal output device that outputs an acoustic signal (that is, sound) based on an instruction from the occupant state estimation device 10. The speaker 26 should just be provided in the several location suitably designed in a vehicle interior so that a three-dimensional sound image can be provided with respect to a driver's seat passenger | crew.

  In the present embodiment, as an example, it is assumed that the host vehicle is provided with two speakers 26, a right speaker 26R and a left speaker 26L, at desired positions on the instrument panel, as shown in FIG. Thereby, the passenger | crew state estimation apparatus 10 in this embodiment implement | achieves provision of a three-dimensional sound image using a 2-channel speaker.

  For example, the right speaker 26R may be provided diagonally to the right of the driver's seat on the instrument panel, and the left speaker 26L may be provided diagonally to the left of the driver's seat on the instrument panel. Of course, the installation position of the speaker 26 is not limited to the above-described mode. The speaker 26 may be provided on a steering column, a headrest of a driver seat, a pillar portion, a ceiling, or the like.

  In the present embodiment, a two-channel speaker is used to realize a three-dimensional sound field and a three-dimensional sound source, but is not limited thereto. The occupant state estimation device 10 may realize a three-dimensional sound source using a three-channel speaker. The speaker 26 may be a speaker using a bone conduction vibrator or a parametric speaker.

  The display 27 displays text and an image corresponding to the signal input from the occupant state estimation device 10. The display 27 is capable of full color display, for example, and can be configured using a liquid crystal display, an organic EL display, or the like. Further, the display 27 may be a head-up display.

  The vibrator 28 is a device that vibrates a vibration element by driving a motor (not shown). Vibrator 28 is used as a device for stimulating the sense of touch of the driver's seat occupant. Therefore, it is preferable that the vibrator 28 is provided at a place where it is assumed that the body of the driver's seat occupant is touched in the passenger compartment, such as a driver's seat, a pedal (for example, a brake pedal), and a handle.

  During automatic driving, it is assumed that the driver's occupant's hand is away from the handle or the foot is away from the pedal. Therefore, it is preferable that the vibrator 28 is provided on a member assumed to be in contact with the body of the driver's seat occupant even during automatic driving, such as a seat for a driver's seat or a seat belt.

  The air conditioner 29 is a device that adjusts the temperature, humidity, and the like of the passenger compartment. The operation of the air conditioner 29 is controlled by the occupant state estimation device 10. The air conditioner 29 may have a function of blowing out scented air (hereinafter referred to as a scent generating function).

  The automatic driving ECU 30 is an ECU (Electronic Control Unit) that provides an automatic driving function. That is, steering, acceleration, and deceleration are automatically executed so that the host vehicle travels along the planned travel route set by the user. Control such as automatic steering and acceleration / deceleration may be realized by outputting a control signal to an on-vehicle actuator or motor to be controlled. Further, the automatic driving ECU 30 provides the occupant state estimation device 10 with operation information indicating whether or not the automatic driving function is operating.

<Regarding the functions of the occupant state estimation device 10>
Next, the function with which the passenger | crew state estimation apparatus 10 is provided is demonstrated using FIG. The occupant state estimation device 10 provides functions corresponding to various functional blocks shown in FIG. 3 when the CPU 11 executes the above-described occupant state estimation program. That is, the occupant state estimation device 10 includes, as functional blocks, a peripheral information acquisition unit F1, a moving body information extraction unit F2, an approaching body extraction unit F3, a three-dimensional sound image generation unit F4, a visual recognition direction identification unit F5, a visual recognition determination unit F6, and an appropriateness degree. A determination unit F7, an appropriateness prediction unit F8, and a stimulus application unit F9 are provided.

  Part or all of the functional blocks included in the occupant state estimation device 10 may be realized using one or a plurality of ICs (in other words, as hardware). Further, some or all of the functional blocks provided in the occupant state estimation device 10 may be realized by a combination of software execution by the CPU and hardware members.

  In addition, the occupant state estimation device 10 includes a sound data storage unit M1 realized using a part of the storage area included in the ROM 13, a result storage unit M2 realized using a rewritable storage medium such as the RAM 12, and the like. Is provided.

  The sound data storage unit M1 stores pseudo sound data for each moving body type. The moving body type means a moving body type such as a four-wheeled vehicle, a two-wheeled vehicle, a bicycle, and a pedestrian. The four-wheeled vehicle may be further subdivided like a truck or a passenger car.

  The onomatopoeia data of a certain mobile body type is data for outputting onomatopoeia representing the characteristics of the mobile body type from the speaker 26. For example, the pseudo sound data of a two-wheeled vehicle may be data for outputting an acoustic signal imitating the engine sound of a two-wheeled vehicle. The onomatopoeia data may be stored in MIDI (Musical Instrument Digital Interface) format, or may be stored in other formats. The sound data storage unit M1 corresponds to the onomatopoeia data storage unit described in the claims.

  The result storage unit M2 stores a determination result by a visual recognition determination unit F6 described later. The determination result stored in the result storage unit M2 will be described later separately.

  The peripheral information acquisition unit F1 performs mutual communication with each of the camera unit 21, the millimeter wave radar 22, the inter-vehicle communication unit 23, and the wide area communication unit 24, and acquires the peripheral information acquired by each. That is, the peripheral information acquisition unit F1 functions as a communication interface with the peripheral information detection device. The peripheral information acquired by the peripheral information acquisition unit F1 may be stored in the RAM 12 with a time stamp indicating the acquisition time.

  The mobile body information extraction unit F2 extracts mobile body information for each mobile body existing around the host vehicle based on the peripheral information acquired by the peripheral information acquisition unit F1. Here, as an example, the moving body information includes the moving body type, the relative position where the moving body exists with respect to the own vehicle (that is, the relative position), the relative speed with respect to the own vehicle, and the relative traveling direction. , Shall be included. The relative position may be represented by a direction and a distance. The relative speed may be calculated so as to take a positive value when the distance to the host vehicle is small.

  In addition, the mobile body information contained in the peripheral information provided from each peripheral information detection device should just be complementarily couple | bonded and utilized by a well-known sensor fusion technique. The moving body information for each moving body that is sequentially acquired by the moving body information extraction unit F2 may be distinguished for each moving body and stored in the RAM 12.

  Based on the moving body information for each moving body, the approaching body extraction unit F3 extracts a moving body (hereinafter referred to as an approaching body) in an approaching relationship with the own vehicle from among the moving bodies existing around the own vehicle. The moving body that is in an approaching relationship with the host vehicle is a moving body whose distance from the host vehicle tends to decrease. The approaching body may be specified from the time change of the relative position.

  Of course, the approaching body may be determined based on the relative position of the target moving body, the traveling direction of the target moving body, the traveling direction of the host vehicle, and the like. The moving body (that is, the approaching body) extracted by the approaching body extraction unit F3 corresponds to a moving body that may come into contact with the host vehicle. In addition, since the relative position etc. for every moving body are specified by the moving body information extraction part F2, if the approaching body is specified, the relative position of the approaching body is also specified. The direction in which the approaching body exists is the direction in which the moving body that can come into contact with the host vehicle exists. Therefore, the approaching body extraction unit F3 corresponds to the risk direction specifying unit described in the claims.

  Moreover, the approaching body extraction unit F3 includes a risk evaluation unit F31 as a finer functional block. The risk evaluation unit F31 is a functional block that evaluates the collision risk for each approaching body. For example, the risk evaluation unit F31 calculates, for each approaching body, the remaining time until the collision (so-called TTC: Time-To-Collision) based on the relative position, relative speed, and relative traveling direction of the approaching body. It is determined that the smaller the TTC, the higher the collision risk.

  The collision risk may be calculated as a continuous value, but here, as an example, the height of the collision risk is expressed in four stages from level 1 to level 4. Level 1 represents the state with the lowest collision risk, and level 4 represents the state with the highest collision risk.

  For example, level 1 represents a state where there is still sufficient time until a collision or a state where the possibility of a collision is sufficiently low so that it can be regarded as zero. For example, when TTC is equal to or greater than a predetermined first threshold (for example, 15 seconds), it is determined as level 1, and TTC is less than the first threshold and is equal to or greater than a predetermined second threshold (for example, 10 seconds). If it is, level 2 is determined.

  Further, when the TTC is less than the second threshold and is not less than the third threshold (for example, 5 seconds), it is determined as level 3, and when the TTC is less than the third threshold, it is determined as level 4. To do. In addition, what is necessary is just to design the specific value of various threshold values suitably.

  In the present embodiment, as an example, the level is determined based on the size of the TTC. However, the present invention is not limited to this. The level may be determined by a known method.

  In addition, although this embodiment illustrates the aspect which calculates a collision risk (in other words, a level), without using the information which shows the driving state of the own vehicle, it is not restricted to this. A collision risk may be calculated based on the traveling speed, traveling direction, steering angle, and the like of the host vehicle. Since a well-known method can be used as the method for calculating the collision risk from the vehicle information of the host vehicle and the surrounding environment, the details thereof are omitted here.

  The three-dimensional sound image generation unit F4 generates a three-dimensional sound image in which the virtual sound source is localized in an arbitrary direction and an arbitrary distance in the three-dimensional space by outputting an acoustic signal from each of the right speaker 26R and the left speaker 26L. Adjustment of the position of the virtual sound source can be realized by controlling the level difference, time difference, phase difference, etc. of the sound output from each speaker. For such sound image localization, a well-known sound image localization technique used in a known sound field reproduction system can be used.

  In the present embodiment, the case where the three-dimensional sound image generation unit F4 generates a three-dimensional sound image is a case where an approaching body that satisfies a predetermined notification condition exists in the approaching body. The notification condition is a condition for notifying a driver's seat occupant of the presence of a target approaching object by providing a three-dimensional sound image. Specific contents of the notification condition will be described later separately. For convenience, an approaching body that satisfies the notification condition is referred to as a notification target moving body.

  In addition, the three-dimensional sound image generation unit F4 determines the position of the virtual sound source so that the direction in which the notification target moving body exists when viewed from the driver's seat matches the direction perceived by the driver's seat occupant as the virtual sound source. To do. That is, the virtual sound source is arranged in the direction in which the notification target moving body exists when viewed from the driver's seat. For example, when the notification target moving body exists diagonally right behind the host vehicle, the stereoscopic sound image generation unit F4 generates a stereoscopic sound image so that the driver's seat occupant perceives that the virtual sound source exists diagonally right behind. .

  According to such an aspect, the driver's seat occupant can intuitively recognize the direction in which the notification target moving body is present (in other words, the direction where attention should be paid). When the three-dimensional sound image is generated, the three-dimensional sound image generation unit F4 provides information indicating the direction in which the virtual sound source is arranged (hereinafter, the localization direction) to the visual recognition determination unit F6.

  Furthermore, the three-dimensional sound image generation unit F4 generates a three-dimensional sound image using pseudo sound data corresponding to the moving body type of the notification target moving body. For example, when the notification target moving body is a four-wheeled vehicle, a three-dimensional sound image is generated using pseudo sound data corresponding to the four-wheeled vehicle. According to such an aspect, it is possible to make the driver's seat occupant intuitively recognize the moving body type of the notification target moving body based on the types of pseudo sounds constituting the three-dimensional sound image.

  In addition, the three-dimensional sound image generation unit F4 adjusts the sounding mode (in other words, the output mode) of the three-dimensional sound image according to the collision risk determined by the risk evaluation unit F31 for the notification target moving body. For example, the higher the collision risk, the more the three-dimensional sound image is generated in a sounding manner that makes the driver's occupant feel a sense of crisis. For example, the higher the collision risk is, the higher the treble included in the onomatopoeia is output, the volume is increased, or the number of times that the three-dimensional sound image is continuously generated is increased. It should be noted that a pseudo sound signal adjusted so that a three-dimensional sound image in a desired sounding mode is generated may be output to each speaker 26.

  The visual recognition direction specifying unit F5 sequentially acquires information (hereinafter referred to as visual line information) indicating the direction of the driver's occupant's face and the visual line direction from the DSM 25. And the direction (henceforth visual recognition direction) which the driver | operator passenger | crew is looking from the direction of a driver | operator's passenger | crew's face and a gaze direction is specified.

  In the present embodiment, the DSM 25 is configured to identify the face direction and the line-of-sight direction of the driver's seat occupant from the driver's seat occupant's face image, but this is not limitative. As another aspect, the visual recognition direction specifying unit F5 may acquire a face image of the driver's seat occupant and specify the face direction and the line-of-sight direction of the driver's occupant from the image data.

  When the stereoscopic sound image generation unit F4 generates a stereoscopic sound image, the visual recognition determination unit F6 determines whether the driver's seat occupant visually recognizes the localization direction provided from the stereoscopic sound image generation unit F4 based on the visual recognition direction described above. Determine whether or not.

  For example, the visual recognition determination unit F6 may determine that the driver's seat occupant has visually recognized the localization direction when the localization direction and the visual recognition direction match. The match here is not limited to a complete match. For example, when the viewing direction is within a predetermined range determined with the localization direction as a reference, it may be determined that the driver's seat passenger has visually recognized the localization direction.

  In addition, the determination by the visual recognition determination part F6 shall respond | correspond also when a driver's seat passenger | crew visually recognizes via a door mirror or a rearview mirror. Specifically, when the viewing direction of the driver's seat occupant is the direction in which the right door mirror is disposed, it is determined that the driver's seat occupant has visually recognized the rear right side of the vehicle. The same applies to the left door mirror. Further, when the viewing direction of the driver's seat occupant is the direction in which the rearview mirror is disposed, it is determined that the driver's seat occupant has visually recognized the rear of the vehicle.

  In recent years, a system has also been proposed that supports the driver's seat occupant in recognizing traffic conditions around the vehicle by displaying on the display 27 an image captured by a camera that captures a predetermined range outside the vehicle (eg, behind the vehicle). Has been. When the video showing such a real-time situation outside the passenger compartment is displayed on the display 27 and the line-of-sight direction of the driver seat occupant is the direction of the display 27, the visual recognition determination unit F6 displays the driver seat What is necessary is just to determine with the passenger | crew having visually recognized the direction corresponding to the image | video currently displayed on the display 27. FIG.

  Moreover, the visual recognition determination part F6 is provided with the reaction time measurement part F61 as a finer functional block. The reaction time measurement unit F61 calculates the time required for the occupant to turn his / her line of sight in the localization direction after the acoustic signal providing the virtual sound image is output (in other words, after the three-dimensional sound image is generated). taking measurement.

  The visual recognition determination unit F6 outputs an acoustic signal that provides a virtual sound image from the speaker 26, and if the driver's seat occupant does not visually recognize the localization direction even after a predetermined reaction waiting time Tmis has elapsed, the driver's seat occupant Determines that the orientation direction is not visually recognized. The specific time employed as the reaction waiting time Tmis may be appropriately designed. Here, 5 seconds is taken as an example.

  Further, when the driver's seat occupant's viewing direction is directed to the localization direction from when the three-dimensional sound image is generated until the reaction waiting time Tmis elapses, the elapsed time up to that point is adopted as the reaction time.

  The recording processing unit F62 is a functional block that records in the result storage unit M2 the determination result of whether or not the driver's seat occupant has visually recognized the localization direction. Further, when the driver's seat occupant visually recognizes the localization direction, the recording processing unit F62 also records the reaction time specified at that time together with the determination result that the driver's seat occupant visually recognized the localization direction.

  The determination result is collected every time a three-dimensional sound image is generated. The determination results at a plurality of time points may be stored in the result storage unit M2 in chronological order. For convenience, data in which the determination results are arranged in time series is referred to as determination result data. FIG. 4 is a diagram illustrating an example of a schematic configuration of determination result data. The individual determination results are preferably stored in association with time information indicating the time at which the three-dimensional sound image is generated and the localization direction. Note that “OK” in the determination result column of FIG. 4 indicates that the driver's seat occupant visually recognized the localization direction, and “NG” indicates that the driver's seat occupant did not visually recognize the localization direction.

  The recording processing unit F62 stores the determination result collected every time a three-dimensional sound image is generated for a predetermined number of times. For example, the recording processing unit F62 stores the determination results for the latest N times in the result storage unit M2. N may be a positive integer designed as appropriate, for example, 10 or 20.

  The appropriateness determination unit F7 determines which of the plurality of preset levels the appropriateness of the driver's seat occupant is based on the determination result for the last N trials stored in the result storage unit M2. Judging. The level indicating the appropriateness is a stepwise representation of the degree of appropriateness (that is, the appropriateness) as the state of the driver's seat occupant during automatic driving. The appropriateness determination unit F7 corresponds to the occupant state estimation unit recited in the claims.

  Here, as an example, the appropriateness level is expressed in five stages so as to include a level corresponding to an inappropriate state as the state of the driver's seat occupant during automatic driving. The procedure for determining the support level will be described later separately. The higher the number of levels, the closer to the appropriate state for the driver's seat occupant state during automatic driving. That is, level 5 represents the most appropriate state, and level 1 represents the most inappropriate state. The number of levels of appropriateness may be designed as appropriate.

  Here, the level corresponding to the state inappropriate for the state of the driver's seat occupant during automatic driving is assumed to be only level 1. That is, if the state is level 2 or higher, it means that the state is appropriate as the state of the driver's seat occupant during automatic driving, although there is a difference in degree.

  In addition, the appropriate state of the driver's seat occupant during automatic driving means that the driving operation can be resumed at any time, and the inappropriate state depends on the surrounding traffic conditions even if the driving authority is transferred. It shall represent a state where there is a high possibility that the driving operation cannot be performed. In other words, the lower the degree of appropriateness, the more difficult it is for the automatic operation ECU 30 to transfer the authority of the driving operation to the driver's seat occupant.

  In the present embodiment, the appropriateness determination unit F7 prepares for determining the appropriateness as described above, based on the determination result for the last N trials stored in the result storage unit M2, the miss rate Rn, and the average The reaction time Trep is calculated. The trial here refers to generating a three-dimensional sound image.

  The miss rate Rn is a parameter that represents the ratio that the driver's seat passenger missed the localization direction for N trials. For example, when the number of times the localization direction is visually recognized is 10 times out of 10 trials, the miss rate Rn is calculated as 0.7. This is because when the number of times the localization direction is visually recognized is 3 times for 10 trials, the number of missed positions is 7 times.

  The average reaction time Trep is a representative value of the reaction time obtained for the latest N trials. The representative value may be an average value of measured reaction times or a median value. Note that when determining the representative value of the reaction time, trials in which the localization direction is not visually recognized may be excluded.

  Of course, as another aspect, the average reaction time Trep may be determined as the reaction waiting time Tmis as the reaction time in the trial in which the localization direction was not visually recognized. The reaction time in the trial in which the orientation direction was not visually recognized may be calculated as the average reaction time Trep as a time obtained by adding a predetermined penalty time (for example, 3 seconds) to the reaction waiting time Tmis.

  And the appropriateness determination part F7 determines an appropriateness based on two parameters, the miss rate Rn and the average reaction time Trep. In this embodiment, as an example, the appropriateness determination unit F7 determines the appropriateness level from the miss rate Rn and the average reaction time Trep using the appropriateness evaluation map registered in advance in the ROM 13.

  As shown in FIG. 5, the appropriateness evaluation map is data obtained by mapping levels corresponding to two parameters in advance by a test or the like. As shown in FIG. 5, the degree of appropriateness is set to be evaluated higher as the miss rate Rn is smaller and the average reaction time Trep is shorter. In FIG. 5, for example, the range described as level 5 represents the range of the miss rate Rn and the average reaction time Trep that are evaluated as having the appropriateness level 5. Further, the range described as level 4 represents the range of the miss rate Rn and the average reaction time Trep that are evaluated as having the appropriateness level 4.

  In the present embodiment, as an example, the appropriateness is specified using the appropriateness evaluation map, but the present invention is not limited to this. When the miss rate Rn and the average reaction time Trep are input, the appropriateness may be determined using a predetermined function that outputs the appropriateness corresponding to the input values.

  The appropriateness determined by the appropriateness determining unit F7 is stored in the RAM 12 in time series so that the latest appropriateness becomes the head of the list. For convenience, data in which the appropriateness levels specified by the appropriateness level determination unit F7 are arranged in time series will be referred to as appropriateness transition data. The determination of the appropriateness may be performed every time new data is added to the result storage unit M2.

  The appropriateness prediction unit F8 determines whether or not the appropriateness is in a decreasing tendency based on the appropriateness transition data. For example, as shown in the specific results at time Tj−2, Tj−1, and time Tj in FIG. 6, when the appropriateness level is continuously down twice, it may be determined that the trend is decreasing. . The specified result at time Tj represents the latest specified result, the specified result at time Tj-1 is the specified result one time before the latest specified result, and the specified result at time Tj-2 is the latest specified result. The specific result two times before the result is shown.

  In addition, the appropriateness prediction unit F8 determines that the appropriateness is in a decreasing tendency, and when the latest appropriateness is evaluated as level 2 or higher, from the degree of decrease in the appropriateness per certain time, The timing at which the appropriateness shifts to level 1 is specified, and the remaining time Trmn until that timing is specified. The remaining time Trmn is provided to the stimulus application unit F9. The appropriateness prediction unit F8 corresponds to the prediction unit described in the claims.

  The stimulus application unit F9 performs a process of stimulating the sensory organs of the driver's seat occupant by operating the speaker 26, the display 27, the vibrator 28, and the air conditioner 29 in a predetermined operation pattern. For example, the stimulus application unit F9 applies a stimulus to the hearing of the driver's seat occupant by outputting an alarm sound or a voice message from the speaker 26.

  Further, the stimulus application unit F9 stimulates the driver's occupant's vision by displaying an image on the display 27 or blinking the image. Note that a device that stimulates the sight of the driver's seat occupant (hereinafter, a visual stimulation device) is not limited to a display. The visual stimulation device may be an indicator made of LEDs, a lighting device for a vehicle interior, or the like.

  Furthermore, the stimulus application unit F9 stimulates the tactile sensation of the driver's seat occupant by operating the vibrator 28. Note that the device that stimulates the tactile sensation of the driver's seat occupant (hereinafter, tactile stimulation device) is not limited to the vibrator 28. For example, the air conditioner 29 may be employed as a tactile stimulation device. When the air conditioner 29 is employed as a tactile stimulation device, air having a temperature difference that makes a human feel hot or cold with respect to the temperature in the passenger compartment is blown out toward the driver's seat with a predetermined air volume. To instruct. Thereby, the warm sense which belongs to a tactile sense can be stimulated. Further, the tactile sensation of the driver's seat occupant may be stimulated by blowing air to the driver's seat with a predetermined air volume.

  In addition, if the air conditioner 29 has a scent generating function, the stimulus applying unit F9 can stimulate the scent of the driver's seat occupant by operating the scent generating function of the air conditioner 29.

  Thus, the speaker 26, the display 27, the vibrator 28, and the air conditioner 29 function as a device that stimulates human sensory organs (hereinafter referred to as a stimulator). The timing at which the stimulus applying unit F9 applies the stimulus to the driver's seat occupant via the stimulator will be described later.

<Occupant status management process>
Next, an occupant state management process performed by the occupant state estimation device 10 will be described with reference to a flowchart shown in FIG. The occupant state management process is a process for maintaining the state of the driver's seat occupant in an appropriate state as a state during automatic driving.

  The flowchart shown in FIG. 7 may be started at a predetermined execution cycle (for example, every 100 milliseconds) when the automatic driving function is on. In addition, acquisition of the periphery information by the periphery information acquisition part F1 and acquisition and update of the mobile body information for every mobile body by the mobile body information extraction part F2 are implemented independently of this flow.

  Further, as a more preferable aspect in the present embodiment, when a three-dimensional sound image is generated once in step S104 to be described later, this process is not re-executed until the reaction waiting time Tmis elapses from that point. . This is because if the three-dimensional sound image is generated a plurality of times in a relatively short time, the user may be bothered.

  First, in step S101, the approaching body extraction unit F3 specifies an approaching body based on the moving body information for each moving body acquired by the moving body information extraction unit F2. Accordingly, the moving body type, relative position, relative speed, etc. for each approaching body are specified. When the process in step S101 is completed, the process proceeds to step S102.

  In step S102, the risk evaluation unit F31 evaluates the collision risk for each approaching body, and proceeds to step S103. In step S103, it is determined whether or not there is an approaching body (in other words, an approaching body that the driver's seat occupant should visually recognize) that the three-dimensional sound image generation unit F4 should inform the driver's seat occupant of the presence of the three-dimensional sound image. It is a step which determines. That is, in step S103, it is determined whether there is an approaching body that satisfies the notification condition.

  As described above, the notification condition is a condition for notifying the driver's seat occupant of the presence of an approaching body by generating a three-dimensional sound image. Here, as an example, an approaching body that satisfies the notification condition is estimated that the collision risk is equal to or higher than a predetermined notification level (for example, level 2), and the driver's seat occupant does not recognize the presence of the approaching body. Approaching body. The notification level corresponds to the notification threshold described in the claims.

  Whether or not the driver's seat occupant recognizes the presence of the approaching body is determined by whether or not the driver's seat occupant has looked in the direction in which the approaching body exists within a predetermined time (for example, 5 seconds) from now. Just do it. For an approaching body in which the driver's seat occupant is present in the direction in which the line of sight is directed within a predetermined time from the present time, it is determined that the driver's seat occupant has recognized the presence of the approaching body. In other words, the approaching body estimated that the driver's seat occupant does not recognize the presence of the approaching body is an approaching body that exists in a direction in which the line of sight is not directed within a predetermined time from the present time.

  In this embodiment, as an example, the notification condition includes that the driver's seat occupant does not recognize the presence of the approaching body, but the present invention is not limited thereto. For example, the notification condition may be that the collision risk is not less than the notification level. Moreover, it is good also as an aspect which includes the conditions decided from another viewpoint as alerting | reporting conditions. That is, the notification condition may be appropriately designed.

  In the determination process in step S103, if there is no approaching object that satisfies the notification condition, a negative determination is made in step S103, and this flow ends. On the other hand, when there is a moving body that satisfies the notification condition, an affirmative determination is made in step S103 and the process proceeds to step S104. An approaching body that satisfies the notification condition is a notification target moving body. When there are a plurality of approaching bodies that satisfy the notification condition, the approaching body having the highest collision risk is set as the notification target moving body, and the process proceeds to step S104.

  In step S104, the three-dimensional sound image generation unit F4 generates a three-dimensional sound image using the pseudo sound data corresponding to the moving object type of the notification target moving body so that the virtual sound source is localized at the relative position of the notification target moving body. The process moves to S105. At that time, the three-dimensional sound image generation unit F4 generates a three-dimensional sound image in a sounding manner corresponding to the collision risk evaluated by the risk evaluation unit F31.

  In step S105, the visual recognition determination unit F6 determines whether or not the driver's seat occupant has seen the localization direction. When the driver's seat occupant sees the localization direction within the reaction waiting time Tmis after the three-dimensional sound image is generated, the elapsed time until that time is adopted as the reaction time in the current trial. Further, if the driver's seat occupant does not see the localization direction even after the reaction waiting time Tmis has elapsed since the three-dimensional sound image is generated, it is determined that the visual recognition has failed.

  In step S106, the recording processing unit F62 stores the determination result in step S105 in the result storage unit M2, and proceeds to step S107. In step S107, the appropriateness determination unit F7 determines whether or not the determination results stored in the result storage unit M2 have been accumulated N times. If determination results for N times are accumulated, an affirmative determination is made in step S107 and the process proceeds to step S108. On the other hand, when the determination results for N times are not accumulated, a negative determination is made in step S107, and this flow ends.

  In step S108, the appropriateness determination unit F7 determines the appropriateness of the driver's seat occupant based on the latest N determination results stored in the result storage unit M2. That is, the miss rate Rn and the average reaction time Trep are calculated based on the most recent N determination results, and the appropriateness for the current trial is determined using these two parameters and the appropriateness evaluation map. When the determination of the appropriateness is completed, the process proceeds to step S109.

  In step S109, the appropriateness prediction unit F8 predicts future transition of the appropriateness based on the time series data of the appropriateness stored in the RAM 12, and proceeds to step S110. Specifically, it is determined whether or not the appropriateness is decreasing. If it is determined that the appropriateness is in a decreasing tendency and it is determined in step S108 that the appropriateness is level 2 or higher, the remaining time Trmn is calculated.

  In step S110, the stimulus application unit F9 determines whether or not a predetermined stimulus application condition is satisfied. The stimulus application condition is a condition for applying a stimulus to the sensory organ of the driver's seat occupant by operating a predetermined stimulator. The specific content of the stimulation condition may be determined as appropriate.

  Here, as an example, when the appropriateness determined in step S108 is level 1, it is determined that the stimulus application condition is satisfied. Further, when the remaining time Trmn is calculated in step S109 and the calculated remaining time Trmn is equal to or less than a predetermined remaining threshold, it is determined that the stimulus application condition is satisfied.

  If the stimulus application condition is satisfied, an affirmative determination is made in step S110 and the process proceeds to step S111. On the other hand, if the stimulus application condition is not satisfied, a negative determination is made in step S110, and this flow ends.

  In step S111, the stimulus application unit F9 operates a predetermined stimulation device with a predetermined operation pattern, and ends this flow. What stimulation device is operated and what operation pattern should be used may be appropriately designed. However, when the remaining time Trmn is equal to or less than the predetermined remaining threshold value, the operation is performed so that a stronger stimulus is applied to the driver's seat occupant when the appropriateness is level 1. It is preferable. The case where the remaining time Trmn is equal to or less than the remaining threshold means that the state of the driver's seat occupant is still in an appropriate state as the state during automatic driving, whereas the appropriateness is level 1. This means that it is already in an inappropriate state.

  For example, when the appropriateness is level 1, the vibrator 28 is vibrated more strongly than when the remaining time Trmn is less than or equal to a predetermined remaining threshold value. Further, a stronger stimulus may be applied by changing the combination of stimulating devices to be operated. For example, when the remaining time Trmn is less than or equal to a predetermined remaining threshold value, a stimulus is applied to the sense of touch or vision using the display 27 and the vibrator 28, while when the appropriateness is level 1, In addition to the stimulus, the air conditioning device 29 may be used to stimulate the sense of warmth and smell.

<Summary of Embodiment>
In the above configuration, when the automatic driving function is operating, the approaching body extraction unit F3 identifies a moving body that is in an approaching relationship with the host vehicle, and evaluates the risk of collision between the moving body and the host vehicle. The three-dimensional sound image generation unit F4 sequentially determines whether or not there is a moving body (that is, a notification target moving body) that should be notified to the driver's seat occupant based on the calculated collision risk. When the notification target moving body exists, a three-dimensional sound image is generated so that the driver's seat occupant perceives that the virtual sound source exists in the direction in which the notification target moving body exists.

  The visual recognition determination unit F6 determines whether the driver's seat occupant has visually recognized the direction in which the virtual sound source is localized (that is, the localization direction) with the generation of the three-dimensional sound image as a trigger. The required time (that is, reaction time) is specified. And the appropriateness determination part F7 determines the appropriateness of a driver | operator seat passenger | crew based on the miss rate Rn and average reaction time Trep which are determined from the determination result of the visual recognition determination part F6.

  According to such an aspect, in determining the appropriateness of the driver's seat occupant, there is no need for the driver's seat occupant to see the display once. This is because the direction in which the virtual sound source is localized is the direction in which the line of sight should be directed. Therefore, according to the above configuration, the state of the occupant can be estimated even if the driver's seat occupant does not look at the display.

  Further, in the above configuration, the pseudo sound corresponding to the moving body type of the notification target moving body is provided to the driver's seat occupant as a three-dimensional sound image. Therefore, the driver's seat occupant can recognize what kind of moving body the occupant state estimation device 10 is calling attention by listening to the three-dimensional sound image.

  Furthermore, the three-dimensional sound image provided to the driver's seat occupant is generated in a sounding manner corresponding to the collision risk determined by the risk evaluation unit F31 for the notification target moving body. Therefore, the driver's seat occupant can recognize the collision risk calculated by the occupant state estimation device 10 by listening to the three-dimensional sound image.

  By the way, when the vehicle is traveling in automatic driving, it is foreseen that the frequency of the driver's seat occupant looking at the display 27 will be lower than that during manual driving. Therefore, with the method disclosed in Patent Document 1, it is considered difficult to estimate the state of the driver's seat occupant during automatic driving with the same accuracy as during manual driving. On the other hand, according to the present embodiment, since it is not necessary to visually recognize the display 27 in estimating the state of the driver's seat occupant, the state of the driver's seat occupant can be determined with accuracy similar to that during manual operation even during automatic operation. It can be estimated.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the above-mentioned embodiment, The various modifications described below are also contained in the technical scope of this invention, and also in addition to the following However, various modifications can be made without departing from the scope of the invention. Further, the above-described embodiment and various modifications can be combined as appropriate within a range where no contradiction occurs.

  In addition, about the member which has the same function as the member described in the above-mentioned embodiment, the same code | symbol is attached | subjected and the description is abbreviate | omitted. In addition, when only a part of the configuration is mentioned, the configuration of the above-described embodiment can be applied to the other portions.

[Modification 1]
In the above-described embodiment, a mode in which the appropriateness as the state of the driver's seat occupant during automatic driving is expressed in a stepwise manner using a plurality of levels, but may be expressed by a continuous value. In such an aspect, it is possible to calculate an approximate straight line (dashed line in the figure) indicating a tendency of appropriateness from the history of appropriateness sequentially calculated as shown in FIG. 8, and to specify the remaining time Trmn. it can. In addition, the lower side of the broken line in FIG. 8 represents a range (hereinafter referred to as an inappropriate range) meaning that the state is an inappropriate state as a driver's seat occupant state during automatic driving, and the time Tx is a degree of appropriateness. Represents a timing at which is predicted to be in an inappropriate range.

  It should be noted that the function for calculating the appropriateness based on the miss rate Rn and the average reaction time Trep (hereinafter referred to as the appropriateness evaluation function) may be designed as appropriate. Naturally, the appropriateness evaluation function is configured to calculate the appropriateness higher as the miss rate Rn is smaller and as the average reaction time Trep is smaller.

[Modification 2]
Although the occupant state estimation device 10 has disclosed the aspect in which the degree of appropriateness is evaluated in three or more stages or is evaluated with continuous values, the present invention is not limited to this. The occupant state estimation device 10 may determine whether or not the state of the driver's seat occupant is appropriate as the state during automatic driving.

[Modification 3]
In the above, the mode in which the appropriateness is determined from the two parameters of the miss rate Rn and the average reaction time Trep is exemplified, but the present invention is not limited to this. The appropriateness determination unit F7 may determine the appropriateness only from the miss rate Rn without using the average reaction time Trep. Further, instead of the miss rate Rn, the appropriateness may be determined using the number of misses.

  Furthermore, the appropriateness determination unit F7 may determine the appropriateness only from the average reaction time Trep without using the miss rate Rn. In addition, the appropriateness may be determined using only the latest reaction time instead of the average reaction time Trep.

[Modification 4]
In the above, as an embodiment of the occupant state estimation device 10, the mode of determining the degree (that is, the degree of appropriateness) of an appropriate state as the state of the driver's seat occupant at the time of automatic driving is exemplified, but the present invention is not limited thereto. The occupant state estimation device 10 may be a device that determines a degree (that is, a degree of appropriateness) that is an appropriate state as a state during manual driving. The appropriateness at the time of manual driving may be determined based on whether or not the driver's seat occupant has visually recognized the direction in which the virtual sound source is localized (that is, the localization direction).

  Also, depending on each operation mode, the degree of appropriateness as the state during automatic operation is determined during automatic operation, while the state during the manual operation is estimated as to whether the state is appropriate as the state during manual operation. The operation mode may be switched. Note that the automatic operation corresponds to a state where the automatic operation function is activated, and the manual operation corresponds to a state where the automatic operation function is not activated.

  Moreover, although the passenger | crew state estimation apparatus 10 illustrated the aspect applied to the vehicle provided with the automatic driving function above, it is not restricted to this. The present invention may be applied to a vehicle that does not have an automatic driving function.

[Modification 5]
In the above description, the mode in which the presence of the moving body is notified to the driver's seat occupant by generating a three-dimensional sound image when there is a moving body to be visually recognized by the driver's seat occupant is not limited to this. When there is a stationary object to be visually recognized by the driver's seat occupant, a three-dimensional sound image may be generated and the presence of the stationary object may be left in the driver's seat occupant. That is, the target of notification by a three-dimensional sound image is not limited to a moving object, and may be a stationary object.

  For example, when traveling on an expressway, the presence of various signs provided on the expressway may be notified by generating a three-dimensional sound image. According to such a configuration, it is possible to periodically determine the appropriateness of the driver's seat occupant even when a safe traveling can be maintained without causing a collision risk. The sign provided on the expressway is, for example, an entrance / exit guide sign for guiding the entrance / exit of the expressway, a route guide sign, a sign indicating the remaining distance to the service area and the parking area, and the like. Of course, the stationary object that triggers the generation of the three-dimensional sound image is not limited to the sign but may be a falling object on the road.

[Modification 6]
In the above, the occupant state estimation device 10 has exemplified the mode of estimating the state of the driver's seat occupant, but is not limited thereto. The occupant state estimation device 10 may estimate the state of passengers in the passenger seat or the occupant state of the rear seat (specifically, appropriateness or the like). Moreover, you may respond | correspond to several seats, such as estimating each state of a driver's seat passenger and a passenger seat passenger.

100 occupant state estimation system, 10 occupant state estimation device, 11 CPU, 12 RAM, 13 ROM, 14 I / O, 21 camera unit, 22 millimeter wave radar, 23 inter-vehicle communication unit, 24 wide area communication unit, 25 DSM, 26 Speaker, 26R right speaker, 26L left speaker, 27 display, 28 vibrator, 29 air conditioner, 30 automatic operation ECU, F1 peripheral information acquisition unit, F2 moving body information extraction unit, F3 approaching body extraction unit (risk direction identification unit), F31 risk evaluation unit, F4 three-dimensional sound image generation unit, F5 visual recognition direction identification unit, F6 visual recognition determination unit, F61 reaction time measurement unit, F62 recording processing unit, F7 appropriateness determination unit (occupant state estimation unit), F8 appropriateness prediction unit (Prediction unit), F9 stimulus application unit, M1 sound data storage unit (pseudo sound data storage unit), M Result storage unit

Claims (15)

Used in vehicles,
A three-dimensional sound image generation unit (F4) that generates a three-dimensional sound image in which a virtual sound source is localized at a desired position in a three-dimensional space by outputting sound signals from each of the two or more sound signal output devices;
A viewing direction identifying unit (F5) that identifies a viewing direction that is a direction in which an occupant seated in a predetermined seat is viewing based on a captured image of a camera provided in a vehicle interior;
Based on the identification result of the visual recognition direction identification unit, it is determined whether or not the occupant has viewed a localization direction that is a direction in which the virtual sound source is localized within a predetermined reaction waiting time after the stereoscopic sound image is generated. A visual recognition determination unit (F6);
An occupant state estimation unit (F7) that estimates the state of the occupant based on the determination result of the visual recognition determination unit;
A peripheral information acquisition unit (F1) that acquires the peripheral information from a peripheral information detection device that acquires peripheral information indicating traffic conditions around the vehicle;
A risk direction specifying unit (F3) for detecting the presence of an object that can come into contact with the vehicle based on the peripheral information acquired by the peripheral information acquiring unit, and specifying a direction in which the object exists ;
The peripheral information acquisition unit acquires, as the peripheral information, a moving body type of a moving body existing around the vehicle, a relative speed with respect to the vehicle, and a relative position with respect to the vehicle,
The risk direction specifying unit specifies a direction in which an approaching body that is the moving body in which the distance to the vehicle tends to decrease is based on the peripheral information acquired by the peripheral information acquisition unit,
A risk evaluation unit (F31) for evaluating a collision risk for each approaching body based on the peripheral information;
Further comprising an onomatopoeia data storage unit (M1) for storing onomatopoeia data for outputting an onomatopoeia corresponding to a mobile body type as the acoustic signal for each mobile body type,
The surrounding information includes at least a moving body type for each moving body existing around the vehicle,
The three-dimensional sound image generator is
The occupant is informed of the presence of the approaching body by generating the three-dimensional sound image so that the localization direction in which the virtual sound source is located for the occupant matches the direction specified by the risk direction specifying unit. And
Among the approaching bodies, for the approaching body that is evaluated by the risk evaluation unit to be less than a predetermined notification threshold, the approaching body does not generate the three-dimensional sound image corresponding to the moving body. Among these, when there is the approaching body evaluated by the risk evaluation unit as the collision risk is equal to or higher than a predetermined notification threshold, the direction in which the approaching body exists and the localization direction of the virtual sound source coincide with each other To generate the three-dimensional sound image,
When the approaching body that is evaluated to be equal to or higher than a predetermined notification threshold is present, the stereoscopic sound image is generated with an imitation sound according to the moving body type of the approaching body, and
According to the collision risk evaluated by the risk evaluation unit for the approaching body that is about to notify the occupant of the presence of the three-dimensional sound image, the blowing mode is changed without changing the type of the pseudo sound of the three-dimensional sound image. occupant state estimating apparatus according to claim to Rukoto. Oite to claim 1,
The occupant state estimation unit determines whether or not the occupant state is an appropriate state designed in advance as a state of a driver's seat occupant during automatic driving. Oite to claim 1,
The occupant state estimation unit sequentially determines an appropriate degree of a state as a driver's seat occupant. In claim 3 ,
A stimulus application unit (F9) for controlling the operation of a stimulation device that stimulates at least one of the five senses of the occupant;
Prediction for predicting the timing at which the appropriateness level shifts to an inappropriate level designed in advance as the state of the driver's seat occupant during automatic driving, based on the time change of the appropriateness level sequentially determined by the occupant state estimating unit Part (F8),
The occupant state estimation device, wherein the stimulation application unit operates the stimulation device when the prediction unit is predicted to decrease to an inappropriate level within a predetermined remaining time. In any one of Claim 2 to 4 ,
The visual recognition direction specifying unit is configured to specify the visual recognition direction of the occupant seated in each of a plurality of seats,
The visual recognition determination unit determines, for each occupant, whether or not the virtual sound source has been localized.
The said occupant state estimation part estimates the state for every said occupant based on the determination result by the said visual recognition determination part, The occupant state estimation apparatus characterized by the above-mentioned. Used in vehicles,
A three-dimensional sound image generation unit (F4) that generates a three-dimensional sound image in which a virtual sound source is localized at a desired position in a three-dimensional space by outputting sound signals from each of the two or more sound signal output devices;
A viewing direction identifying unit (F5) that identifies a viewing direction that is a direction in which an occupant seated in a predetermined seat is viewing based on a captured image of a camera provided in a vehicle interior;
Based on the identification result of the visual recognition direction identification unit, it is determined whether or not the occupant has viewed a localization direction that is a direction in which the virtual sound source is localized within a predetermined reaction waiting time after the stereoscopic sound image is generated. A visual recognition determination unit (F6);
An occupant state estimation unit (F7) that estimates the state of the occupant based on the determination result of the visual recognition determination unit ,
The occupant state estimation unit sequentially determines the appropriate degree of state as a driver's seat occupant,
A stimulus application unit (F9) for controlling the operation of a stimulation device that stimulates at least one of the five senses of the occupant;
Prediction for predicting the timing at which the appropriateness level shifts to an inappropriate level designed in advance as the state of the driver's seat occupant during automatic driving, based on the time change of the appropriateness level sequentially determined by the occupant state estimating unit Part (F8),
The stimulus applying unit, the if it is expected to drop to inappropriate levels within a predetermined residual hours by the prediction unit, the occupant state estimating apparatus according to claim Rukoto to operate the stimulator. In claim 6 ,
The visual recognition direction specifying unit is configured to specify the visual recognition direction of the occupant seated in each of a plurality of seats,
The visual recognition determination unit determines, for each occupant, whether or not the virtual sound source has been localized.
The said occupant state estimation part estimates the state for every said occupant based on the determination result by the said visual recognition determination part, The occupant state estimation apparatus characterized by the above-mentioned. Used in vehicles,
A three-dimensional sound image generation unit (F4) that generates a three-dimensional sound image in which a virtual sound source is localized at a desired position in a three-dimensional space by outputting sound signals from each of the two or more sound signal output devices;
A viewing direction identifying unit (F5) that identifies a viewing direction that is a direction in which an occupant seated in a predetermined seat is viewing based on a captured image of a camera provided in a vehicle interior;
Based on the identification result of the visual recognition direction identification unit, it is determined whether or not the occupant has viewed a localization direction that is a direction in which the virtual sound source is localized within a predetermined reaction waiting time after the stereoscopic sound image is generated. A visual recognition determination unit (F6);
An occupant state estimation unit (F7) that estimates the state of the occupant based on the determination result of the visual recognition determination unit ,
The visual recognition direction specifying unit is configured to specify the visual recognition direction of the occupant seated in each of a plurality of seats,
The visual recognition determination unit determines, for each occupant, whether or not the virtual sound source has been localized.
The occupant state estimation unit determines whether or not the state for each occupant is an appropriate state designed in advance as the state of the driver's seat occupant during automatic driving based on the determination result by the visual recognition determination unit . An occupant state estimation device. Used in vehicles,
A three-dimensional sound image generation unit (F4) that generates a three-dimensional sound image in which a virtual sound source is localized at a desired position in a three-dimensional space by outputting sound signals from each of the two or more sound signal output devices;
A viewing direction identifying unit (F5) that identifies a viewing direction that is a direction in which an occupant seated in a predetermined seat is viewing based on a captured image of a camera provided in a vehicle interior;
Based on the identification result of the visual recognition direction identification unit, it is determined whether or not the occupant has viewed a localization direction that is a direction in which the virtual sound source is localized within a predetermined reaction waiting time after the stereoscopic sound image is generated. A visual recognition determination unit (F6);
An occupant state estimation unit (F7) that estimates the state of the occupant based on the determination result of the visual recognition determination unit,
The visual recognition direction specifying unit is configured to specify the visual recognition direction of the occupant seated in each of a plurality of seats,
The visual recognition determination unit determines, for each occupant, whether or not the virtual sound source has been localized.
The occupant state estimating unit is configured to determine the proper degree of state of the driver seat occupant sequentially based on a determination result of the visual judgment unit, occupant state estimating characterized that you estimate the state of each of the occupant apparatus. In any one of Claim 6 to 9 ,
A peripheral information acquisition unit (F1) that acquires the peripheral information from a peripheral information detection device that acquires peripheral information indicating traffic conditions around the vehicle;
A risk direction specifying unit (F3) for detecting the presence of an object that can come into contact with the vehicle based on the peripheral information acquired by the peripheral information acquiring unit, and specifying a direction in which the object exists;
The three-dimensional sound image generation unit generates the three-dimensional sound image so that a localization direction that is a direction in which the virtual sound source is located for the occupant coincides with a direction estimated by the risk direction specifying unit. Estimating device. In claim 10 ,
The peripheral information acquisition unit acquires, as the peripheral information, a moving body type of a moving body existing around the vehicle, a relative speed with respect to the vehicle, and a relative position with respect to the vehicle,
The risk direction specifying unit specifies a direction in which an approaching body that is the moving body in which the distance to the vehicle tends to decrease is based on the peripheral information acquired by the peripheral information acquisition unit,
A risk evaluation unit (F31) for evaluating a collision risk for each approaching body based on the peripheral information;
The three-dimensional sound image generator is
Informing the occupant of the presence of the approaching object by generating the stereoscopic sound image so that the virtual sound source is localized in the direction in which the approaching object exists,
Among the approaching objects, the three-dimensional sound image corresponding to the moving body is not generated for the approaching body evaluated by the risk evaluation unit as the collision risk being less than a predetermined notification threshold. Crew state estimation device. In claim 11 ,
The three-dimensional sound image generation unit has a direction in which the approaching body is present when the approaching body is evaluated by the risk evaluation unit as being more than a predetermined notification threshold. The occupant state estimation device characterized in that the three-dimensional sound image is generated so that the localization direction of the virtual sound source matches. In claim 12 ,
The three-dimensional sound image generator is
The occupant state characterized by changing the output mode of the three-dimensional sound image according to the collision risk evaluated by the risk evaluation unit with respect to the approaching body that is about to notify the occupant of the presence of the three-dimensional sound image. Estimating device. In any one of Claims 10 to 13 ,
An onomatopoeia data storage unit (M1) for storing onomatopoeia data for outputting an onomatopoeia according to a moving body type for each moving body type;
The surrounding information includes at least a moving body type for each moving body existing around the vehicle,
The three-dimensional sound image generator is
An occupant state estimation device characterized in that, when there is a moving body that satisfies a predetermined notification condition, the three-dimensional sound image is generated with a pseudo sound corresponding to a moving body type of the moving body that satisfies the notification condition. . In any one of Claims 1-14,
A reaction time measuring unit (F61) for measuring a reaction time from when the three-dimensional sound image is generated until the occupant turns his line of sight toward the localization direction;
A recording processing unit (F62) that records the determination result of the visual recognition determination unit and the reaction time measured by the reaction time measurement unit in a result storage unit realized using a predetermined storage medium;
The occupant state estimation unit is at least one of the reaction time recorded in the recording processing unit and the number of times the occupant looks at the direction of the virtual sound source with respect to the generation of the stereoscopic sound image a predetermined number of times. An occupant state estimating device that estimates the state of the occupant based on the above.

Images