JP7432198B2 - Situation awareness estimation system and driving support system - Google Patents

Description

The present invention relates to a situation recognition estimation system and a driving support system, and more specifically, a situation in which it is estimated whether or not a target person appropriately recognizes a recognition target object that moves relatively around a predetermined target person. The present invention relates to a recognition estimation system and a driving support system that uses the situation recognition estimation system to assist a target person in driving a mobile object such as a car.

In order to drive a car safely and efficiently, a driver's correct situational awareness is extremely important. The situational awareness that a driver should perform varies greatly depending on driving tasks such as right turns and merging, how crowded the vehicles are, and the presence of surrounding vehicles and pedestrians. Situational awareness here is not only the perception of knowing the state and characteristics of elements existing in the environment, but also the understanding of integrating the perceived elements and grasping the current event, and the understanding of the future based on the understood information and one's own knowledge and experience. It is sufficient to perform the prediction step by step to estimate the movement of the .

By the way, Patent Document 1 discloses a driving support device that, when the driver overlooks a direction that the driver should originally check, transmits the visual recognition in that direction to the driver. This driving support device uses GPS signals to obtain location information of the vehicle being driven by the target driver, and also obtains road information such as the type of branch the vehicle is traveling on based on pre-stored map information. Based on the road information and the direction of travel according to the searched route, the direction in which the driver should visually observe is determined at this point. Then, the direction of the driver's line of sight is acquired from the captured image of the driver, and an overlooked direction in which attention should be called is identified based on whether or not the driver is visually recognized in the required visual direction within a predetermined period of time.

Patent No. 6419401

However, the driving support device of Patent Document 1 only determines whether the driver's line of sight is directed in the direction that requires visual inspection, and does not recognize people or objects that actually exist in the direction that requires visual inspection. There is no consideration as to whether or not the object was visually recognized, and it is not possible to estimate whether or not the above-mentioned situational awareness is sufficient. That is, for example, as the traffic situation around the host vehicle changes over time, a recognition target such as another vehicle to be visually recognized may appear or disappear in the direction in which visual recognition is required within the predetermined time period. In such a case, the driving support device cannot specify whether or not the driver actually visually recognized the direction in which visual observation is required at an appropriate timing when the recognition target object appears. In addition, in the driving support device, even if the driver actually visually recognizes the direction in which visual recognition is required at the timing when the recognition target object appears, the time necessary for the above-mentioned understanding and prediction to sufficiently recognize the situation is required. It is not possible to determine whether or not the person is being visually recognized. In other words, in the driving support device, if the driver turns his/her eyes in the direction that requires visual observation even once, the driver does not visually recognize the recognition target existing in the direction that requires visual visibility at all, or the recognition target exists for a sufficient period of time. Even when the vehicle is not visually recognized, it is determined that there is no oversight in the necessary direction, and no alert is issued. Therefore, the driving support device of Patent Document 1 cannot estimate whether or not the above-mentioned situation recognition is being performed appropriately, and cannot necessarily be said to be sufficient as a driving evaluation or driving support for the driver regarding safe driving.

The present invention was devised to solve such problems, and its purpose is to enable a subject moving in a predetermined environment to appropriately recognize the status of recognition objects existing around him/her. An object of the present invention is to provide a situation recognition estimation system and a driving support system that can estimate whether or not the vehicle is in a vehicle.

In order to achieve the above object, the present invention mainly provides a situation recognition estimation system for estimating whether or not the target person appropriately recognizes a recognition target that moves relatively around the target person. information acquisition means for acquiring various information related to the operation of the target person and the surrounding environment of the target person; and visibility situation grasping means for determining the visibility status of the recognition target by the target person based on the information from the information acquisition means. , comprising a task specifying means for specifying a task to be performed by the target person in the near future based on the information, and a recognition determining means for determining the situational awareness of the target person in the task based on the visual recognition situation, The situation grasping means includes a viewing area specifying unit that classifies the target person's line of sight direction into a plurality of viewing areas, and determining the existence of the recognition target for each viewing area based on the classification results of the viewing area specifying unit. and a gaze data generation unit that generates time-series gaze data by associating the visual recognition situation at each time, and the recognition determining means stores a gaze model corresponding to reference situation recognition for each task. , a determination unit that determines from the line-of-sight data and the line-of-sight model whether or not the situation with respect to the recognition target object is properly recognized; the determination unit determines whether the target person should visually recognize the recognition target object; The situation recognition is determined by determining whether the timing of viewing the recognition target object in each of the viewing areas and the viewing time of viewing the recognition target object are appropriate. .

Further, in the present invention, the situational awareness estimation system is applied, the target person is a driver of a moving object, and the situational awareness estimation system provides information to the driver based on the estimation result of the situation recognition around the moving object. A driving support system that provides driving support includes support means that provides support to the driver when the recognition determining means determines that the driver is not properly recognizing the driving situation. are taken.

According to the present invention, the presence status of the recognition target in each visibility area is made to correspond to the visibility status of each visibility area by the subject at each time using line of sight data. It becomes possible to estimate whether the target person is visually recognizing it. In addition, as a line-of-sight model, we set the visibility area that should be seen for each task or each phase that makes up the task, and set the reference time that is the visibility time necessary for understanding and predicting the situation to sufficiently recognize it. By setting this, it is possible to know whether the target person is viewing the relevant viewing area at the timing when the recognition target object actually exists, and whether or not the viewing time at this time has reached the reference time. Become. This makes it possible to estimate whether or not the subject is appropriately recognizing the situation.

Furthermore, by using the situational awareness estimation system of the present invention to estimate the situational awareness of the surroundings of a driver of a moving object, it becomes possible to objectively evaluate the situational awareness ability of the driver. By feeding back the evaluation contents to the driver, etc., it will contribute to improving the driver's driving ability from the viewpoint of accident prevention. In addition, during actual driving, it is possible to estimate in real time the situational awareness that the driver lacks in each driving task, so if the situational awareness is insufficient, the driver can be reminded to look again by alerting the driver, etc. It becomes possible to provide various types of support, such as prompting or issuing operational commands to devices and systems to prevent accidents caused by a decline in situational awareness. Therefore, it is possible to provide support for safe driving as if there were a driving assistant beside the driver.

FIG. 1 is a block diagram showing the overall configuration of a driving support system according to the present embodiment. FIG. 3 is a conceptual diagram for explaining a visual recognition area. FIG. 3 is a schematic diagram showing the traffic situation in the first case. FIG. 3 is a diagram conceptually representing the configuration of line-of-sight data in a first case using a graph. It is a schematic diagram showing the traffic situation in a second case. FIG. 7 is a diagram conceptually representing the configuration of line-of-sight data in a second case using a graph. It is a schematic diagram showing the traffic situation in a third case. FIG. 7 is a diagram conceptually representing the configuration of line-of-sight data in a third case using a graph.

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 shows a block diagram showing the overall configuration of a driving support system according to this embodiment. In this figure, the driving support system 10 includes a situation for estimating whether or not a subject moving in a predetermined environment appropriately recognizes the situation of objects to be recognized such as people and objects existing around the subject. A recognition estimation system 11 is applied. In this driving support system 10, the target person is a car driver, and it is determined whether or not the driver's awareness of the situation while driving is appropriate. Support is now available. Note that, in the following description, the vehicle driven by the driver who is the subject of situation recognition will be referred to as the "self-vehicle", and the vehicles etc. located around the vehicle will be referred to as "other vehicles".

That is, this driving support system 10 includes a situation recognition estimation system 11 that estimates whether or not the driver of the vehicle is appropriately aware of the situation while driving; and a support means 12 that provides support to the driver when it is estimated that the driver is not doing so properly.

The situation recognition estimation system 11 includes an information acquisition means 14 that acquires various information related to the driver's line of sight, actions such as driving operations, and the surrounding environment around the own vehicle, and information acquisition means 14 that acquires various information related to the driver's line of sight, actions such as driving operations, and the surrounding environment around the own vehicle, and information acquisition means 14 that acquires various information related to the driver's line of sight, actions such as driving operations, and the surrounding environment around the own vehicle, and information acquisition means 14 that acquires various information related to the driver's line of sight and actions such as driving operations, and the surrounding environment around the own vehicle, A visibility situation grasping means 15 for grasping, a task identification means 16 for identifying a driving task to be performed by the driver in the near future, and a recognition means for determining the situational awareness of the driver based on the driver's visibility situation in the driving task. determination means 17.

The information acquisition means 14 includes an information acquisition device 19 including a predetermined sensor, camera, etc. attached to the own vehicle, and a detection section 20 that detects various information from signals from the information acquisition device 19.

The information acquisition device 19 includes a range sensor 22 such as a lidar for detecting surrounding vehicle information representing the presence and relative positional relationship of objects to be recognized, such as people and objects, existing around the own vehicle, and A vehicle exterior imaging camera 23 is provided to capture an image of the surrounding space within a certain range, an in-vehicle imaging camera 24 is provided to capture an image of the interior space of the vehicle including the driver's face, and A rain sensor 25 for measuring the amount of rainfall around the vehicle, a light sensor 26 for measuring the brightness around the vehicle, a GPS transceiver 27 for using GPS, and externally generated map information. The vehicle includes a data communication device 28 capable of receiving road traffic information such as road traffic information and traffic jam information, and a driving operation state detector 29 that detects the driver's operating state of the own vehicle. This driving operation state detector 29 detects the steering angle corresponding to the blinking operation of the turn signal (blinker) by the driver, the operation of the steering wheel by the driver, and the operation of the brake or accelerator during driving by the driver. It consists of various sensors that detect changes in speed, acceleration, etc.

The detection unit 20 includes a vehicle position information detection unit 31 that uses a GPS transceiver 27 to identify current position information of the vehicle, and a surrounding environment information detection unit that detects various external environment information around the vehicle. 32, an own vehicle operation information detection unit 33 that detects information on the driver's operation of the own vehicle based on data from the driving operation state detector 29, and image data of the driver's face obtained by the in-vehicle imaging camera 24. The driver's line-of-sight information detection unit 34 detects line-of-sight information regarding the driver's line-of-sight direction using a known image processing method using machine learning.

Note that, although the detection unit 20 is not particularly limited, the detection unit 20 may be placed within the own vehicle, or may be configured such that data communication is possible with the information acquisition device 19 attached to the own vehicle. , it may be placed in an external server or the like that is remote from the own vehicle.

The surrounding environment information detection unit 32 obtains the following first to sixth surrounding environment information as the external environment information.

As the first surrounding environment information, road information regarding whether the road on which the host vehicle is traveling is an intersection or a single road is acquired together with the current position of the host vehicle. In other words, this road information can be created by acquiring or newly creating map information for the vehicle traveling from the departure point to the destination, and combining it with the vehicle's current position information from GPS, so that it can be used both now and in the foreseeable future. required for roads.

As the second surrounding environment information, from the image data of the space in front of the vehicle acquired by the external imaging camera 23, using a known image processing method, white lines dividing road lanes, white stop lines, and black traffic areas are calculated. Lane information that distinguishes between the two is required. That is, this lane information is information specifying the number of lanes of the road on which the vehicle is traveling, the presence of a stop line in front of the vehicle, and the lane in which the vehicle is currently traveling.

As the third surrounding environment information, from the image data of the space in front of the vehicle acquired by the external imaging camera 23, a known image processing method is used to determine the presence and color of traffic lights in front of the vehicle. Signal sign information consisting of the type of road sign existing in front of the vehicle is required. Although not particularly limited, this signal sign information is obtained by a known object detection method such as R-CNN (Regions with Convolutional Neural Networks).

As the fourth surrounding environment information, surrounding vehicle information including the traveling directions and relative positions of other vehicles existing around the own vehicle is obtained. Here, the presence and traveling direction of another vehicle are determined from image data around the vehicle acquired by the external imaging camera 23, and the relative distance to the other vehicle is determined from the detection results of the range sensor 22.

As the fifth surrounding environment information, weather information including the amount of rainfall, brightness, etc. of the space surrounding the own vehicle is obtained. That is, here, the amount of rainfall in the space around the host vehicle is determined from the detection result of the rain sensor 25, and the brightness of the space around the host vehicle is determined from the detection result of the optical sensor 26.

As the sixth surrounding environment information, surrounding pedestrian information including the running direction and relative position of pedestrians existing around the own vehicle is obtained. Here, the presence of a pedestrian and the traveling direction are determined from the image data around the vehicle acquired by the external imaging camera 23, and the relative distance to the pedestrian is determined from the detection result of the range sensor 22.

The own vehicle operation information detection unit 33 detects the driver's direction indicator operation, the steering angle by the driver's steering wheel operation, the amount of brake and accelerator operation, and the speed change during driving corresponding to the amount of operation, and the driver's operation information. Vehicle operation information such as changes in vehicle acceleration is sequentially detected.

Next, the visual recognition situation grasping means 15, the task specifying means 16, and the recognition determining means 17 will be explained.

Each of these means 15 to 17 is executed by a computer comprising an arithmetic processing unit such as a CPU and a storage device such as a memory or a hard disk. Although this computer is not particularly limited, it may be placed inside or outside the own vehicle, or the computer may be placed inside or outside the own vehicle, and some means may be executed inside the own vehicle. It is also possible to adopt a mode in which the remaining means are executed outside the own vehicle.

The visibility situation grasping means 15 includes a visibility area specifying unit 37 that classifies the direction of the driver's line of sight into a plurality of visibility areas based on the driver's line of sight information detected by the driver's line of sight information detection unit 34, and a visibility area specifying unit. Based on the classification results in step 37, time-series line-of-sight data is generated for each visibility area by associating the presence of a recognition target within a predetermined space from the own vehicle with the driver's visibility status at each time. The line of sight data generation section 38 is also provided.

The visual recognition area specifying unit 37 identifies which visual recognition area the driver's line of sight direction at each time is in the direction of, based on the driver's line of sight information acquired at each time. Although not particularly limited, in this embodiment, as shown in FIG. It is designed to be sorted into either A7. That is, the visibility areas in this embodiment include a front area A1 in front of the vehicle that is visible through the windshield, a rearview mirror area A2 directly behind that is visible through the rearview mirror, and a rearview area A2 that is visible through the rearview mirror. A left window area A3 on the left side that is visible to the driver, a left rear left side mirror area A4 that is visible through the side mirror on the left side, and a right window area A5 on the right side that is visible through the window on the right side. A right rear right side mirror area A6 that is visible through the right side mirror and a speedometer area A7 located at the front of the vehicle are set.

The line-of-sight data generation unit 38 generates line-of-sight data that is time-series data for each visible area. This line-of-sight data includes position information of objects to be recognized such as other vehicles, pedestrians, traffic lights, and signs identified from the detection results of the surrounding environment information detection unit 32 and of the driver identified by the visibility area identification unit 37. The line of sight direction is made to correspond at each time. In other words, the line-of-sight data is generated by dividing the presence status of recognition objects around the own vehicle at each time into visibility areas, and superimposing the visibility areas where the driver's line of sight exists at each time. Therefore, based on this line of sight data, it is possible to estimate whether the driver was able to visually recognize the recognition target existing in each viewing area at the same time.

The task specifying means 16 is configured to specify the driving task that the driver will perform from now on based on the operating state of the own vehicle detected by the own vehicle operation information detection section 33. Examples of driving tasks here include turning left and right, changing lanes, going straight, driving on left and right curves, merging, parking, overtaking, stopping, and starting, and each pattern of driving tasks is stored in advance. For example, the next driving task is selected from among the stored driving tasks by the driver's blinking operation of the direction indicator. First, if the turn signal is operated and there is an intersection in the immediate vicinity of the direction of travel of the vehicle based on the road information detected by the surrounding environment information detection unit 32, it is determined that the vehicle is in the direction in which the direction signal flashes. A driving task of turning right or turning left is selected. Furthermore, if the turn signal is blinked and the vehicle is traveling on a single road most recently based on the road information, a driving task in which the vehicle changes lanes is selected. On the other hand, if the direction indicator blinking operation is not performed, a driving task in which the own vehicle moves straight is selected.

The recognition determining means 17 includes a line-of-sight model generation unit 41 that generates a driver's line-of-sight model corresponding to appropriate situation recognition as a reference for each driving task, and a line-of-sight model generated by the line-of-sight model generation unit 41 is stored. , a determination unit 42 that determines whether the driver's situation recognition is appropriate based on the line-of-sight model and the actual driver's line-of-sight data generated by the line-of-sight data generation unit 38.

The line-of-sight model generation unit 41 generates a line-of-sight model for each driving task by statistically processing a large number of past line-of-sight data for which situation recognition was deemed appropriate in the past.

The line of sight model determines, for each driving task, the visibility area in which the driver should see the recognition target and the standard of the viewing time for the recognition target to be seen in the vicinity of the driver's vehicle, in units of phases that make up the driving task. A reference time, which is a value, and a visual confirmation interval, which indicates the time interval at which the driver should check the surrounding situation during driving tasks such as driving straight ahead, are each set.

The reference time is a time corresponding to the driver's visual recognition time required for situation recognition, and is based on various information actually acquired during the determination by the determination unit 42 with respect to the temporarily set default time. It is updated based on data on That is, here, the relevant data includes optical elements that affect the transmission of light in the surrounding space of the own vehicle, positional elements that affect the visual accessibility of the driver to the visibility area, and/or the same timing. An example of this is the number of times the driver visually recognized the target recognition object. In this embodiment, the standard time is calculated by multiplying the default time by the optical element term, the positional element term, and the visibility count term, which are parameters corresponding to the optical element, the positional element, and the visibility count, respectively, as shown in the following equation. Desired.
Standard time = Optical element term × Position element term × Number of visibility terms × Default time

The optical element term is obtained by multiplying the rain element term based on the amount of rainfall obtained from the rain sensor 25 and the brightness element term based on the brightness obtained from the optical sensor 26. The rainfall element term is obtained by multiplying the amount of rainfall acquired at the time of determination by the determination unit 42 by a preset rainfall amount coefficient, and is proportional to the amount of rainfall. Further, the brightness element term is obtained by dividing the brightness obtained at the time of determination by the determination unit 42 by a preset brightness coefficient, and is inversely proportional to the brightness.

The position element term is composed of an area coefficient set for each visible area according to the position of the visible area where the recognition target object to be visually recognized exists. For example, the driver's visibility differs in each visibility area, such as the left window area A3 being harder to see than the front area A1, so the area coefficient is calculated based on the driver's visibility in each visibility area. This is a coefficient for weighting according to the visibility, and the harder it is to see the visible area, the higher the value is taken.

The visibility number term is composed of a visibility frequency coefficient that is determined according to the number of visibility. For example, in the same phase, if the number of times a recognition target is viewed in a certain viewing area is divided into two times, there will be a time difference between the first and second viewings. For this reason, the total time of these two times needs to be a reference time that is different from the time in the case of only one visual recognition, and a visual recognition frequency coefficient that is determined according to the number of visual recognitions is set.

When generating the above line-of-sight model, it is necessary to set the visibility area in which the driver should visually recognize the recognition target object and the various coefficients described above for each phase of each driving task. In these settings, information is acquired regarding various driving tasks actually performed by various drivers in the past, when the relevant driving tasks were performed with appropriate situational awareness, that is, when it was determined that the driving was safe. It is obtained by collecting the line-of-sight data and performing statistical processing.

Here, whether or not the actual driving by the driver was safe driving is automatically determined based on the detection result obtained by the information acquisition means 14. That is, examples of safe driving include no accidents, no violations, and a smooth movement trajectory of the own vehicle, and it is determined whether or not driving is safe based on data acquired before and after the driving task. For example, if the speed of your vehicle suddenly decreases even though the operating state of the accelerator pedal remains constant, this indicates that some type of accident, such as a collision with another vehicle, has occurred, and the information obtained in that case It is determined that the driver's line of sight data is not for safe driving. In addition, if the acceleration suddenly increases even though the brake pedal is being operated, it is likely that some sort of accident has occurred, such as your vehicle being rear-ended by another vehicle. It is determined that the driver's line-of-sight data obtained is not for safe driving. In other words, here, the input value related to the operation of the own vehicle by the driver is compared with the output value from the own vehicle, and when the expected output value is not obtained, the line of sight data at that time is It is determined that the data is not for safe driving. Therefore, the line-of-sight model generation unit 41 extracts only the other line-of-sight data during safe driving and statistically processes it. Here, the above-mentioned various coefficients are determined using a statistical method such as multiple regression analysis using a large number of collected line-of-sight data during safe driving.

Further, the visual recognition interval is determined based on line-of-sight data during past safe driving during driving tasks such as driving straight ahead, and based on the average interval at which the driver looks at the room mirror, side mirror, and window.

The determination unit 42 determines situation recognition as follows. That is, first, for the driving task that the driver specified by the task specifying means 16 is about to perform, a line-of-sight model related to the driving task is extracted and compared with the actually obtained line-of-sight data. At this time, it is determined whether the driver is viewing the appropriate viewing area set by the line of sight model at an appropriate timing and for an appropriate viewing time. In other words, it is determined whether or not the driver has been able to visually recognize the recognition target object that is present in the visible area to be visually recognized in each phase of the driving task for a sufficient viewing time. The visibility time here is determined based on the above-mentioned standard time, which is determined according to the amount of rainfall and brightness of the space around the vehicle, the position of the target visibility area, and the number of times of visibility for each phase in each visibility area. be done. In other words, if the viewing time for any recognition target existing in the visibility area that should be visually recognized does not reach the reference time, it is presumed that the situation of the recognition target in the relevant driving task is not properly recognized. , if not, it is presumed that situational awareness is appropriate.

When it is determined that the situation recognition in the target driving task is not appropriate, the support means 12 prompts the driver to take the correct recognition action at an appropriate timing, or sufficiently performs the situation recognition that has been determined to be insufficient. In this way, driving support information to alert the driver is transmitted through a predetermined transmission device.

The transmission device here is not particularly limited, and can be any device such as a display, speaker, vibration generator, etc. that can transmit driving support information through the driver's visual, auditory, and/or tactile senses. Various other human interfaces are also applicable as long as various types of driving support information can be transmitted through various senses.

Note that the support provided by the support means 12 includes, in addition to the above-mentioned warning and the like to urge the driver to re-visit the vehicle, operational commands to devices and systems to prevent accidents caused by a decline in situational awareness, etc. Various types of support are available.

Next, the situation recognition evaluation by the driving support system 10 and the driving support procedure based on the evaluation will be described below using cases set as examples for several driving tasks.

First, as a first case, assuming the traffic situation shown in Fig. 3, for a driving task in which the own vehicle M changes lanes to the right lane, situation recognition evaluation and driving support are performed as follows. It will be done.

First, the traffic situation in this case is as follows. The host vehicle M is currently traveling in the center lane of a three-lane road, and in the same lane, another vehicle A is running in front of the host vehicle M, and another vehicle B is running behind the same lane. Further, in the left lane located on the left side of the center lane, another vehicle C is running diagonally to the left rear of the host vehicle M. Furthermore, in the right lane (passing lane) located on the right side of the center lane, another vehicle D and another vehicle E are sequentially running at a higher speed than the own vehicle M from diagonally right behind the own vehicle M. Under this traffic situation, there is a scenario in which the driver of own vehicle M moves between another vehicle D and another vehicle E while changing lanes to the right lane, as shown by the dashed line in FIG. It has become.

Here, it is assumed that the current position information of the own vehicle M, various environmental information around the own vehicle M, and operation information of the driver regarding the own vehicle M are acquired by the information acquisition means 14 at regular intervals and are shown in the figure. Not stored in memory.

Then, the line-of-sight situation understanding means 15 sequentially generates the driver's line-of-sight data from the various information acquired by the information acquisition means 14 and stores it in the memory. The line of sight data in this case has a configuration schematically illustrated in FIG. 4. That is, in this figure, line-of-sight data is expressed as a graph in which the vertical axis in the figure is the visible areas A1 to A6 and the horizontal axis is the time t. , the visibility time period Tm (hatched portion in the figure) in which the driver's line of sight exists and the time periods Ta to Te in which other vehicles A to E exist are identified as being superimposed. This line of sight data is generated within a target range R (dashed line in FIG. 3) that is determined according to the speed of the own vehicle M and the legal speed of the road. This target range R is within a predetermined distance around the own vehicle M in the front, rear, left and right directions.

Next, the task specifying means 16 specifies the driving task. In the first case, based on the information acquired by the information acquisition means 14, when it is detected that the turn signal flashes in the direction of a right turn and that the vehicle is traveling on a single road that is not an intersection, what will be done from now on? The driving task identified is changing lanes to the right.

Then, the recognition determining means 17 determines whether or not the driver's situation recognition in the driving task related to changing lanes to the right is appropriate, and if it is not appropriate, the support means 12 provides support.

That is, in this first case, the driver's situational awareness is determined from the line-of-sight data of the time periods before and after the input of the turn signal that started flashing the right turn turn signal. Specifically, the line of sight data used here is such that the steering angle of the steering wheel is set at a predetermined angle ( For example, the period of time until the lane change starts when the lane change occurs (for example, 5 degrees) or more is used.

Here, in a driving task related to changing lanes to the right, the following situational recognition is required. First, in the first phase from the start of the lane change plan to the time when the direction indicator is input, the driver checks the vehicle behind him through the rearview mirror, and if the vehicle behind him is about to overtake, the vehicle M cannot change lanes. Also, in this first phase, it is necessary to confirm the presence of other vehicles to some extent through the right side mirror and right window.

In the second phase, from when the turn signal is input to when the lane change starts, the driver checks other vehicles in the right lane near own vehicle M through the right side mirror and right window, and enters behind any other vehicle. It is necessary to assume that the

In addition, in the first and second phases, it is also necessary to check the front so that the driver does not get distracted by other vehicles in other lanes and collide with another vehicle A ahead in the same lane.

Therefore, the determining unit 42 of the recognition determining means 17 makes a determination regarding the actual visual line data using a visual line model in which the above-mentioned situational recognition necessary for the driving task is set for each visual recognition area. In other words, it is determined whether the driver has properly recognized the situation based on whether the driver has visually recognized the appropriate visual area at the appropriate timing and for the appropriate viewing time. If it is determined that the situation is not properly recognized, the support means 12 alerts the user.

In the first case, in the first phase, other vehicles A existing in the front area A1, other vehicles B existing in the rearview mirror area A2, and other vehicles D and E existing in the right side mirror area A6 are , visual confirmation by the driver is required. Therefore, the determination unit 42 extracts from the line-of-sight data the presence or absence of visual confirmation of each other vehicle, which is a recognition target object, and the visual confirmation time in these visual recognition areas, and determines it using a line-of-sight model generated in advance. As for the visual recognition time, it is determined whether the total time during which other vehicles to be viewed in each visual recognition area in the first phase were visually recognized has reached the reference time of the line-of-sight model. The reference time here is specified from the information acquisition result by the information acquisition means 14 using the above-mentioned relational expression.

In the example of line-of-sight data in FIG. 4, it is considered that the other vehicle A existing in the front area A1 is visually recognized twice in the first phase, but it is determined whether the total time has reached the reference time or not. be done. If the reference time has not yet been reached, immediately after the direction indicator is input, the support means 12 provides support to the effect that the confirmation of the front is insufficient.

In addition, regarding other vehicle B existing in the rearview mirror area A2, since the driver has never seen it in the first phase, the reference time has not been reached, and immediately after the turn signal is input, support is provided. Means 12 provides support for requesting rear confirmation.

Further, in the first phase, it is considered that the other vehicles D and E present in the right side mirror area A6 are visually recognized once, and it is determined whether or not that time has reached a reference time. If the reference time has not yet been reached, immediately after inputting the direction indicator, the support means 12 provides support to the effect that confirmation of the right rear side is insufficient. Furthermore, in the line-of-sight model, for example, if it is necessary to confirm the recognition target in the right window area A5 after visual recognition in the right side mirror area A6, if this is not done, support by the support means 12 will also be required. will be done.

In addition, in the first phase, for example, even if there is a time period in which the driver has visually observed a predetermined visibility area for more than a reference time, there is another vehicle that is a recognition target that should be visible in the same phase. If the timing is different from any time period within the same phase, it is assumed that the appropriate viewing timing has been missed, and the above-mentioned support is provided. In this way, the comparison with the reference time is based on the relationship between the evaluation of appropriate visibility timing, and the actual visibility time within the visibility area in each phase when other vehicles exist in the visibility area in that phase. This is done for the time that overlaps with the time. This also applies to all evaluations based on reference times below.

In the example shown in FIG. 4, in the second phase, immediately before starting to change lanes, the driver checks the other vehicle A existing in the front area A1 and the other vehicle E existing in the right window area A5 and right side mirror area A6. Visual confirmation is required. Therefore, from the line-of-sight data of the driver of the own vehicle M up to a predetermined period of time (for example, 3 seconds) before the start of the lane change, the visual recognition of other vehicles in each of the visual recognition areas A1, A5, and A6 is performed in the same way as in the first phase. It is determined whether the time has reached the reference time. As a result, if there is a visible area that has not been visually observed, the support means 12 will provide support immediately after the start of the lane change to encourage visual confirmation of the visible area, and if the visual area has not reached the reference time even if the lane change is being visually observed, If there is an area, support is provided to the effect that visibility to the visibility area is insufficient. The support here is set so that it can influence the driver more strongly than the support in the first phase.

For the driving task of changing lanes to the left, situation recognition evaluation and driving support are performed using the same processing as in the first case by changing the left and right visibility areas, but the reference time is based on the position described above. Due to the relationship of coefficients depending on the elements, this is different from the case of the driving task of changing lanes to the right.

Next, as a second case, assuming the traffic situation shown in Figure 5, situation recognition evaluation and driving support are performed as follows for a driving task in which the own vehicle M turns right at an intersection under that situation. . The traffic situation in this case is as follows. Another vehicle B, which is a motorcycle, is traveling from the rear right side of own vehicle M toward the right side, and another vehicle C, which is another automobile that is an oncoming vehicle traveling in the opposite lane, is about to pass through an intersection. Furthermore, when the own vehicle M turns right at an intersection, a pedestrian H is walking on the crosswalk.

In this second case as well, the information acquisition means 14 acquires and stores various pieces of information at regular intervals, and based on this information, the driver's line of sight data is sequentially generated and stored in the visibility situation grasping means 15. Ru. As shown in FIG. 6, the line-of-sight data in this case includes, for each visibility area A1 to A6, the presence or absence of the driver's visibility, the visibility time period Tm, and other vehicles B, C and pedestrians around the own vehicle M. The presence of signal H is stored in correspondence with the presence and light color of the signal S that determines whether the own vehicle M is turning right at the intersection. Note that the line of sight data here is also generated within a certain range R in the front, rear, left and right directions around the own vehicle M.

Next, the task specifying means 16 specifies the driving task. In the second case, based on the information acquired by the information acquisition means 14, when an operation is performed to blink the turn signal indicating the right turn direction, and it is detected that the intersection will be passed from now on, the driving task related to the right turn at the intersection is performed. is specified.

Then, the recognition determining means 17 determines whether or not the driver's situational recognition in the driving task regarding a right turn at an intersection is being performed appropriately, and if the driver is not properly recognizing the situation, the support means 12 provides support. It will be done.

That is, in this second case as well, the driver's situation recognition is determined from the line of sight data for each phase specified by the detection result by the information acquisition means 14.

The phases here include the time from when the turn signal is inputted, when the right turn direction signal starts flashing, to when the own vehicle M enters just before the stop line, when it enters within a predetermined distance (for example, 30 m) from the stop line. The first phase, the second phase from when the vehicle M enters just before the stop line until the vehicle M starts or crosses the stop line, and the third phase from when the vehicle M passes the stop line to when the right turn is completed. On the other hand, the driver's situational awareness is determined.

Here, in a driving task related to a right turn at an intersection, the following situational recognition is required. First, in the first phase, before moving the own vehicle M to the right, the driver confirms the presence of a two-wheeled vehicle such as a motorcycle through the right side mirror and right window, and also looks ahead to check for traffic lights, etc. There is a need.

In the second phase, it is necessary to look ahead again and check the signals, etc. in order to understand the change in the signals. Also, in this case, it is necessary to confirm the presence of two-wheeled vehicles such as motorcycles through the right side mirror or right window.

Furthermore, in the third phase, if the vehicle M is temporarily stopped due to a red light, it is necessary to check the signal again when starting, and also to confirm the presence of a two-wheeled vehicle such as a motorcycle on the right side of the vehicle M. be. On the other hand, even if the own vehicle M turns right without stopping at a green light, it is necessary to check the signal again and to confirm the presence of a two-wheeled vehicle such as a motorcycle on the right side of the own vehicle M. Also, while turning right, it is necessary to check for oncoming vehicles and pedestrians from ahead.

Therefore, similarly to the first case, the determination unit 42 of the recognition determination means 17 uses the obtained line-of-sight data to determine whether the driver is driving in an appropriate visual area at an appropriate timing and visual duration based on the line-of-sight model. It is determined whether or not the person has visually recognized the situation, and if it is determined that the situation is not recognized appropriately, the support means 12 alerts the user.

In this second case, in the first phase, the driver needs to visually check the signal S existing in the front area A1 and the other vehicle B existing in the right window area A5 or right side mirror area A6. Become. Therefore, the determination unit 42 extracts the presence or absence of visual confirmation in these visual recognition areas and the visual recognition time from the line-of-sight data, and compares it with the reference time of the line-of-sight model generated in advance. Here, as in the first case, it is determined in the line of sight data whether the total time of the visual recognition time of the other vehicle to be visually recognized has reached the reference time.

In the example of line-of-sight data in FIG. 6, it is considered that the signal S present in the front area A1 is visually recognized twice in the first phase, but it is determined whether the total time reaches the reference time or not. Ru. If the reference time has not yet been reached, the support means 12 provides support to the effect that the front confirmation is insufficient before the own vehicle M reaches the stop line.

Furthermore, in the first phase, it is considered that the other vehicle B present in the right side mirror area A6 is visually recognized once, and it is determined whether or not that time has reached a reference time. If the reference time has not yet been reached, the support means 12 provides support to the effect that confirmation of the right side is insufficient before the own vehicle M reaches the stop line.

In addition, when the signal changes before and after the time of entering just before the stop line, it is determined whether or not the signal existing in the front area A1 is visually recognized when the time of entering just before the stop line. Immediately after the vehicle enters just before the stop line, the support means 12 provides support for reconfirming the signal ahead.

In the example of FIG. 6, in the second phase, immediately before passing the stop line, a signal S existing in the front area A1 and another vehicle C, which is an oncoming vehicle, exist in the right window area A5 or right side mirror area A6. The driver must visually check the other vehicle B. Therefore, from the visual line data of the driver of the own vehicle M up to a predetermined time (for example, 1.5 seconds) before passing the stop line, in the same way as in the first phase, the visibility area A1 and the visibility area A5 or A6 are determined. It is determined whether the visual recognition time for the existing recognition target has reached the reference time. In this second phase, it is considered that the driver visually recognizes the signal S and the other vehicle C, which is an oncoming vehicle, present in the area A1 in front of him once. Therefore, it is determined whether the visual recognition time at that time has reached the reference time or not, and if it has not reached the reference time, the support means 12 determines whether the reference time has been reached immediately after passing the stop line. Support is provided to indicate that visibility is insufficient for areas where there is no visibility. In addition, since the right window area A5 is not visually recognized and the timing of the right side mirror area A6 is shifted, it is thought that other vehicles B existing in these areas A5 and A6 are not visually recognized. It will be done. Therefore, the support means 12 provides support for requesting visual recognition of the visual recognition area immediately after passing the stop line.

Further, in the third phase, the driver needs to visually check the other vehicle C, which is an oncoming vehicle, and the pedestrian H, which are present in the front area A1. However, in the example of FIG. 6, the other vehicle C has already passed and does not exist in the front area A1 in the third phase, and only the pedestrian H exists. Therefore, the object to be recognized in the front area A1 in the third phase is only the pedestrian H, and the driver has visually recognized the front area A1 once at the timing when the pedestrian H is present. It is determined whether the time has reached the reference time. If the reference time has not yet been reached, the support means 12 provides support to the effect that the confirmation ahead is insufficient.

For the driving task of turning left at an intersection, situation recognition evaluation and driving support are performed by changing the left and right visibility areas and performing the same processing as in the second case, but for the reference time, the coefficients based on the position factors described above are used. This is different from the case of the driving task of changing lanes to the right.

Next, as a third case, assuming the traffic situation shown in FIG. 7, driving assistance is performed as follows for a driving task in which the own vehicle M moves straight under that situation. The traffic situation in this case is as follows. The host vehicle M is currently traveling in the left lane of a three-lane road, and in the same lane, another vehicle A is running in front of the host vehicle M, and another vehicle B is running behind the same lane. Further, in the right lane of the own vehicle M, another vehicle C and another vehicle D are sequentially running at a higher speed than the own vehicle M from diagonally behind the own vehicle M.

In this third case as well, the information acquisition means 14 acquires and stores various information at regular intervals, and based on this information, the driver's line of sight data is sequentially generated and stored in the visibility situation grasping means 15. Ru. As shown in FIG. 8, the line-of-sight data in this case is such that, for each visibility area A1 to A6, the presence or absence of the driver's visibility and the visibility time period Tm correspond to the presence of other vehicles A to D. be remembered. Note that the line of sight data here is also generated within a certain range R in the front, rear, left and right directions around the own vehicle M, as described above.

Next, the task specifying means 16 specifies the driving task. In the third case, from the information acquired by the information acquisition means 14, when there is no operation to flash the right turn turn signal and it is detected that the driver will be driving on a single road that is not an intersection, the driving task related to going straight is determined. is specified.

Then, the recognition determining means 17 determines whether or not the driver's situational recognition in the driving task related to driving straight ahead is being performed appropriately. If it is determined that the driver's situation recognition is not being performed appropriately, the support means 12 provides support. It will be done.

In this third case, the driver's situational recognition is determined from the line-of-sight data at each visual recognition interval set when the line-of-sight model was generated.

Here, in the driving task related to straight driving, the following situation recognition is required. It is necessary for the driver to check other vehicles within a certain range by looking at the rearview mirror, side mirror, etc. at certain intervals. In other words, in the third case of driving straight in the leftmost lane, during each visibility interval, the driver checks the vehicle in front through the windshield, the vehicle behind through the rearview mirror, and the driver in the right side mirror. It is necessary to check other vehicles in the right lane of own vehicle M through the vehicle M.

That is, from FIGS. 7 and 8, it is necessary for the driver to visually check other vehicles A to D that are present in the front area A1, the rearview mirror area A2, and the right side mirror area A6. Therefore, the determination unit 42 extracts the presence or absence of visual confirmation in these visual recognition areas and the visual recognition time from the line-of-sight data, and compares it with the reference time of the line-of-sight model generated in advance. Here, as in each case described above, it is determined whether or not the total time of visual recognition time of the other vehicle to be visually recognized has reached the reference time based on the line of sight data. If the reference time has not been reached, it is determined that the driver is not properly aware of the situation, and the support means 12 determines in each case that the driver is not visualizing in the necessary visual area, or A warning is issued to the effect that viewing time is insufficient.

In the above embodiment, the case in which various types of driving support information including alerting information is presented to the driver of a car has been illustrated, but the present invention is not limited to this, and the present invention is not limited to this. The present invention can be used as a situation recognition estimation system 11 that estimates whether a subject who moves in a predetermined environment with a mobile object while operating the object appropriately recognizes the situation of the surrounding environment. In other words, the situation recognition estimation system 11 according to the present invention can be used in various ways as a system for estimating whether or not a target person appropriately recognizes a recognition target object that moves relatively around the target person. Available.

In addition, the configuration of each part of the device according to the present invention is not limited to the illustrated configuration example, and various changes can be made as long as substantially the same effect is achieved.

10 Driving support system 11 Situation recognition estimation system 12 Support means 14 Information acquisition means 15 Visibility situation understanding means 16 Task identification means 17 Recognition determination means 37 Visibility area identification section 38 Gaze data generation section 41 Gaze model generation section 42 Judgment section

Claims (6)

In a situational recognition estimation system that estimates whether or not the target person appropriately recognizes a recognition target that moves relatively around the target person,
information acquisition means for acquiring various information related to the target person's movements and the surrounding environment of the target person; and a visibility situation for understanding the visibility status of the recognition target by the target person based on the information from the information acquisition means. comprising a grasping means, a task specifying means for specifying a task to be performed by the subject in the near future based on the information, and a recognition determining means for determining the situational awareness of the subject in the task based on the visual recognition situation. ,
The visibility situation grasping means includes a visibility area specifying unit that classifies the target person's line of sight direction into a plurality of visibility areas, and a visibility area identifying unit that classifies the target person's line of sight for each of the visibility areas based on the classification results of the visibility area specifying unit. a line-of-sight data generation unit that generates time-series line-of-sight data by associating the presence and the visibility situation at each time;
The recognition determining means stores a line-of-sight model corresponding to reference situation recognition for each task, and determines from the line-of-sight data and the line-of-sight model whether or not the situation recognition for the recognition target is appropriately performed. It is equipped with a determination unit that
The line-of-sight model defines, in units of phases constituting the task, the viewing area in which the subject should visually recognize the recognition target, and a reference time that is a reference value of the viewing time for the recognition target to be visually recognized. set,
In the determination unit, when the task that the subject is about to perform is specified by the task specifying means, the gaze model related to the task is extracted, and compared with the actually obtained gaze data , and the phase Depending on whether the total viewing time of the recognition target object in the visibility area has reached the reference time, the recognition target object is displayed in each of the viewing areas where the target person should visually recognize the recognition target object. A situation recognition estimation system characterized in that the situation recognition is determined by determining whether the timing of visually recognizing the object and the viewing time are appropriate or not. The situation recognition estimation system according to claim 1, wherein the recognition determining means further comprises a gaze model generation unit that generates the gaze model using the past gaze data in which the situation recognition was determined to be appropriate. . The reference time includes an optical element that affects the transmittance of light in the surrounding space of the subject at the time of determining the situational awareness, a positional element that affects the visual accessibility of the subject to each of the visibility areas, 2. The situation recognition estimation system according to claim 1 , wherein the situation recognition estimation system is determined by a predetermined relational expression in which the number of times the target person visually recognizes the recognition target object at the same timing is used as a parameter. 4. The situation recognition estimation system according to claim 3 , wherein the constants constituting the relational expression are obtained by statistically processing the past gaze data in which the situation recognition was deemed appropriate. The task is a driving task in which the subject is a driver of a mobile object,
The information acquisition means obtains various information including current position information of the mobile object, road information regarding whether the road on which the mobile object is traveling is an intersection or a single road, and driver operation information for the mobile object. is obtained,
The task specifying means selects the corresponding driving task from among a plurality of patterns stored in advance for the driving task based on the current position information, the road information, and the operation information. The situation recognition estimation system according to claim 1. The situation recognition estimation system according to any one of claims 1 to 5 is applied, wherein the target person is a driver of a moving object, and the situation recognition estimation system estimates the situation around the moving object by the situation recognition estimation system. In driving support systems that provide driving support to drivers,
A driving support system comprising: support means for providing support to the driver when the recognition determination means determines that the driver is not properly recognizing the driving situation.

Images