JP2012103849A - Information provision device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information provision device with which the control of information provided can be performed according to the situation of a driver.SOLUTION: An attention distribution is created based on an attention model prepared and determined information detected at different times, thereby estimating the position of attention of a driver (gazing point) in a random manner (S110 to S230), the attention amount of the driver with respect to the respective objects extracted from a front image is set according to the attention distribution (S240). Further, by using preset rules, the importance of respective objects is set according to the situation determined from the determined information (S250) and the provision of information to the driver is controlled according to the attention amount and the importance (S260).

Description

  The present invention relates to an information providing apparatus that provides information about an object to be recognized by a driver.

  Conventionally, various devices for recognizing an object around a vehicle based on an image taken in front of the vehicle and providing the driver with information on the object to be recognized by display, sound, vibration, or the like are known. Yes.

In this type of apparatus, it is desirable to provide the minimum necessary information because excessive information that the driver does not need may interfere with driving.
On the other hand, information on objects that are difficult for drivers to see and notice by using the visibility that indicates the ease of recognizing the object in the image and the attractiveness that indicates the ease of recognizing the object as indicators. An apparatus to be provided is known (see, for example, Patent Document 1).

  The visibility is calculated by using the contrast between the object and the background, and the higher the contrast, the higher the visibility is determined. And the contrast with the background, the closer the distance and the higher the contrast, the higher the attractiveness.

  In addition, as a technique for obtaining attractiveness, the use of low-order feature quantities (luminance, color, edge, etc.) of an image has been studied (see Non-Patent Document 1).

JP 2004-30212 A

L. Itti and C. Koch, “Computational Modeling of Visual Attention,” Nature Reviews Neuroscience, Vol. 2 (3), pp. 194-203, (2001)

  By the way, in the calculation of visibility and attractiveness, after all, only the driver's gazing point and information uniquely determined from the image (contrast of the image and the position of the object in the conventional device) are used.

  However, in actual driving, objects that require attention vary in various ways depending on the driver's intention at the time (what kind of vehicle operation is being performed), etc., even in the same situation.

  For example, FIG. 7 is an example of an image obtained by imaging the front of the vehicle from the driver's seat of the vehicle. In the scene shown in this image (before the T-junction), when the driver is going straight ahead, the information on the object in the traveling direction of the vehicle, that is, the information on the preceding vehicle (truck) is high, The importance of information regarding an object at a position deviating from the traveling direction of the vehicle, for example, an oncoming vehicle, is relatively low. In particular, the importance of information regarding an oncoming vehicle that is about to turn left (turned to the right when viewed from the host vehicle) and pedestrians ahead is very low. On the other hand, when the driver is going to turn right, the importance of the information on the oncoming vehicle is relatively higher than that of the preceding vehicle, and it is necessary to pay certain attention to the oncoming vehicle that is about to turn left. Furthermore, if there is a pedestrian at the right turn destination, the importance of the information about the pedestrian will be very high.

  However, in the conventional device, the situation surrounding the driver (including the driver's intention) and the importance of the object that changes according to the situation are not considered, and the information uniquely determined from the driver's gaze point and the image However, there is a problem that important information may not be provided.

  Further, in the conventional apparatus that detects and uses the driver's gazing point in real time, it is necessary to separately prepare an apparatus for measuring the driver's gazing point, which causes a problem that the apparatus becomes expensive.

  In order to solve the above problems, an object of the present invention is to provide an information providing apparatus capable of controlling provided information in accordance with a situation surrounding a driver.

  In the information providing apparatus of the present invention made to achieve the above object, the image acquisition means acquires an image corresponding to the driver's field of view, and the object extraction means extracts an object from the image acquired by the image acquisition means. The information providing means provides information related to the object extracted by the object extracting means. Further, information reflecting the driver's intention for driving behavior or information affecting the driving behavior of the driver is specified information, and the model storage means is directed to the driver's attention on the value indicated by the specific information and the image. An attention model indicating a relationship with a gaze point that is a position is stored.

  Then, when the information acquisition unit acquires the specific information, the attention distribution generation unit calculates the attention distribution in which the gaze point is estimated from the specific information detected by the information acquisition unit based on the attention model stored in the model storage unit. Generate.

  Based on the attention distribution generated by the attention distribution generation means, the attention amount calculation means obtains an attention amount indicating the degree of attention of the driver for the object for each object extracted by the object extraction means, and provides information The means controls information to be provided according to the attention amount calculated by the attention amount calculation means.

  In addition, in the “control of information to be provided” mentioned here, for each object, it is determined whether or not it is necessary to provide information about the object, and a method of providing information (image, sound, vibration, etc.) is selected. Of these, at least one of them is included.

  As described above, according to the information providing apparatus of the present invention, the provided information is controlled using the attention distribution in which the information indicating the intention of the driver or the information affecting the driving behavior of the driver is reflected. It is possible to provide accurate information according to the situation.

  That is, even if the attention distribution is the same image, the attention distribution differs depending on the situation specified from the specific information based on other than the image, and in that situation, where many drivers are paying attention, In other words, in that situation, it is an estimate of where the driver's attention is directed.

  Moreover, according to the information providing apparatus of the present invention, since it is not necessary to measure the driver's point of sight, it is more accurate despite the simple and inexpensive configuration compared to the conventional apparatus that measures this. Control can be realized.

  Note that the attention model stored in the model storage means may be prepared with only one piece of specific information, but for each of a plurality of types of generated information groups composed of one or more pieces of information belonging to the specific information. May be provided.

  In this case, the attention distribution generation unit generates an individual distribution in which the attention point is estimated from the specific information detected by the information acquisition unit for each attention model, and generates the attention distribution by integrating the individual distributions. Good.

  In this case, since the attention distribution can be generated by the individual distribution obtained from the specific information that can be acquired even if the specific information is not all available, regardless of what specific information the vehicle can detect. The information providing apparatus of the present invention can be applied, and a highly versatile apparatus can be provided.

  The specific information includes, for example, vehicle information including at least one of information related to driver operation or information indicating vehicle behavior, driver information that is information related to a driver's biological reaction, and information related to a driving environment of the vehicle. It is conceivable to use environmental information.

  However, since the driver's operation reflects the driver's intention, the vehicle information related to the driver's operation corresponds to the information reflecting the driver's intention. In addition, vehicle information related to the behavior of the vehicle, environmental information, and biological information correspond to information that affects the driving behavior of the driver. Note that part of the vehicle information and part of the biological information related to the behavior of the vehicle may be treated as information reflecting the driver's intention when the correlation with the driver's intention is clear.

  In the information providing apparatus of the present invention, at least part of the environmental information may be acquired by road-to-vehicle communication or vehicle-to-vehicle communication. Specifically, it is conceivable to acquire information that cannot be accurately grasped from various sensors mounted on the vehicle, such as traffic jam information and information on the behavior of preceding vehicles and oncoming vehicles.

  By the way, in the information providing apparatus of the present invention, the importance setting unit further sets the importance of the object extracted by the object extracting unit according to a preset rule, and the information providing unit adds the attention amount to the attention amount. The information to be provided may be controlled in accordance with the importance set by the importance setting means.

  According to the information providing apparatus of the present invention configured as described above, not only the information about the position where the driver's attention is directed but also the information about the importance of the object can be used. It is possible to control more accurately according to.

  It should be noted that it is desirable that the rules used by the importance level setting means for setting the importance level include at least those that are conditional on the situation specified from the specific information acquired by the information acquisition means.

  In other words, since the importance level of an object changes depending on the situation, the provided information can be controlled more accurately by using the importance level set in this way.

  Further, it is desirable that the rules used by the importance setting unit for setting the importance include at least a rule that is based on the position of the object extracted by the object extraction unit and the correlation between the objects.

  That is, the importance of an object may change in the relationship between objects, such as when a certain type of object exists at the same time or when the objects are in a specific positional relationship. Therefore, by using such information, it is possible to more accurately control the provided information.

The block diagram which shows the outline | summary of a driving assistance system, and the whole structure of a vehicle-mounted apparatus. The flowchart which shows the content of the information provision process which the control part of a vehicle-mounted apparatus performs. Explanatory drawing which showed the image of the process at the time of generating attention distribution. The table | surface which illustrated the content of the presentation control table used at the time of judgment whether information is shown, etc. The block diagram which shows the structure of the vehicle-mounted apparatus for learning used when learning an attention model. The graph which shows the effect by the method of this invention. The figure which illustrated the front picture.

Embodiments of the present invention will be described below with reference to the drawings.
[overall structure]
FIG. 1 is a block diagram showing an outline of a driving support system 1 to which the present invention is applied.

  As shown in FIG. 1, a driving support system 1 is mounted on a vehicle and provides information to a driver by providing an on-vehicle device (information providing device) 2 and various information on the traveling environment of the vehicle on the roadside of the VICS. And an information server 4 provided to a vehicle (on-vehicle device 2) through a communication infrastructure 3 for wireless communication including a base station of a machine or a mobile phone.

  The in-vehicle device 2 is configured to perform not only wireless communication with the information server 4 via the communication infrastructure 3, that is, road-to-vehicle communication, but also wireless communication directly performed between the in-vehicle devices 2, that is, vehicle-to-vehicle communication. .

  Information acquired by the in-vehicle device 2 through road-to-vehicle communication includes, for example, traffic jam information, signal color and waiting time, weather (such as rain and wind information at the exit of a tunnel), presence of falling objects, road surface conditions, traffic regulations, etc. The information exchanged between the in-vehicle devices 2 by inter-vehicle communication includes, for example, vehicle speed, acceleration, steering angle, and driving behavior of the vehicle predicted from the information (that is, a vehicle to be described later) Part of the information).

[In-vehicle device]
The in-vehicle device 2 detects a vehicle sensor group 21 including various sensors for detecting vehicle behavior, vehicle operation by a driver, operating states of various in-vehicle devices, and the like, and detecting a physical change (biological reaction) of the driver. A biosensor group 22 composed of various sensors, an environment sensor group 23 composed of various sensors for detecting the traveling environment around the vehicle, and a GPS signal transmitted from a GPS satellite are received, and the position of the vehicle (latitude, Longitude) and a GPS receiver 24 for detecting the traveling direction.

  The in-vehicle device 2 also provides information presentation that uses visual, auditory, and tactile information to the driver and the wireless communication unit 25 that performs road-to-vehicle communication using the communication infrastructure 3 and vehicle-to-vehicle communication between vehicles. Unit 26, command input unit 27 including various input devices for inputting commands to in-vehicle device 2, map database 28 storing map data, CPU, ROM, RAM, I / O, and the like. A control unit 29 that is configured around a microcomputer and executes various processes is provided.

[Sensor group]
The detection target of the vehicle sensor group 21 includes vehicle behavior such as vehicle body speed, acceleration, and yaw rate, as well as operation amounts of the accelerator pedal, brake pedal, steering, etc. (accelerator opening rate, brake M / C pressure, steering angle, etc.) ), Turn signal, wiper, headlight, seat belt operating state, travel mode setting (when there is a sport mode or an auto cruise function), and the like.

  The detection targets of the biometric sensor group 22 are line of sight, face orientation, heart rate, blood pressure, sweating, and the like. The environmental sensor group 23 includes a laser radar, a millimeter wave radar, various cameras for photographing the front and rear of the vehicle, a short-range sensor such as sonar, and the like. The detection target is various objects (vehicle, roadside object, walking). Distance, relative speed, object size, luminance and color for each pixel in the image, edge portion in the image, and the like.

  In addition, each sensor which comprises these sensor groups 21-23 is arrange | positioned at each part of a vehicle, and the control part 29 seems to acquire the detection result in each sensor via vehicle-mounted LAN (CAN and LIN) which is not shown in figure. Has been. Further, the individual sensors constituting the sensor groups 21 to 23 are not necessarily required to have all types of all vehicles, and may be partially different for each vehicle. However, in the environment sensor group 23, a camera that captures the front of the vehicle provided to acquire an image corresponding to the driver's field of view is essential. Hereinafter, an image captured by the camera is referred to as a “front image”. Called.

[Information presentation section]
The information presenting unit 26 displays various images and visually presents information, a speaker for outputting an alarm sound and sound and presents information audibly, and a position in contact with a driver such as a steering wheel or a seat And a vibration generator for tactilely presenting information by generating vibration.

[Control unit]
The control unit 29 uses the map data stored in the map database 28, the position data obtained from the GPS receiver 24, and the like, via the information presentation unit 26, displays a map around the vehicle position, a guidance display regarding the set route, Processing for realizing a so-called navigation function for performing voice guidance or the like is executed. That is, the in-vehicle device 2 is basically configured to function as a known navigation device.

  The control unit 29 acquires various information via the sensor groups 21 to 24 and the wireless communication unit 25, and further processes the acquired information (executes image processing, various calculations, statistical processing, and the like). Various information is generated, and the acquired and generated various information is recorded as specific information for specifying a situation surrounding the driver.

  Here, as the information obtained by processing, specifically, the physiological state of the driver generated from the information from the biosensor group 22 in addition to the differential value, the integral value, and various average values of the information ( For example, the position and type of an object (for example, a vehicle, a pedestrian, a sign, a fallen object, etc.) that is generated from information representing ill-mannered, concentrated, looking aside, drowsiness, fatigue, impatience, etc. (Pedestrian crossings, sidewalks), information specific to the object being a vehicle (presence / absence of preceding or oncoming vehicles, distance between vehicles, relative speed, relative acceleration), road shape (curvature, gradient, intersection), road surface condition, number of lanes When the object is a sign, there is information such as the content (speed limit, entry prohibition, one-way). Information obtained by processing information from the environmental sensor group 23 may partially overlap with information acquired via the wireless communication unit 25. The environment information may include information obtained from the GPS receiver 24 and the map database 28.

  Hereinafter, various information acquired from the vehicle sensor group 21 and information obtained by processing the information are referred to as “vehicle information”, and various information acquired from the biological sensor group 22 and the information are processed. The information obtained by doing this is referred to as “driver information”, and various information obtained from the environmental sensor group 23 (including the GPS receiver 24 and the map database 28) and information obtained by processing the information. Is referred to as “environment information”, various information acquired through the wireless communication unit 25, and information obtained by processing the information is referred to as “communication information”.

  Further, among the specific information (vehicle information, driver information, environmental information, communication information), the vehicle information related to the driver's operation is also referred to as “intention reflecting information” as information reflecting the driver's intention for driving behavior. The other information is also referred to as “behavior influence information” as information that affects the driving behavior of the driver.

  Further, the control unit 29 estimates a position where the driver is paying attention based on the specific information recorded by the specific information recording process, and executes an information providing process for providing information to the driver based on the estimation result. To do.

  In addition, a caution model used in the information providing process is stored in a non-volatile memory (for example, ROM) constituting the control unit 29. This attention model divides specific information into a plurality of types of generation information groups composed of one or a plurality of information belonging to the specific information, and for each generation information group, the value of the generation information group and the driver information It shows the relationship with the position where attention is directed (hereinafter referred to as “gaze point”), and is generated by prior learning.

  The generated information group may be based on classification of specific information (vehicle information, environmental information, driver information, communication information), or may be a unit obtained by grouping specific information for each highly relevant information. However, individual specific information may be used as a unit. However, in this embodiment, a caution model is prepared for each classification of specific information.

  Specifically, the attention model is input with information belonging to the generation information group, and whether the forward image is a gazing point for each predetermined area (for example, pixel) is binary of 0 and 1. Is realized by a classifier that classifies according to the above, an estimator that probabilistically estimates whether or not it is a gazing point by a continuous value between 0 and 1.

Note that the format of the attention model (the above-described discriminator and estimator) may be the same for all the generation information groups, but may be different for each generation information group.
[Information provision processing]
Here, the information provision process which the control part 29 performs is demonstrated along the flowchart shown in FIG.

This process is started every time the above-described specific information acquisition process is executed, that is, every time the specific information is recorded in the RAM or the like.
When this process is started, in S110, an object existing in the front image is extracted based on the environment information from the specific information recorded by the specific information acquisition process. The objects to be extracted are predetermined such as various vehicles, pedestrians, signs (including road signs) traffic lights, obstacles on the road, and the like.

  In S120, it is determined whether or not an object has been extracted in S110. If no object has been extracted, this process ends. If any object has been extracted, the process proceeds to S130.

  In S130, an attention distribution based on the extracted object is created. The attention distribution created here indicates the position where the object exists, and the distribution value of the area where the object is detected may be set uniformly high, or depending on the position, size, type, etc. of the object The distribution value may be changed (for example, higher as the vehicle is closer).

  In subsequent S140, an attention distribution based on image information (luminance, color, edge) of the front image is created. The attention distribution created here indicates a position where the driver can easily pay attention, and is generated by setting a distribution value corresponding to the image information for each pixel according to a preset rule. That is, the attention distribution created in S130 and S140 is created using only the information obtained from the image without using the attention model. However, the attention distribution based on the object may be generated using a attention model that receives the position, size, color, and the like of the object.

  In subsequent S150, it is determined whether or not the vehicle information exists in the specific information recorded by the specific information acquisition process. If not, the process proceeds to S170 as it is, and if it exists, based on the vehicle information in S160. The attention distribution is calculated using the learning model, and the process proceeds to S170.

  In S170, it is determined whether or not environmental information exists in the specific information recorded by the specific information acquisition process. If there is no environmental information, the process proceeds directly to S190. The distribution is calculated using the learning model, and the process proceeds to S190.

  In S190, it is determined whether or not the driver information exists in the specific information recorded by the specific information acquisition process. If it does not exist, the process proceeds to S210 as it is. The distribution is calculated using the learning model, and the process proceeds to S210.

  In S210, it is determined whether or not the communication information exists in the specific information recorded by the specific information acquisition process. If there is no communication information, the process proceeds to S230 as it is. The distribution is calculated using the learning model, and the process proceeds to S230.

  Note that the generation of the attention distribution based on the vehicle information (S160) uses a learning model (identifier or estimator) that receives the vehicle information as an input to determine whether or not it is a gaze point, or the probability that it is a gaze point. Is estimated by calculating for each area to be determined or estimated. Here, a caution distribution is generated for each vehicle information, and the generated caution distribution is integrated / added / averaged. For this reason, the vehicle information used when generating the attention distribution does not necessarily have all the vehicle information used when generating the attention model. It is only necessary to calculate the attention distribution based on the vehicle information using only the vehicle information that can be used. The same applies to generation of an attention distribution based on environment information (S180), generation of an attention distribution based on driver information (S200), and generation of an attention distribution based on communication information (S220).

  In S230, the attention distribution (hereinafter also referred to as “individual distribution”) generated in S130, S140, S160, S180, S200, and S220 is integrated. In this case, all may be integrated (simple addition or simple average) with the same weight, or may be individually weighted and integrated (weighted addition or weighted average). Hereinafter, the attention distribution generated in this step is referred to as “integrated distribution”.

  FIG. 3 is an explanatory diagram showing an image of the processes of S130 to S230, that is, a process of generating a plurality of individual distributions based on the specific information and integrating these to generate a final integrated distribution. is there.

  In subsequent 240, for each of the objects extracted in S110, an attention amount indicating an estimation result of how much the driver's attention is directed to the object is obtained using the integrated distribution obtained in S230. Specifically, the attention amount is set to either large, medium, or small according to the distribution value of the area where the object exists on the integrated distribution. That is, if the distribution value is equal to or greater than the first threshold, the attention amount is large. If the distribution value is equal to or smaller than the second threshold set to a value smaller than the first threshold, the attention amount is small. If the distribution amount is smaller than the first threshold and greater than the second threshold. The amount of attention is medium.

  As the distribution value of the area where the object exists, the sum of all distribution values belonging to the area may be used, or the average value of the distribution values belonging to the area may be used. Then, the attention value is set by comparing the distribution value of the area calculated in this way with a preset threshold value.

In subsequent S250, the importance is set to one of large, medium, and small according to a predetermined rule for each of the objects extracted in S110.
As the rule here, an IF-THEN rule that uses a situation specified from specific information, a position of an object, a correlation between objects, or a combination thereof as a condition part is used.

  Specifically, for example, if the “turner state” that is one of the vehicle information is set to “turn right” (condition part), the importance level of the object located on the lower right side of the screen is set to be large. If the rule (consequence part), “distance to object (presumed to be a traffic signal or road sign)” is less than or equal to “predetermined value” (condition part), the importance of the object is set to a large value (consequence part) ), “Object presumed to be a pedestrian crossing” (condition part), the importance of an object presumed to be a pedestrian located on that object and near both ends is greatly increased. A rule of setting (consecutive part) can be considered. Regarding rule setting, it is conceivable to use correlation calculation, Apriori algorithm, or the like.

  In subsequent S260, based on the attention amount of the object set in S240 and the importance level of the object set in S250, whether or not to provide the driver with information on the object, and any format to be presented The information is provided according to the determination result, and this processing is terminated.

  Here, FIG. 4 is an explanatory diagram showing the contents of the presentation control table used when the determination in S260 is performed. In FIG. 4, ◯ indicates that information is presented, Δ indicates that presentation / non-presentation is determined according to the driver state, and x indicates that information is not presented. In other words, if the importance of an object is large, information is presented regardless of the amount of attention. If the importance is medium, information is presented if the attention amount is medium or small, and if the attention amount is large. The presentation / non-presentation is determined according to the driver status. Furthermore, when the degree of importance is small, information is presented if the attention amount is small, presentation / non-presentation is determined according to the driver state if the attention amount is medium, and not presented if the attention amount is large. .

  Note that the driver's state refers to an abrupt driving degree, an arousal degree, a driving skill level, etc. required from biological information, vehicle information, etc., for example, whether the driving degree is absurd, the arousal level is low, or the driving skill level is When it is low, control is performed to present information.

  In addition, as a method of presenting information, if the importance level is large, presentation using all of tactile sensation (vibration), hearing (sound), and vision (display) is performed. , Presentation using visual (display) is performed, and if importance is small, presentation using only visual (display) is performed.

  In the above-mentioned control, the method of presenting information is controlled by importance. However, if the amount of attention is large, presentation is performed using only visual information. If the amount of attention is medium, auditory and visual are used. If the attention amount is small, the presentation may be performed using all of tactile sensation, hearing, and vision.

Furthermore, the method of presenting information may be controlled in association with the marks “◯”, “Δ”, and “X” in FIG. 4, that is, based on both the importance level and the attention amount.
[Caution model]
Next, attention model learning performed in advance will be described.

  Note that the caution model may be learned in the vehicle, but in the present embodiment, a caution model is used. When learning the attention model, data collected for many drivers is used.

  For data collection, as shown in FIG. 5, a vehicle equipped with the in-vehicle device for learning 5 in which the gaze detection device 30 for detecting the driver's gaze is added to the in-vehicle device 2 described above, or information necessary for learning is collected. Use a collectable drive simulator. However, in the in-vehicle device 5 for learning, unlike the in-vehicle device 2, the configuration for providing information (information presentation unit 26) may be omitted.

[Learning process]
In the learning process, first, by causing a plurality of subjects to drive a vehicle equipped with the on-vehicle device for learning, or by causing a plurality of subjects to use a drive simulator, the gaze point of the subject (detection result by the line-of-sight detection device 30) and Specific information (vehicle information, environmental information, driver information, communication information) is acquired.

  Then, the specific information is divided into a plurality of types of generated information groups composed of one or a plurality of information belonging to the specific information, and the relationship between the value of the generated information group and the gaze point is determined for each generated information group. learn. This learning uses a statistical method such as a support vector machine (SVM) or a mixed Gaussian distribution (GMM).

  The generated information group may be based on classification of specific information (vehicle information, environmental information, driver information, communication information), or may be a unit obtained by grouping specific information for each highly relevant information. However, individual specific information may be used as a unit.

  In learning using the SVM, the subject's gazing point is used as teacher data, for example, a positive flag is set around the gazing point position of the photographed forward image, and a negative flag is set at a position other than the gazing point to generate information. With the information belonging to the group as an input, learning is performed so that a discriminator that classifies whether or not the forward image is a gazing point for each predetermined region (for example, pixel) based on binary values of 0 and 1 is obtained.

  Also, in learning using GMM, for some of the information contents belonging to the generated information group (for example, steering angle, own vehicle speed, etc.), a GMM that expresses the relationship between each piece of information and the point of interest position in the front image is used. learn. By further combining these GMMs by addition or multiplication, it is probabilistically estimated whether or not the forward image is a gazing point for each predetermined region (for example, pixel) by a continuous value between 0 and 1. An estimator is configured.

[effect]
As described above, the in-vehicle device 2 generates the attention (gaze point) where the driver's attention is directed based on the prepared attention model and the specific information detected at that time. Estimate from the distribution, calculate the driver's attention amount for each object in the forward image from the attention distribution, and use the preset rules to determine the importance of each object according to the situation specified from the specific information Setting and controlling the provision of information to the driver based on the amount of attention and importance.

  Note that the attention model is created so as to reflect information indicating the driver's intention and information affecting the driving behavior of the driver. In other words, by using a caution model, depending on the type of steering operation such as turning left or right or going straight (information that reflects the driver's intention), the level of the vehicle speed or the color of the traffic light (information that affects the driver's driving behavior) Different attention distributions will be obtained.

  Therefore, according to the vehicle-mounted device 2, according to the occasional situation (including the presence of an object that affects the driver's intention and the driving behavior of the driver, etc.), accurate information provision considering the situation is realized. Can do.

  Moreover, according to the in-vehicle device 2, since the driver's gaze point is estimated from the attention model and the specific information, it is simpler and less expensive than a conventional device that requires a measuring instrument that measures the gaze point. Despite the simple configuration, more accurate control can be realized.

  Here, FIG. 6A is a caution distribution percent area that indicates what percentage of the distribution value (attention level) is included in the attention distribution generated from the learning model described above. Is a graph showing the result of measuring the coincidence ratio of (2), (b) shows the attention distribution (integrated distribution) generated based on the learning model in comparison with the original forward image, and the distribution value is high The top three objects are also indicated by circle numbers.

  As the front image, actual vehicle data acquired in a general city road including various situations such as turning right and left, going straight, following traveling, and passing in the actual environment as shown in FIG. 7 was used. The estimator using the GMM described above is used as the caution model, and the own vehicle speed, the steering angle, and the vehicle detection result are used as the specific information input to the caution model. As a result, a gaze model that can cope with the various situations described above was constructed.

  In FIG. 6A, as a comparative example, the result of estimating the gazing point from the low-order feature amount (luminance, color, edge, etc.) of the image using the method of Non-Patent Document 1 (conventional method), In addition, the coincidence rate (chance level) when gaze points are set at random is also shown.

  As can be seen from the figure, the conventional method is less than or equal to the chance level, whereas the method applied to the in-vehicle device 2 has a high coincidence rate with the actual line of sight by 30% or more, and the driver's intention and driver It can be understood that the accuracy of gaze point estimation can be remarkably improved by reflecting information such as objects that affect the driving behavior of the user in the attention model.

  Further, according to the in-vehicle device 2, various types of specific information are used when generating the attention distribution (individual distribution) from a plurality of attention models, but the individual distribution, As a result, since an integrated distribution can be generated, the present invention can be applied to various vehicles having different specific information that can be acquired.

  In the above embodiment, the camera that acquires the front image in the environmental sensor group 23 is the image acquisition unit, S110 is the object extraction unit, the information presentation unit 26 and S260 are the information provision unit, and the specific information recording process that the control unit 29 executes Is an information acquisition means, a non-volatile memory constituting the control unit 29 in which an attention model is stored, a model storage means, S130 to S230 are attention distribution generation means, S240 is an attention amount calculation means, and S250 is an importance setting. Corresponds to means.

  Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and can be implemented in various modes without departing from the gist of the present invention.

  For example, in the above embodiment, the information to be presented is controlled using both the attention amount of the object and the importance level of the object, but the control is performed using only one of them. Also good.

  DESCRIPTION OF SYMBOLS 1 ... Driving assistance system 2 ... In-vehicle apparatus 3 ... Communication infrastructure 4 ... Information server 5 ... In-vehicle apparatus 21 for learning 21 ... Vehicle sensor group 22 ... Biosensor group 23 ... Environmental sensor group 24 ... GPS receiver 25 ... Wireless communication part 26 ... Information presentation unit 27 ... command input unit 28 ... map database 29 ... control unit 30 ... gaze detection device

Claims (9)

Image acquisition means for acquiring an image corresponding to the driver's field of view;
Object extraction means for extracting an object from the image acquired by the image acquisition means;
Information providing means for providing information about the object extracted by the object extracting means;
In an information providing device comprising:
Information acquisition means for acquiring specific information that is information that reflects the driver's intention for driving behavior or information that affects the driving behavior of the driver;
Model storage means for storing a caution model indicating a relationship between a value indicated by the specific information and a gaze point that is a position on the image where the driver's attention is directed;
Based on the attention model stored in the model storage means, attention distribution generation means for generating an attention distribution in which the gaze point is estimated from specific information detected by the information acquisition means;
Based on the attention distribution generated by the attention distribution generation means, for each object extracted by the object extraction means, attention amount calculation means for obtaining an attention amount representing the degree of attention of the driver with respect to the object;
With
The information providing apparatus controls the information to be provided according to the attention amount calculated by the attention amount calculation means. The attention model stored in the model storage means is prepared for each of a plurality of types of generated information groups composed of one or a plurality of information belonging to the specific information,
The attention distribution generation unit generates, for each attention model, an individual distribution in which the gaze point is estimated from the specific information detected by the information acquisition unit, and generates the attention distribution by integrating the individual distributions. The information providing apparatus according to claim 1.   3. The information providing apparatus according to claim 1, wherein the specific information includes vehicle information including at least one of information related to a driver's operation or information indicating a behavior of the vehicle.   4. The information providing apparatus according to claim 1, wherein the specific information includes driver information representing information related to a biological reaction of the driver. 5.   The information providing apparatus according to any one of claims 1 to 4, wherein the specific information includes environment information that is information related to a driving environment of the vehicle.   The information providing apparatus according to claim 5, wherein at least a part of the environmental information is acquired by road-to-vehicle communication or vehicle-to-vehicle communication. Importance level setting means for setting the importance level of the object extracted by the object extraction means according to a preset rule,
The information providing means controls information to be provided according to the importance set by the importance setting means in addition to the attention amount. The information providing apparatus according to the description.   The rule used by the importance level setting unit for setting the importance level includes at least a rule specified by a situation specified from the specific information acquired by the information acquisition unit. Item 8. The information providing device according to Item 7.   The rule used by the importance level setting means for setting the importance level includes at least a rule based on the position of the object extracted by the object extraction means and the correlation between objects. The information providing apparatus according to any one of claims 1 to 8.