JP5742180B2 - Gaze point estimation device - Google Patents

Description

  The present invention relates to a gaze point estimation apparatus that estimates a driver's gaze point.

  Conventionally, various devices for recognizing an object around a vehicle based on an image taken in front of the vehicle and providing the driver with information on the object to be recognized by display, sound, vibration, or the like are known. Yes.

  In this type of device, excessive information that the driver does not need may interfere with driving, and therefore the information to be provided is selected based on information obtained from the driver status and images. ing.

  Note that it is very important to know the driver's point of sight to know the driver's condition. Generally, a camera or other sensor is installed in the passenger compartment to obtain the driver's face image, and the driver's line-of-sight direction. Is detected.

With this method, accurate gazing point information can be obtained, but there is a problem that the apparatus becomes expensive because a dedicated sensor is required to detect the direction of the line of sight.
On the other hand, in the field of cognitive science, research on human attention models has been carried out, and when an image is input, the low-order feature quantity (brightness / It has been proposed to obtain it from color, edge, etc. (see Non-Patent Document 1).

  In other words, it is known that there are neurons in the brain (low-order visual cortex) that respond to luminance, color, and edge gradient, and use the models of these neurons' activities and information transmission pathways. In this way, it is possible to know points on the image where people's attention is easily directed, in other words, an index (attractiveness) representing the ease of orientation of people's attention at each point on the image.

L. Itti and C. Koch, “Computational Modeling of Visual Attention,” Nature Reviews Neuroscience, Vol. 2 (3), pp. 194-203, (2001)

  By the way, in actual driving, the position where the driver's attention is directed (gaze point) varies depending on the driver's intention (what kind of vehicle operation he is trying to perform), etc., even if the surrounding situation is the same To change.

  For example, FIG. 7 is an example of an image obtained by imaging the front of the vehicle from the driver's seat of the vehicle. In the scene shown in this image (before the T-junction), when the driver is going straight ahead, the information on the object in the traveling direction of the vehicle, that is, the information on the preceding vehicle (truck) is high, The importance of information regarding an object at a position deviating from the traveling direction of the vehicle, for example, an oncoming vehicle, is relatively low. In particular, the importance of information regarding an oncoming vehicle that is about to turn left (turned to the right when viewed from the host vehicle) and pedestrians ahead is very low. On the other hand, when the driver is going to turn right, the importance of the information on the oncoming vehicle is relatively higher than that of the preceding vehicle, and it is necessary to pay certain attention to the oncoming vehicle that is about to turn left. Furthermore, if there is a pedestrian at the right turn destination, the importance of the information about the pedestrian will be very high.

  However, in the conventional method, the situation surrounding the driver (including the driver's intention) is not considered, and the driver's gaze point is estimated only by information uniquely determined from the image characteristics. As a result, there has been a problem in that it may not function sufficiently in applications such as selecting information to be provided to the driver.

  In order to solve the above-described problems, an object of the present invention is to provide a gaze point estimation device that estimates a gaze point according to a situation surrounding a driver.

  In the gaze point estimation device of the present invention made to achieve the above object, information reflecting the driver's intention for driving behavior or information affecting the driving behavior of the driver as specific information, the model storage means, An attention model indicating a relationship between a value indicated by the specific information and a gaze point that is a position where the driver's attention is directed on the image is stored.

  Then, when the information acquisition unit acquires the specific information, the attention distribution generation unit calculates the attention distribution in which the gaze point is estimated from the specific information detected by the information acquisition unit based on the attention model stored in the model storage unit. Generate.

  As described above, according to the gaze point estimation device of the present invention, it is possible to generate a caution distribution that reflects information indicating the driver's intention and information that affects the driving behavior of the driver. It is possible to know where the attention is directed without measuring the driver's gaze direction.

Incidentally, the model attention model stored in the storage means, that are provided one or a plurality of a plurality of types that have been configured with the information for each product information group belonging to a specific information.

Then, careful distribution generating means, for each attention model to generate individual distribution estimating the gazing point from the specific information detected by the information acquisition means, that generates the attention distribution by integrating the individual distribution.

As a result, the attention distribution can be generated by the individual distribution obtained from the specific information that can be acquired even if the specific information is not all available, so regardless of what specific information the vehicle can detect. The gaze point estimation apparatus of the present invention can be applied, and a highly versatile apparatus can be provided.

  The specific information includes, for example, vehicle information including at least one of information related to driver operation or information indicating vehicle behavior, driver information that is information related to a driver's biological reaction, and information related to a driving environment of the vehicle. It is conceivable to use environmental information.

  However, since the driver's operation reflects the driver's intention, the vehicle information related to the driver's operation corresponds to the information reflecting the driver's intention. In addition, vehicle information related to the behavior of the vehicle, environmental information, and biological information correspond to information that affects the driving behavior of the driver. Note that part of the vehicle information and part of the biological information related to the behavior of the vehicle may be treated as information reflecting the driver's intention when the correlation with the driver's intention is clear.

  In the gaze point estimation device of the present invention, at least a part of the environmental information may be acquired by road-to-vehicle communication or vehicle-to-vehicle communication. Specifically, it is conceivable to acquire information that cannot be accurately grasped from various sensors mounted on the vehicle, such as traffic jam information and information on the behavior of preceding vehicles and oncoming vehicles.

  By the way, in the gaze point estimation calculation device of the present invention, the image acquisition unit further acquires an image corresponding to the driver's field of view, the object extraction unit extracts an object from the image acquired by the image acquisition unit, The setting means sets the importance of the object extracted by the object extracting means according to a preset rule, the attention distribution generating means sets the information providing means by the importance setting means in addition to the attention amount It may be configured to control information to be provided according to the degree of importance.

  According to the gaze point estimation device of the present invention configured as described above, not only information on the position where the driver's attention is directed but also information on the importance of the object can be used. More accurate control can be performed according to the situation.

Note that the rules used by the importance level setting means for setting the importance level preferably include an IF-THEN rule whose condition is a condition specified from the specific information acquired by the information acquisition means. .

  In other words, since the importance level of an object changes depending on the situation, the provided information can be controlled more accurately by using the importance level set in this way.

Further, the rules used by the importance setting means for setting the importance include IF-THEN rules that use the position of the object extracted by the object extraction means or the correlation between objects as a condition part . Is desirable.

  That is, the importance of an object may change in the relationship between objects, such as when a certain type of object exists at the same time or when the objects are in a specific positional relationship. Therefore, by using such information, it is possible to more accurately control the provided information.

The block diagram which shows the outline | summary of a driving assistance system, and the whole structure of a vehicle-mounted apparatus. The flowchart which shows the content of the information provision process which the control part of a vehicle-mounted apparatus performs. Explanatory drawing which showed the image of the process at the time of generating attention distribution. The table | surface which illustrated the content of the presentation control table used at the time of judgment whether information is shown, etc. The block diagram which shows the structure of the vehicle-mounted apparatus for learning used when learning an attention model. The graph which shows the effect by the method of this invention. The figure which illustrated the front picture. The graph which shows the effect by the method of this invention.

Embodiments of the present invention will be described below with reference to the drawings.
[overall structure]
FIG. 1 is a block diagram showing an outline of a driving support system 1 to which the present invention is applied.

As shown in FIG. 1, a driving support system 1 is mounted on a vehicle and provides on-board device 2 that provides driving assistance by providing information to a driver, and various information relating to the traveling environment of the vehicle is transmitted to a VICS (registered trademark) roadside machine. And an information server 4 provided to a vehicle (vehicle-mounted device 2) via a communication infrastructure 3 for wireless communication including a base station of a mobile phone and the like.

  The in-vehicle device 2 is configured to perform not only wireless communication with the information server 4 via the communication infrastructure 3, that is, road-to-vehicle communication, but also wireless communication directly performed between the in-vehicle devices 2, that is, vehicle-to-vehicle communication. .

  Information acquired by the in-vehicle device 2 through road-to-vehicle communication includes, for example, traffic jam information, signal color and waiting time, weather (such as rain and wind information at the exit of a tunnel), presence of falling objects, road surface conditions, traffic regulations, etc. The information exchanged between the in-vehicle devices 2 by inter-vehicle communication includes, for example, vehicle speed, acceleration, steering angle, and driving behavior of the vehicle predicted from the information (that is, a vehicle to be described later) Part of the information).

[In-vehicle device]
The in-vehicle device 2 detects a vehicle sensor group 21 including various sensors for detecting vehicle behavior, vehicle operation by a driver, operating states of various in-vehicle devices, and the like, and detecting a physical change (biological reaction) of the driver. A biosensor group 22 composed of various sensors, an environment sensor group 23 composed of various sensors for detecting the traveling environment around the vehicle, and a GPS signal transmitted from a GPS satellite are received, and the position of the vehicle (latitude, Longitude) and a GPS receiver 24 for detecting the traveling direction.

  The in-vehicle device 2 also provides information presentation that uses visual, auditory, and tactile information to the driver and the wireless communication unit 25 that performs road-to-vehicle communication using the communication infrastructure 3 and vehicle-to-vehicle communication between vehicles. Unit 26, command input unit 27 including various input devices for inputting commands to in-vehicle device 2, map database 28 storing map data, CPU, ROM, RAM, I / O, and the like. A control unit 29 that is configured around a microcomputer and executes various processes is provided.

[Sensor group]
The detection target of the vehicle sensor group 21 includes vehicle behavior such as vehicle body speed, acceleration, and yaw rate, as well as operation amounts of the accelerator pedal, brake pedal, steering, etc. (accelerator opening rate, brake M / C pressure, steering angle, etc.) ), Turn signal, wiper, headlight, seat belt operating state, travel mode setting (when there is a sport mode or an auto cruise function), and the like.

  The detection targets of the biometric sensor group 22 are line of sight, face orientation, heart rate, blood pressure, sweating, and the like. The environmental sensor group 23 includes a laser radar, a millimeter wave radar, various cameras for photographing the front and rear of the vehicle, a short-range sensor such as sonar, and the like. The detection target is various objects (vehicle, roadside object, walking). Distance, relative speed, object size, luminance and color for each pixel in the image, edge portion in the image, and the like.

  In addition, each sensor which comprises these sensor groups 21-23 is arrange | positioned at each part of a vehicle, and the control part 29 seems to acquire the detection result in each sensor via vehicle-mounted LAN (CAN and LIN) which is not shown in figure. Has been. Further, the individual sensors constituting the sensor groups 21 to 23 are not necessarily required to have all types of all vehicles, and may be partially different for each vehicle. However, in the environment sensor group 23, a camera that captures the front of the vehicle provided to acquire an image corresponding to the driver's field of view is essential. Hereinafter, an image captured by the camera is referred to as a “front image”. Called.

[Information presentation section]
The information presenting unit 26 displays various images and visually presents information, a speaker for outputting an alarm sound and sound and presents information audibly, and a position in contact with a driver such as a steering wheel or a seat And a vibration generator for tactilely presenting information by generating vibration.

[Control unit]
The control unit 29 uses the map data stored in the map database 28, the position data obtained from the GPS receiver 24, and the like, via the information presentation unit 26, displays a map around the vehicle position, a guidance display regarding the set route, Processing for realizing a so-called navigation function for performing voice guidance or the like is executed. That is, the in-vehicle device 2 is basically configured to function as a known navigation device.

  The control unit 29 acquires various information via the sensor groups 21 to 24 and the wireless communication unit 25, and further processes the acquired information (executes image processing, various calculations, statistical processing, and the like). Various information is generated, and specific information recording processing is performed for recording the acquired and generated various information as specific information for specifying a situation surrounding the driver.

  Here, as the information obtained by processing, specifically, the physiological state of the driver generated from the information from the biosensor group 22 in addition to the differential value, the integral value, and various average values of the information ( For example, the position and type of an object (for example, a vehicle, a pedestrian, a sign, a fallen object, etc.) that is generated from information representing ill-mannered, concentrated, looking aside, drowsiness, fatigue, impatience, etc. (Pedestrian crossings, sidewalks), information specific to the object being a vehicle (presence / absence of preceding or oncoming vehicles, distance between vehicles, relative speed, relative acceleration), road shape (curvature, gradient, intersection), road surface condition, number of lanes When the object is a sign, there is information such as the content (speed limit, entry prohibition, one-way). Information obtained by processing information from the environmental sensor group 23 may partially overlap with information acquired via the wireless communication unit 25. The environment information may include information obtained from the GPS receiver 24 and the map database 28.

  Hereinafter, various information acquired from the vehicle sensor group 21 and information obtained by processing the information are referred to as “vehicle information”, and various information acquired from the biological sensor group 22 and the information are processed. The information obtained by doing this is referred to as “driver information”, and various information obtained from the environmental sensor group 23 (including the GPS receiver 24 and the map database 28) and information obtained by processing the information. Is referred to as “environment information”, various information acquired through the wireless communication unit 25, and information obtained by processing the information is referred to as “communication information”.

  Further, among the specific information (vehicle information, driver information, environmental information, communication information), the vehicle information related to the driver's operation is also referred to as “intention reflecting information” as information reflecting the driver's intention for driving behavior. The other information is also referred to as “behavior influence information” as information that affects the driving behavior of the driver.

  Further, the control unit 29 estimates a position where the driver is paying attention based on the specific information recorded by the specific information recording process, and executes an information providing process for providing information to the driver based on the estimation result. To do.

  In addition, a caution model used in the information providing process is stored in a non-volatile memory (for example, ROM) constituting the control unit 29. This attention model divides specific information into a plurality of types of generation information groups composed of one or a plurality of information belonging to the specific information, and for each generation information group, the value of the generation information group and the driver information It shows the relationship with the position where attention is directed (hereinafter referred to as “gaze point”), and is generated by prior learning.

  The generated information group may be based on classification of specific information (vehicle information, environmental information, driver information, communication information), or may be a unit obtained by grouping specific information for each highly relevant information. However, individual specific information may be used as a unit. However, in this embodiment, a caution model is prepared for each classification of specific information.

  Specifically, the attention model is input with information belonging to the generation information group, and whether the forward image is a gazing point for each predetermined area (for example, pixel) is binary of 0 and 1. Is realized by a classifier that classifies according to the above, an estimator that probabilistically estimates whether or not it is a gazing point by a continuous value between 0 and 1.

Note that the format of the attention model (the above-described discriminator and estimator) may be the same for all the generation information groups, but may be different for each generation information group.
[Information provision processing]
Here, the information provision process which the control part 29 performs is demonstrated along the flowchart shown in FIG.

This process is started every time the above-described specific information acquisition process is executed, that is, every time the specific information is recorded in the RAM or the like.
When this process is activated, an attention distribution based on image information (luminance, color, edge) of the front image is created in S110. The attention distribution created here indicates a position where the driver can easily pay attention, and is generated by setting a distribution value corresponding to the image information for each pixel according to a preset rule. That is, the attention distribution created in S110 is created using only the information obtained from the image without using the attention model.
In S120, an object existing in the front image is extracted based on the environment information from the specific information recorded by the specific information acquisition process. The objects to be extracted are predetermined such as various vehicles, pedestrians, signs (including road signs) traffic lights, obstacles on the road, and the like.

  In S130, it is determined whether or not an object is extracted in S120. If no object is extracted, the process proceeds to S150, and if any object is extracted, the process proceeds to S140.

  In S140, an attention distribution based on the extracted object is created. The attention distribution created here indicates the position where the object exists, and the distribution value of the area where the object is detected may be set uniformly high, or depending on the position, size, type, etc. of the object The distribution value may be changed (for example, higher as the vehicle is closer).

  The attention distribution created in S140 indicates a position where the driver can easily pay attention, similarly to the attention distribution created in S110, and a distribution value corresponding to image information for each pixel according to a preset rule. Generate by setting. That is, the attention distribution created in S140 is also created using only the information obtained from the image without using the attention model. However, the attention distribution based on the object may be generated using a attention model that receives the position, size, color, and the like of the object.

In subsequent S150, it is determined whether or not the vehicle information exists in the specific information recorded by the specific information acquisition process. If not, the process proceeds to S170 as it is, and if it exists, based on the vehicle information in S160. The attention distribution is calculated using the attention model , and the process proceeds to S170.

In S170, it is determined whether or not environmental information exists in the specific information recorded by the specific information acquisition process. If there is no environmental information, the process proceeds directly to S190. The distribution is calculated using the attention model , and the process proceeds to S190.

In S190, it is determined whether or not the driver information exists in the specific information recorded by the specific information acquisition process. If it does not exist, the process proceeds to S210 as it is, and if it exists, the notice based on the driver information is determined in S200. The distribution is calculated using the attention model , and the process proceeds to S210.

In S210, it is determined whether or not the communication information exists in the specific information recorded by the specific information acquisition process. If there is no communication information, the process proceeds to S230 as it is. The distribution is calculated using the attention model , and the process proceeds to S230.

Note that the generation of the attention distribution based on the vehicle information (S160) uses a attention model (identifier or estimator) with the vehicle information as an input to determine whether or not it is a gazing point, or the probability of being a gazing point. Is estimated by calculating for each area to be determined or estimated. Here, a caution distribution is generated for each vehicle information, and the generated caution distribution is integrated / added / averaged. For this reason, the vehicle information used when generating the attention distribution does not necessarily have all the vehicle information used when generating the attention model. It is only necessary to calculate the attention distribution based on the vehicle information using only the vehicle information that can be used. The same applies to generation of an attention distribution based on environment information (S180), generation of an attention distribution based on driver information (S200), and generation of an attention distribution based on communication information (S220).

  In S230, the attention distribution (hereinafter also referred to as “individual distribution”) generated in S130, S140, S160, S180, S200, and S220 is integrated. In this case, all may be integrated (simple addition or simple average) with the same weight, or may be individually weighted and integrated (weighted addition or weighted average). Hereinafter, the attention distribution generated in this step is referred to as “integrated distribution”.

  FIG. 3 is an explanatory diagram showing an image of the processes of S130 to S230, that is, a process of generating a plurality of individual distributions based on the specific information and integrating these to generate a final integrated distribution. is there.

  In subsequent 240, for each of the objects extracted in S110, an attention amount indicating an estimation result of how much the driver's attention is directed to the object is obtained using the integrated distribution obtained in S230. Specifically, the attention amount is set to either large, medium, or small according to the distribution value of the area where the object exists on the integrated distribution. That is, if the distribution value is equal to or greater than the first threshold, the attention amount is large. If the distribution value is equal to or smaller than the second threshold set to a value smaller than the first threshold, the attention amount is small. If the distribution amount is smaller than the first threshold and greater than the second threshold. The amount of attention is medium.

  As the distribution value of the area where the object exists, the sum of all distribution values belonging to the area may be used, or the average value of the distribution values belonging to the area may be used. Then, the attention value is set by comparing the distribution value of the area calculated in this way with a preset threshold value.

In subsequent S250, the importance is set to one of large, medium, and small according to a predetermined rule for each of the objects extracted in S110.
As the rule here, an IF-THEN rule that uses a situation specified from specific information, a position of an object, a correlation between objects, or a combination thereof as a condition part is used.

  Specifically, for example, if the “turner state” that is one of the vehicle information is set to “turn right” (condition part), the importance level of the object located on the lower right side of the screen is set to be large. If the rule (consequence part), “distance to object (presumed to be a traffic signal or road sign)” is less than or equal to “predetermined value” (condition part), the importance of the object is set to a large value (consequence part) ), “Object presumed to be a pedestrian crossing” (condition part), the importance of an object presumed to be a pedestrian located on that object and near both ends is greatly increased. A rule of setting (consecutive part) can be considered. Regarding rule setting, it is conceivable to use correlation calculation, Apriori algorithm, or the like.

  In subsequent S260, based on the attention amount of the object set in S240 and the importance level of the object set in S250, whether or not to provide the driver with information on the object, and any format to be presented The information is provided according to the determination result, and this processing is terminated.

  Here, FIG. 4 is an explanatory diagram showing the contents of the presentation control table used when the determination in S260 is performed. In FIG. 4, ◯ indicates that information is presented, Δ indicates that presentation / non-presentation is determined according to the driver state, and x indicates that information is not presented. In other words, if the importance of an object is large, information is presented regardless of the amount of attention. If the importance is medium, information is presented if the attention amount is medium or small, and if the attention amount is large. The presentation / non-presentation is determined according to the driver status. Furthermore, when the degree of importance is small, information is presented if the attention amount is small, presentation / non-presentation is determined according to the driver state if the attention amount is medium, and not presented if the attention amount is large. .

  Note that the driver's state refers to an abrupt driving degree, an arousal degree, a driving skill level, etc. required from biological information, vehicle information, etc., for example, whether the driving degree is absurd, the arousal level is low, or the driving skill level is When it is low, control is performed to present information.

  In addition, as a method of presenting information, if the importance level is large, presentation using all of tactile sensation (vibration), hearing (sound), and vision (display) is performed, and if the importance level is medium, hearing (sound) , Presentation using visual (display) is performed, and if importance is small, presentation using only visual (display) is performed.

  In the above-mentioned control, the method of presenting information is controlled by importance. However, if the amount of attention is large, presentation is performed using only visual information. If the amount of attention is medium, auditory and visual are used. If the attention amount is small, the presentation may be performed using all of tactile sensation, hearing, and vision.

Furthermore, the method of presenting information may be controlled in association with the marks “◯”, “Δ”, and “X” in FIG. 4, that is, based on both importance and attention amount.
[Caution model]
Next, attention model learning performed in advance will be described.

  Note that the caution model may be learned in the vehicle, but in the present embodiment, a caution model is used. When learning the attention model, data collected for many drivers is used.

  For data collection, as shown in FIG. 5, a vehicle equipped with the in-vehicle device for learning 5 in which the gaze detection device 30 for detecting the driver's gaze is added to the in-vehicle device 2 described above, or information necessary for learning is collected. Use a collectable drive simulator. However, in the in-vehicle device 5 for learning, unlike the in-vehicle device 2, the configuration for providing information (information presentation unit 26) may be omitted.

[Learning process]
In the learning process, first, a plurality of subjects drive a vehicle equipped with the on-vehicle device for learning or cause a plurality of subjects to use a drive simulator, and thereby the subject's gazing point (detection result by the line-of-sight detection device 30) and Specific information (vehicle information, environmental information, driver information, communication information) is acquired.

  Then, the specific information is divided into a plurality of types of generated information groups composed of one or a plurality of information belonging to the specific information, and the relationship between the value of the generated information group and the gaze point is determined for each generated information group. learn. This learning uses a statistical method such as a support vector machine (SVM) or a mixed Gaussian distribution (GMM).

  The generated information group may be based on classification of specific information (vehicle information, environmental information, driver information, communication information), or may be a unit obtained by grouping specific information for each highly relevant information. However, individual specific information may be used as a unit.

  In learning using the SVM, the subject's gazing point is used as teacher data, for example, a positive flag is set around the gazing point position of the photographed forward image, and a negative flag is set at a position other than the gazing point to generate information. With the information belonging to the group as an input, learning is performed so that a discriminator that classifies whether or not the forward image is a gazing point for each predetermined region (for example, pixel) based on binary values of 0 and 1 is obtained.

  Also, in learning using GMM, for some of the information contents belonging to the generated information group (for example, steering angle, own vehicle speed, etc.), a GMM that expresses the relationship between each piece of information and the point of interest position in the front image is used. learn. By further combining these GMMs by addition or multiplication, it is probabilistically estimated whether or not the forward image is a gazing point for each predetermined region (for example, pixel) by a continuous value between 0 and 1. An estimator is configured.

[effect]
As described above, the in-vehicle device 2 generates the attention (gaze point) where the driver's attention is directed based on the prepared attention model and the specific information detected at that time. Estimate from the distribution, calculate the driver's attention amount for each object in the forward image from the attention distribution, and use the preset rules to determine the importance of each object according to the situation specified from the specific information Setting and controlling the provision of information to the driver based on the amount of attention and importance.

  Note that the attention model is created so as to reflect information indicating the driver's intention and information affecting the driving behavior of the driver. In other words, by using a caution model, depending on the type of steering operation such as turning left or right or going straight (information that reflects the driver's intention), the level of the vehicle speed or the color of the traffic light (information that affects the driver's driving behavior) Different attention distributions will be obtained.

  Therefore, according to the vehicle-mounted device 2, according to the occasional situation (including the presence of an object that affects the driver's intention and the driving behavior of the driver, etc.), accurate information provision considering the situation is realized. Can do.

  Moreover, according to the in-vehicle device 2, since the driver's gaze point is estimated from the attention model and the specific information, it is simpler and less expensive than a conventional device that requires a measuring instrument that measures the gaze point. Despite the simple configuration, more accurate control can be realized.

Here, FIG. 6 (a) shows the attention distribution percentage area indicating the percentage of the distribution value (attention level) included in the attention distribution generated from the above attention model , the driver's attention point, and Is a graph showing the result of measuring the coincidence rate within a predetermined time width (0.33 [s]), and (b) shows the attention distribution (integrated distribution) generated based on the attention model as the original forward image. The top three objects with the highest distribution values are also indicated by circle numbers.

  As the front image, actual vehicle data acquired in a general city road including various situations such as turning right and left, going straight, following traveling, and passing in the actual environment as shown in FIG. 7 was used. The estimator using the GMM described above is used as the caution model, and the own vehicle speed, the steering angle, and the vehicle detection result are used as the specific information input to the caution model. As a result, a gaze model that can cope with the various situations described above was constructed.

  In FIG. 6A, as a comparative example, the result of estimating the gazing point from the low-order feature amount (luminance, color, edge, etc.) of the image using the method of Non-Patent Document 1 (conventional method), In addition, the coincidence rate (chance level) when gaze points are set at random is also shown.

  As can be seen from the figure, the conventional method is less than or equal to the chance level, whereas the method applied to the in-vehicle device 2 has a high coincidence rate with the actual line of sight by 30% or more, and the driver's intention and driver It can be understood that the accuracy of gaze point estimation can be remarkably improved by reflecting information such as objects that affect the driving behavior of the user in the attention model.

  Further, according to the in-vehicle device 2, various types of specific information are used when generating the attention distribution (individual distribution) from a plurality of attention models, but the individual distribution, As a result, since an integrated distribution can be generated, the present invention can be applied to various vehicles having different specific information that can be acquired.

  FIG. 8 uses the same attention distribution (integrated distribution) as that used in the measurement of FIG. 6, and how the coincidence rate with the gazing point changes by changing the time width required for calculating the coincidence rate. It is a graph which shows the result of having measured whether it arises. However, the time width required for calculating the coincidence rate is 0.33 [s], 1 [s], 2 [s], 5 [s], 10 [s], and the time width is 0.33 [s]. The difference between the coincidence rate (reference value) and the coincidence rate in other time widths is shown.

  In the conventional method of generating the attention distribution using only the information obtained from the image, the accuracy greatly varies depending on the setting of the time width, whereas the accuracy variation degree is achieved by the method of the present invention applied to the in-vehicle device 2. It can be seen that it is small and has improved robustness.

[Correspondence with Invention]
In the embodiment described above, the camera that acquires the front image in the environment sensor group 23 is the image acquisition unit, S120 is the object extraction unit, and the specific information recording process executed by the control unit 29 constitutes the information acquisition unit and the control unit 29. A nonvolatile memory in which a caution model is stored corresponds to a model storage unit, S110 to S230 (excluding S120) correspond to a caution distribution generation unit, and S250 corresponds to an importance setting unit.

[Other Embodiments]
Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and can be implemented in various modes without departing from the gist of the present invention.

  For example, in the above embodiment, the information to be presented is controlled using both the attention amount of the object and the importance level of the object, but the control is performed using only one of them. Also good.

  In the above embodiment, the provision information is controlled according to the importance of the object. However, when the attention distribution based on the object is generated (S140), or the attention distribution is integrated with another attention distribution ( In S230), the distribution of the object may be reflected in the integrated distribution by weighting the distribution corresponding to each object according to the importance of the object. In this case, the process of setting the importance level of the object (S250) may be performed at least before the process of reflecting the importance level of the object (S140 or S230).

  DESCRIPTION OF SYMBOLS 1 ... Driving assistance system 2 ... In-vehicle apparatus 3 ... Communication infrastructure 4 ... Information server 5 ... In-vehicle apparatus 21 for learning 21 ... Vehicle sensor group 22 ... Biosensor group 23 ... Environmental sensor group 24 ... GPS receiver 25 ... Wireless communication part 26 ... Information presentation unit 27 ... command input unit 28 ... map database 29 ... control unit 30 ... gaze detection device

Claims (8)

Information acquisition means for acquiring specific information that is information that reflects the driver's intention for driving behavior or information that affects the driving behavior of the driver;
Model storage means for storing a caution model indicating a relationship between a value indicated by the specific information and a gaze point that is a position on the image where the driver's attention is directed;
Based on the attention model stored in the model storage means, attention distribution generation means for generating an attention distribution in which the gaze point is estimated from specific information detected by the information acquisition means;
Equipped with a,
The attention model stored in the model storage means is prepared for each of a plurality of types of generated information groups composed of one or a plurality of information belonging to the specific information,
The attention distribution generation unit generates, for each attention model, an individual distribution in which the gaze point is estimated from the specific information detected by the information acquisition unit, and generates the attention distribution by integrating the individual distributions. A gaze point estimation device characterized by the above. Image acquisition means for acquiring an image corresponding to the driver's field of view;
Object extraction means for extracting an object from the image acquired by the image acquisition means;
Importance level setting means for setting the importance level of the object extracted by the object extraction means according to a preset rule,
The attention distribution generation unit generates an individual distribution based on the object extracted by the object extraction unit, and generates an individual distribution based on the object, or an individual distribution based on the object is replaced with another individual distribution. The gaze point estimation apparatus according to claim 1 , wherein when integrating, weighting is applied to a distribution corresponding to each object in accordance with the importance set by the importance setting means. The rule used by the importance setting unit for setting the importance includes an IF-THEN rule whose condition is a condition specified from the specific information acquired by the information acquisition unit. The gaze point estimation apparatus according to claim 2. The rules used by the importance setting means for setting the importance include IF-THEN rules that use the position of the object extracted by the object extraction means or the correlation between objects as a condition part . The gaze point estimation apparatus according to claim 2 or claim 3, wherein Wherein the specific information, according to any one of claims 1 to 4, characterized in that it contains the vehicle information including at least one of information indicating the behavior of the information or the vehicle about the operation of the driver Gaze point estimation device. Wherein the specific information, gaze point estimating apparatus according to any one of claims 1 to 5, characterized in that it contains the information der Ru driver information about the driver of the biological response. The gaze point estimation apparatus according to any one of claims 1 to 6 , wherein the specific information includes environment information that is information related to a traveling environment of the vehicle. The attention viewpoint estimation apparatus according to claim 7 , wherein at least part of the environmental information is acquired by road-to-vehicle communication or vehicle-to-vehicle communication.