JP2016062330A - Hyperprosexia state determination device and hyperprosexia state determination program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a hyperprosexia state determination device capable of highly accurately determining a hyperprosexia state of a determination object person regardless of an error due to installation position of an object detection device and a measurement error in a visual line direction.SOLUTION: On the condition that the visual line direction of a driver is directed to a direction of one specific object for a prescribed time and an angular velocity θev in the eyeball direction is equal to or less than a first threshold Th1 representing that no saccade occurs, the hyperprosexia state determination device determines that the driver is in the hyperprosexia state where the visual line of the driver is directed to the specific object.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an over-attention state determination device and an over-attention state determination program for determining that an eye of a determination target is in an over-attention state fixed to a specific object.

  For example, Patent Document 1 discloses an apparatus that detects that a driver is over-careful of a preceding vehicle based on the line-of-sight direction of the driver of the automobile. In the apparatus of Patent Document 1, an over-attention state determination region with respect to a preceding vehicle is set. Then, under a preset condition, the gaze stop time in the over-attention state determination region for the past 10 seconds is measured, and the ratio of the measured gaze stop time to the reference stop time is calculated. Furthermore, the average line-of-sight angle for the past 10 seconds with respect to the preceding vehicle is calculated. And it determines with an over-attention state when the ratio of a stop time is more than the preset threshold value, and an average gaze angle is less than the preset threshold value.

JP 2007-73011 A

  For example, in an automobile, a camera can be used as an object detection device that detects an object that a driver should pay attention to. However, the mounting position of the camera is different from the head position of the driver. For example, a camera that captures the front in the traveling direction of an automobile has a typical installation location between a rearview mirror and a windshield. In this case, a displacement of about 50 cm in the front-rear direction of the host vehicle and about 50 cm in the left-right direction occurs between the camera and the driver's head position. Due to this deviation, an error occurs between the direction of the object detected by the object detection device and the direction of the driver's line of sight.

  Here, as shown in FIG. 7, taking a case where a pedestrian crosses a pedestrian crossing on a one-lane road as shown in FIG. 7, between the pedestrian direction detected by the camera and the driver's gaze direction, An explanation will be made as to whether a certain degree of error occurs. It should be noted that both the pedestrian direction and the driver's line-of-sight direction detected by the camera are represented by an angle formed with the reference direction with the front direction of the automobile as the reference direction. In the example shown in FIG. 7, the host vehicle is stopped before the intersection, and the camera and the driver's head position are shifted by 50 cm in the front-rear direction and the left-right direction, respectively. It is located 2m from the left end, and the pedestrian crosses the pedestrian crossing on the other side of the intersection as seen from the own vehicle at a position 1.5m away from the intersection road.

  Under the conditions shown in FIG. 7, when the pedestrian on the left in the figure walks to the position of the pedestrian on the right in the figure, the angle θ [ deg] and the angle φ [deg] between the front direction of the host vehicle with the driver head position as the origin and the direction in which the pedestrian is present are shown in the graph of FIG. As shown in the graph of FIG. 8, when there is a pedestrian in front of the host vehicle, an error of about 3.2 [deg] occurs between the angle θ and the angle φ. This error is not constant according to the installation position of the camera, but increases and decreases according to the distance and direction to the object (pedestrian). In the example shown in the graph of FIG. 8, the error changes from about 1.6 [deg] to about 3.7 [deg] before the pedestrian crosses the pedestrian crossing and finishes moving from the beginning.

  Furthermore, in addition to errors due to the camera installation position, there is a possibility that errors may be included in the driver's gaze direction measurement (for example, “Gaze Measurement Method Based on Eyeball Shape Model”, 8th Image Sensing Symposium, pp. 307-312, 2002).

  Therefore, as in the device of Patent Document 1, when trying to determine excessive attention by the gaze stop time for the object, the target is to cover the error range that combines the error due to the camera installation position and the measurement error in the gaze direction. It is necessary to set a region larger than the region of the object and use the line-of-sight dwell time for the set large region.

  However, since the error range has a width of several degrees, a plurality of objects may exist within the error range. For this reason, there is a problem that, for example, a state where safety is confirmed, such as driving while alternately shifting the line of sight between an oncoming vehicle and a pedestrian behind it, is detected as excessive attention.

  The present invention has been made in view of the above-described points, and determines the state of excessive attention of the determination target person with higher accuracy regardless of the error due to the installation position of the object detection device or the measurement error in the line-of-sight direction. An object of the present invention is to provide an over-attention state determination device capable of performing the above.

In order to achieve the above object, an over-attention state determination device according to the present invention provides:
Object detection means (S140, S150) for detecting an object that can be a gaze target of the determination target person and calculating a direction in which the detected target exists, around the determination target person,
Gaze direction detecting means (S110 to S130) for detecting the gaze direction of the person to be determined;
Eye movement speed detection means (S180) for detecting the speed of eye movement of the person to be determined;
The eye movement speed detected by the eye movement speed detection means while the line-of-sight direction of the determination subject is directed to the direction of one specific object detected by the object detection means for a predetermined time. Over-judgment determining means (S190) for determining that the line of sight of the determination target is in an over-attention state in which the gaze of the determination target is poured on a specific object.

  Here, it is known that when a person changes his gaze target, he / she performs a short high-speed eye movement called saccade. The saccade is, for example, much faster than the followable eye movement that occurs when the determination target person follows the moving object with his / her eye, and can be easily distinguished from the followable eye movement.

  In the present invention, it is intended to improve the determination accuracy of the over-attention state of the determination target person using such characteristics of eye movement. Therefore, in the present invention, the eye movement direction of the determination target person is directed to the direction of one specific object for a predetermined time, and the eye movement speed is equal to or less than the first threshold value. As a condition, the determination target person's line of sight was determined to be in an over-attention state poured on a specific object.

  Therefore, in the present invention, it is determined that an over-attention state is determined only when the line-of-sight direction of the determination target person is directed to a specific object and high-speed eye movement is not occurring. In other words, the change in the line-of-sight direction due to eye movement is within the error range due to the error due to the installation position of the object detection means, the detection error due to the line-of-sight direction detection means, etc., and is considered to remain directed to the specific object. However, if the speed of the eye movement is greater than the first threshold value and indicates saccade, the present invention does not determine that the state is over-attention. As a result, the chance of erroneously determining an over-attention state can be reduced, and the determination accuracy of an over-attention state can be improved as compared with the conventional case.

  The reference numerals in the parentheses merely show an example of a correspondence relationship with a specific configuration in an embodiment described later in order to facilitate understanding of the present invention, and are intended to limit the scope of the present invention. Not intended.

  Further, the technical features described in the claims of the claims other than the features described above will become apparent from the description of embodiments and the accompanying drawings described later.

It is a block diagram which shows an example of a structure of the warning system for vehicles containing the excessive attention state determination apparatus by embodiment. It is a flowchart which shows the process performed in an excessive attention state determination part including an excessive attention state determination process. It is a graph which shows the change of the gaze direction of the both eyes measured when the driver | operator of the own vehicle was instruct | indicated not to deflect | deviate the line of sight to the oncoming vehicle which drives toward the own vehicle. It is a graph which shows the change of the gaze direction of both eyes when the driver | operator of the own vehicle changes eyes to an oncoming vehicle and the pedestrian in the back alternately. It is the graph which showed the angular velocity of the eye movement of one eye with the change of the gaze direction of both eyes when the driver | operator of the own vehicle is gazing at an oncoming vehicle. It is explanatory drawing for demonstrating the effect by the excessive attention state determination apparatus by embodiment. It is explanatory drawing for demonstrating how much error arises between the direction of a pedestrian detected by a camera, and a driver | operator's gaze direction. When the pedestrian on the left in the figure walks to the position of the pedestrian on the right in the figure under the conditions shown in FIG. 7, the angle θ between the front direction of the vehicle with the camera position as the origin and the direction in which the pedestrian is present 4 is a graph showing an angle φ formed by the front direction of the host vehicle with the driver's head position as the origin and the direction in which the pedestrian is present.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. In the present embodiment, an example in which the over-attention state determination device is applied to determine the over-attention state of the driver of the automobile will be described. FIG. 1 is a configuration diagram illustrating an example of a configuration of a vehicle warning system 1 including an over-attention state determination device according to the present embodiment. First, an over-attention state determination apparatus will be described with reference to FIG.

  The over-attention state determination device includes a driver camera 10, an out-of-vehicle camera 12, and an over-attention state determination unit 22 embodied as a function of the control device 20 of the vehicle warning system 1. However, the control device for realizing the excessive attention state determination unit 22 may be different from the control device 20 of the vehicle warning system 1. Further, when implementing the over-attention state determination unit 22 as one function of the control device 20, software programmed by the computer so as to exhibit the function as the over-attention state determination unit 22 may be used. The device 20 may be configured by hardware.

  The driver camera 10 is a driver seat camera that captures an image of the entire face of the driver seated in the driver seat. The driver camera 10 is arranged, for example, in an instrument panel in which various instruments are arranged, or on the upper surface of a steering column containing a steering shaft so as to face the driver. The driver camera 10 periodically captures an image including the driver's face and outputs the image to the over-attention state determination unit 22. A plurality of driver cameras 10 may be provided so as to capture the driver's face from different angles.

  For example, in order to photograph the front of the vehicle, the outside-vehicle camera 12 is fixed to the rear side of the rearview mirror, the rear side of the rearview mirror as viewed from the driver, and the inner upper portion of the front windshield. The outside-vehicle camera 12 periodically captures images and outputs the captured images to the excessive attention state determination unit 22. The vehicle camera 12 may be a monocular camera or a stereo camera. In addition, the means for detecting the object is not limited to the outside camera 12, but a radar device that detects a front obstacle using a laser beam, millimeter wave, or the like instead of the outside camera 12 or together with the outside camera 12. May be used.

  The over-attention state determination unit 22 performs image processing such as pattern matching on the image input from the camera 12 outside the vehicle, so that it exists in front of the vehicle and may cause the driver to pay attention (vehicle) , Pedestrians, road signs, billboards, etc.). Of the detected objects, with reference to the previous detection result, a stationary object is recognized as a stationary object, and a moving object is recognized as a moving object. The over-attention state determination unit 22 calculates a direction with respect to the host vehicle with respect to the detected object. Specifically, the direction of the object is calculated as an angle between the front direction of the host vehicle and the reference direction. In this case, for example, the angle from the front direction to the right direction of the host vehicle can be expressed as a positive angle, and the rotation angle in the left direction can be expressed as a negative angle. Further, the over-attention state determination unit 22 calculates the distance from the host vehicle to the target object or detects the size of the target object displayed in the image as necessary.

  As described above, the over-attention state determination unit 22 detects an object with which the driver may pay attention and calculates the direction of the object, while executing the detection process of the driver's gaze direction. To do. Specifically, the over-attention state determination unit 22 first detects the angle of the driver's face direction based on the image including the driver's face captured by the driver camera 10. For example, when the driver's face orientation angle is detected, the over-attention state determination unit 22 first determines a face area that includes the driver's face in the image captured by the driver camera 10, and includes the face area in the face area. , Recognize the main elements that make up the face (eyes, nose, mouth, etc.). Next, the over-attention state determination unit 22 calculates the driver's face orientation angle based on the relative positional relationship of the recognized major elements of the face.

  Further, the over-attention state determination unit 22 detects the angle (eyeball direction) in which the eyeball is directed with respect to each of the left and right eyeballs of the driver from the image captured by the driver camera 10. For example, it is called the Purkinje image that is generated when the eyeball direction is detected (sclera reflection method) using the difference in reflectance to the cornea (black eye) and sclera (white eye) light, or when near infrared light is irradiated. The eyeball direction can be detected from the reflected image (corneal reflection method), or the center position of the pupil can be estimated by image analysis to detect the eyeball direction. Note that the over-attention state determination unit 22 may detect the driver's eye direction by using other known methods (such as the EOG method or the search coil method) without using the driver camera 10.

  Then, the over-attention state determination unit 22 calculates the gaze directions of the left and right eyes of the driver based on the detected driver's face angle and eye direction. That is, the over-attention state determination unit 22 calculates an angle indicating the line-of-sight direction of the driver's left and right eyes from the eye direction of each of the left and right eyes based on the driver's face angle. At this time, the driver's line-of-sight direction is also calculated as an angle between the front direction of the host vehicle and the reference direction as in the detection of the direction of the object. The over-attention state determination unit 22 may determine the driver's line-of-sight direction in a two-dimensional manner as a movement in the horizontal direction (left-right direction), or in a three-dimensional manner as a movement in the horizontal direction and the vertical direction. Also good.

  Furthermore, the over-attention state determination unit 22 is configured such that the driver's line of sight is poured into one specific object based on the detected direction of the object and the line-of-sight direction of the driver's left and right eyes. It is determined whether or not there is an excessive state. Hereinafter, the processing executed in the over-attention state determination unit 22 including this over-attention state determination processing will be described in detail with reference to the flowchart of FIG. Note that the flowchart of FIG. 2 is repeatedly executed at a predetermined cycle as long as the driving of the automobile is continued.

  First, in step S100, the excessive attention state determination unit 22 performs recognition processing of main elements of the driver's face in the image captured by the driver camera 10 as described above. In step S110, the face orientation angle θf is calculated from the relative positional relationship of each element recognized in step S100. In the subsequent step S120, the eyeball direction θe of the driver's left and right eyes is calculated by extracting features related to the angle at which the driver's left and right eyes are facing from the image captured by the driver camera 10.

  In step S130, the driver's line-of-sight direction θ is calculated from the eyeball directions θe of the left and right eyes with respect to the left and right eyes, respectively, based on the driver's face orientation angle θf (θ = θf + θe). An example of the driver's line-of-sight direction θ calculated in this way is shown in the graph of FIG. The graph of FIG. 3 shows the results measured when the driver of the host vehicle is instructed not to deflect the line of sight of the oncoming vehicle traveling toward the host vehicle.

  As shown in FIG. 3, when the driver is gazing at the oncoming vehicle, in principle, the directions of the eyes of the left and right eyes follow the movement of the oncoming vehicle, and therefore change in substantially the same manner. However, as shown by a circle drawn with a dotted line in FIG. 3, a movement of the line of sight that is not normally considered may be detected. Therefore, when calculating the gaze direction θ of the left and right eyes, smoothing processing may be performed using a smoothing filter (for example, a median filter) or the like in order to remove a gaze movement that is not normally considered as noise. desirable.

  In subsequent step S140, based on the eyeball direction θe (t) calculated in the current cycle and the eyeball direction θe (t−1) calculated in the previous cycle, an angular velocity θev that is a temporal change in the eyeball direction θe. Is calculated. Since the eyeball direction θe is calculated for each predetermined control period in step S120, in step S140, as the temporal change of the eyeball direction θe, the eyeball direction θe (t) calculated in the current period and the previous time A simple difference from the eyeball direction θe (t−1) calculated in the period may be obtained.

  Here, when the driver gazes at a certain target object, as shown in FIG. 3, the line-of-sight direction changes so that the left and right eyes follow the movement of the target object. On the other hand, when the driver changes the gaze target, a short high-speed eye movement called saccade is observed in both the left and right eyes. For example, FIG. 4 is a graph showing a change in the line-of-sight direction of the left and right eyes when the driver alternately shifts his / her line of sight to an oncoming vehicle and a pedestrian behind it. FIG. 4 also shows that the gaze direction of the left and right eyes changes rapidly when the driver changes the gaze target. Note that the angle at the left end of the graph of FIG. 4 greatly increases or decreases because of the driver's blink.

  Here, FIG. 5 shows the angular velocity of the eye movement of one eye (amount of change [deg] of eye movement) along with the change in the line-of-sight direction of both eyes when the driver is gazing at the oncoming vehicle. As shown in FIG. 5, when the driver is gazing at the oncoming vehicle, followable eye movement occurs, but the angular velocity of the eye movement is very small. For example, in the data shown in FIG. 5, it is understood that the amount of change in eye movement is within 1 [deg] when the direction of the oncoming vehicle is within 10 [deg] from the front direction of the host vehicle.

  Thus, as is apparent from the graphs of FIGS. 4 and 5, the saccade is considerably faster than the following eye movement that occurs when the driver follows the moving object such as an oncoming vehicle with his eyes. It can be easily distinguished from following eye movement. Therefore, as will be described later, whether or not saccade is generated can be determined with high accuracy by comparing the angular velocity of eye movement (the amount of change in eye movement) with a predetermined first threshold value Th1.

  In the subsequent step S150, an object on which the driver may pay attention is detected based on the image captured by the vehicle exterior camera 12. In step S160, the direction φ of the detected object relative to the host vehicle is calculated.

  In step S170, based on the detection result of the object in step S150, it is determined whether or not there is an object around which the driver may turn his attention. If it is determined that the object is present, the process proceeds to step S170. If it is determined that the object is not present, the process returns to step S100.

In step S180, the left and right eyes of the driver are used as indicators for confirming whether the line-of-sight direction θ of the left and right eyes of the driver of the host vehicle is directed to the direction φ of one specific object. A correlation value Γ indicating how much the change in the line-of-sight direction θ and the change in the direction φ of the object correlate is calculated for each of the left and right eyes using the following Equation 1.

  Note that t−N and t represent the time (time point) at which the line-of-sight direction θ and the object direction φ are detected, and the bar symbol added on θ and φ is a certain time point t from a certain time point t−N. Represents the average value of the line-of-sight direction θ and the average value of the direction φ of the object. Further, when a plurality of objects are detected, the correlation value Γ is calculated for each object.

  The correlation value Γ is the covariance between the time-series data in the line-of-sight direction θ and the time-series data in the object direction φ in a predetermined period (for example, 2 seconds) from the time point tN to the time point t. Is divided by the standard deviation of the time-series data and the standard deviation of the time-series data of the object direction φ. For this reason, the correlation value Γ takes values from −1 to 1, and the line-of-sight direction θ and the object direction φ change in the same manner, and takes a value closer to 1 as the positive correlation is stronger.

  Here, as described above, the line-of-sight direction θ and the object direction φ have an offset error due to the difference between the installation position of the vehicle camera 12 and the head position of the driver. However, in the denominator and numerator, the correlation value Γ of Equation 1 subtracts the average value of the line-of-sight direction θ for the predetermined period from each time-series data of the line-of-sight direction θ for the predetermined period, and The average value of the object direction φ for the predetermined time is subtracted from each of the time series data of the direction φ. For this reason, the correlation value Γ in Equation 1 is an index that correctly indicates whether the line-of-sight direction θ and the object direction φ are changing in the same manner regardless of the offset error described above.

  Furthermore, since the correlation value Γ in Equation 1 uses the standard deviation of the time-series data in the line-of-sight direction θ and the standard deviation of the time-series data in the object direction φ as the denominator, the angle between the line-of-sight direction θ and the object direction φ is large. Normalized so as not to be affected by the difference. Thus, it is possible to determine whether or not the driver's line of sight is directed toward the object by comparing the magnitude with the constant second threshold Th2, regardless of the angle of the line-of-sight direction θ and the object direction φ. .

  In step S190, all the eyeball direction angular velocities θev calculated in step S140 during the predetermined period are equal to or less than the first threshold Th1, and the correlation value between the line-of-sight direction θ and the object direction φ in the predetermined time. It is determined whether Γ is greater than or equal to the second threshold Th2. The determination process in step S190 is executed individually for each object when a plurality of objects are detected. If a positive determination is made in this determination process, the driver can be considered to be in an excessive attention state with respect to the specific object. Therefore, the process proceeds to step S200. On the other hand, if a negative determination is made, it is considered that an over-attention state for a specific object has not occurred, and the process returns to step S100.

  In the determination process in step S190, it may be determined that an over-attention state has occurred for both the left and right eyes when both of the above conditions are satisfied, and regarding the eyeball direction angular velocity θev, either the left or right eyeball direction is determined. It may be determined that an over-attention state has occurred when the angular velocity θev is equal to or less than the first threshold Th1. When the driver changes the gaze target, saccade is observed in both the left and right eyes. In other words, if a large angular velocity θev in the eyeball direction is detected in any one of the eyes, it may be noise as shown in FIG. 3 instead of saccade. Accordingly, if it is considered that no saccade is generated when the left-right eyeball direction angular velocity θev is equal to or less than the first threshold Th1, noise as shown in FIG. 3 is detected in either the left-right eyeball direction θe. Even if this occurs, it is possible to prevent erroneous determination that an over-attention state has not occurred.

  As described above, in the present embodiment, the eyeball indicating that the driver's line-of-sight direction is directed to the direction of one specific target object and that no saccade has occurred for a predetermined time. On the condition that the direction angular velocity θev is equal to or less than the first threshold Th1, it is determined that the driver's line of sight is in an over-attention state. Therefore, it is determined that the state of excessive attention is determined in the present embodiment only when the line-of-sight direction of the determination target is directed to a specific target and high-speed eye movement is not occurring.

  For this reason, as shown in FIG. 6, the change of the gaze direction due to the eyeball movement is within the error range due to the error due to the installation position of the outside camera 12, the detection error of the gaze direction, etc. Even if it is considered that the object is still directed toward the object, it is not determined as an excessive attention state if the angular velocity θev in the eyeball direction is larger than the first threshold Th1 and indicates saccade. . As a result, according to the present embodiment, it is possible to reduce the chance of erroneously determining an over-attention state, and it is possible to improve the determination accuracy of an over-attention state as compared with the prior art.

  In subsequent step S200, the warning output control unit 24 is instructed to alert the driver to notify the specific target object that the user is in an excessive attention state.

  Here, an example of the structure for performing alerting and the content of alerting will be described. The vehicle warning system 1 includes a display 30 for displaying a warning message and a speaker 32 for generating a voice warning message and a warning sound.

  The warning output control unit 24 outputs an operation signal for operating the display control unit 26 and the audio output control unit 28 when instructed to execute alerting from the over-attention state determination unit 22. Then, the display control unit 26 displays a warning message on the display 30, and the voice output control unit 28 generates a voice warning message or a warning sound from the speaker 32. Note that the display 30 may be omitted as a configuration for alerting. Moreover, it replaces with the speaker 32 and it may comprise so that a driver's seat seat and a steering may be vibrated, and you may warn a driver | operator that it is an excessive attention state.

  Finally, in step S210, for example, based on the ignition switch being turned off, it is determined whether or not the driving of the automobile has ended. If it is determined that the operation has ended, the processing shown in the flowchart of FIG. 2 is ended. On the other hand, if it is determined that the operation has not ended yet, the process returns to step S100.

  The preferred embodiments of the present invention have been described above. However, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

  For example, in the above-described embodiment, the first threshold value Th1 for determining whether the angular velocity θev in the eyeball direction indicates saccade has been described as a constant value. It may change depending on the factors.

  For example, when the direction φ of the detected object changes with time, the magnitude of the first threshold Th1 may be changed according to the changing speed at which the object direction φ changes. Specifically, the first threshold value Th1 is changed so as to increase as the change speed of the object direction φ increases. Accordingly, it is possible to more reliably prevent the followable eye movement that follows the movement of the object from being erroneously determined to be saccade.

  Further, when the magnitude of the first threshold Th1 is changed according to the change speed of the object direction φ, only the magnitude of the first threshold Th1 with respect to the speed of eye movement in the direction following the change of the object direction φ. You may make it enlarge. As a result, the first threshold value Th1 with respect to the angular velocity θev in the eyeball direction in the direction following the change in the object direction φ is the first threshold value Th1 with respect to the angular velocity θev in the eyeball direction in the direction opposite to the change direction in the object direction φ. Larger than the size of

  Further, the size of the first threshold Th1 may be changed according to the size of the object in the image. Specifically, the size of the first threshold is increased as the size of the object increases. Thereby, when the gaze location is changed in the same object, the angular velocity of the eye movement can be suppressed from exceeding the first threshold Th1.

  In the above-described embodiment, when it is determined that an over-attention state in which the driver of the vehicle is gazing at a specific object, the driver is alerted. However, instead of the above-described alerting, or together with the above-described alerting, the alerting as described below may be performed.

  In other words, when there is another target other than the target that the driver is over-cautioned from the detection result of the target, there is a possibility that the driver will collide with the other target. You may give a warning to warn. Regarding the possibility of collision, first, it is determined from the detected direction of each object whether or not there is another object on the course of the host vehicle. And when it determines with another target object existing on the course of the own vehicle, the distance with the other target object is acquired. Further, the relative speed between the host vehicle and another object is calculated from the time change of the distance. Then, using the distance and relative speed with the other object, a collision prediction time TTC (Time To Collision) that is a remaining time until it is predicted that the object collides with the other object is calculated. The possibility of collision can be determined from the size of the predicted collision time TTC.

  And when the other target object with a collision possibility is detected, you may alert in the following aspects. For example, when a pedestrian is detected as an object that may collide, the situation in front of the vehicle is displayed on the display, and the pedestrian is emphasized by changing the brightness and color, zooming in, or enclosing with a circle. May be displayed. Further, the pedestrian direction viewed from the driver may be indicated by an arrow on a head-up display or the like. In addition, a message indicating that the alert target is a pedestrian may be displayed on the display.

  In addition, as a reminder using a speaker, a warning sound that is different from the warning sound for notifying an over-attention state is generated to alert the user, or the user may be alerted by a voice message such as “Right, pedestrian”. May be.

  And the aspect of alerting by a display or a speaker may differ according to the high possibility of a collision. For example, when highlighting on a display, the degree of emphasis may be increased as the likelihood of collision increases. Further, the volume of the warning sound or voice message may be increased as the possibility of collision increases.

  In the above-described embodiment, the example in which it is determined whether or not the driver's line of sight is directed in the direction of the specific object based on the magnitude of the correlation value shown in Equation 1 has been described. However, whether or not the driver's line of sight is directed toward a specific object is determined, for example, by setting an over-attention state determination region based on the object, as in the past, and in the over-attention state determination region. You may determine based on the ratio of the stop time of eyes | visual_axis.

  Further, in the above-described embodiment, the angular velocity θev in the line-of-sight direction θ and the eyeball direction is obtained for both the left and right eyes of the driver, and the driver determines based on the line-of-sight direction θ and the angular velocity θev in the eyeball direction for the left and right eyes, respectively. It was determined whether or not an over-attention state in which a gaze is focused on a specific object. However, it may be determined whether or not the driver is in an excessive attention state by detecting only the line-of-sight direction θ and the eyeball direction angular velocity θev of the left or right eye of the driver. However, when the driver is gazing at a specific object and trying to recognize the object, depth information is also a big clue. In order to obtain the depth information, binocular parallax has an important meaning, and the driver can obtain more accurate depth information by capturing the object with both eyes. Therefore, it is possible to make a more accurate determination as to whether or not the driver is paying attention to the object based on the gaze direction θ of the left and right eyes and the angular velocity θev in the eyeball direction.

  Furthermore, in the above-described embodiment, for example, an object is recognized by image processing. For example, when an object that is determined to be an excessive attention state is a road sign, it is possible to watch because the road sign cannot be read. Therefore, you may read out what the road sign means by voice. In addition, if the object that has been identified as being over-charged is a parking sign, it may be looking for a parking space, so the nearest parking space may be guided based on the map information. .

  In addition, when the frequency of determination of an excessive attention state with respect to the preceding vehicle increases, there is a possibility that the driver has suddenly fallen into driving. Therefore, the driver may be encouraged to take a break by voice or a display message.

  In the above-described embodiment, the example in which the over-attention state determination device according to the present invention is applied to determine the over-attention state of the driver of the automobile has been described. However, the application target of the over-attention state determination device according to the present invention is not limited to an automobile. For example, the present invention may be applied to vehicles other than automobiles (for example, trains, ships, airplanes, etc.), or may be applied to detect an excessive attention state for a specific object such as a plant operator. Furthermore, for example, the over-attention state determination apparatus according to the present invention may be used to improve a robot that interacts with a person. That is, in a robot that interacts with a person, if a target that a target person is aware of is known, the possibility that the robot has a more natural conversation can be increased. Further, according to the present invention, the over-attention state determination device may be used for improving the content creation of a moving image advertisement. In other words, if the over-attention state determination device according to the present invention knows what event in the video is attracted to the viewer of the video advertisement, it is expected to be useful for creating video advertisement content that appeals more strongly to people. The

DESCRIPTION OF SYMBOLS 1 Vehicle warning system 10 Driver camera 12 Outside camera 20 Control apparatus 22 Attention excess state determination part 24 Warning output control part 26 Display control part 28 Audio | voice output control part 30 Display 32 Speaker

Claims (14)

Object detection means (S140, S150) for detecting an object that can be a gaze target of the determination target person around the determination target person and calculating a direction in which the detected target exists;
Gaze direction detecting means (S110 to S130) for detecting the gaze direction of the determination target person;
Eye movement speed detection means (S180) for detecting the speed of eye movement of the determination subject,
During the predetermined time, the eye movement of the determination subject is directed to the direction of one specific object detected by the object detection means, and the eye movement detected by the eye movement speed detection means An over-attention determination means (S190) for determining that the gaze of the determination target is in an over-attention state when the specific target is poured when the speed of the image is equal to or less than a first threshold (Th1); An over-attention state determination device comprising: The line-of-sight direction detecting means detects the line-of-sight directions of the left and right eyes of the determination target person,
The eye movement speed detection means detects the speed of eye movement of both the left and right eyes of the person to be judged,
The over-attention determination unit is configured such that the line-of-sight directions of the left and right eyes of the determination target person are both directed toward the specific target object, and the speed of eye movement of at least one of the left and right eyes is the first eye movement speed. The over-attention state determination device according to claim 1, wherein the determination target person determines that the specific object is in an over-attention state when the threshold is equal to or less than a threshold value.   The over-attention determination unit obtains a correlation value (Γ) between time-series data in the line-of-sight direction of the determination target person and time-series data in the direction of the specific object at the predetermined time, and the correlation value is 3. The over-attention state determination according to claim 1, wherein when it is equal to or greater than a second threshold (Th2), the line-of-sight direction of the determination target person is regarded as being directed toward the specific target object. apparatus.   When the excessive attention determination means obtains the correlation value between the time-series data of the gaze direction of the determination target person and the time-series data of the direction of the specific object at the predetermined time, each time series In order to remove the offset error of the data, an average value of the gaze direction of the determination target person at the predetermined time is subtracted from each of the time-series data of the gaze direction of the determination target person, and the direction of the specific target object 4. The over-attention state determination device according to claim 3, wherein an average value of the direction of the specific object at the predetermined time is subtracted from each of the time series data. The object detection means calculates the direction of the object as an angle with respect to a predetermined reference direction,
The line-of-sight direction detecting means detects the line-of-sight direction as an angle with respect to the reference direction,
The over-attention determination unit obtains the correlation value between the time-series data of the gaze direction of the determination target person and the time-series data of the direction of the specific object at the predetermined time. The over-attention state determination device according to claim 3 or 4, wherein normalization is performed so as not to be affected by a difference in angle.   The line-of-sight direction detection means, when detecting the line-of-sight direction of the determination subject, performs a smoothing process for removing noise components in the line-of-sight direction. The over-attention state determination device according to item.   When the direction of the object changes with time, the over-attention determination unit changes the magnitude of the first threshold according to a change speed at which the direction changes, and the change speed is The over-attention state determination device according to any one of claims 1 to 6, wherein the first threshold value is increased as the speed increases.   The over-attention determination means determines the magnitude of the first threshold with respect to the speed of eye movement in a direction following the change in the direction of the object, and determines the amount of eye movement in the direction opposite to the change in the direction of the object. The over-attention state determination device according to claim 7, wherein the over-attention state determination device is configured to be larger than a first threshold value with respect to speed. The object detection means also detects the size of the object,
The over-attention determination means changes the size of the first threshold according to the size of the object. The larger the size of the object, the larger the size of the first threshold. The over-attention state determination device according to claim 1, wherein the over-attention state determination device is increased. It has a warning means (24 to 32) for calling attention to the person to be judged,
The alerting means performs alerting for notifying that the determination target person is in an excessive attention state when the excessive attention determination means determines that the excessive attention state is present. The over-attention state determination device according to any one of 1 to 9.   The over-attention state determination device is configured to determine an over-attention state with respect to an object around the vehicle with the vehicle driver as a determination target. An over-attention state determination device A collision possibility determination means for determining a collision possibility between the vehicle and the object detected by the object detection means;
An alerting means for alerting a driver of the vehicle that is the determination target,
When the collision possibility determination means determines that there is a possibility of collision with an object other than the specific object determined to be an excessive attention state by the excessive attention determination means, the attention calling means includes the The over-attention state determination device according to claim 11, wherein attention is given to a driver of the vehicle.   13. The over-attention state determination device according to claim 12, wherein the alerting unit varies the mode of alerting according to the high possibility of collision determined by the collision possibility determination unit.   The computer as the over-attention state determination device according to any one of claims 1 to 13, wherein the object detection unit, the gaze direction detection unit, the eye movement speed detection unit, and the over-attention determination unit An over-attention state determination program to function.

Images