JP2017224067A - Looking aside state determination device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To properly identify a visual recognition object in a visual line direction of a determination object person, and easily determine a looking aside state in which, a determination object person does not look at the identified visual recognition object.SOLUTION: Based on an image G imaged by an in-vehicle camera 1, a visual feature point which is a visual recognition object which exists on a front side of the vehicle (for example, a preceding vehicle 44, a signal device 45) is detected. Based on the visual feature point and a fixation point of the determination object person, it is determined whether or not, the determination object person is in a looking aside state. For example, when frequency of a visual line movement of the determination object person to the visual feature point is equal to or less than a prescribed value, it can be determined that the determination object person is in a looking aside state in a visual line direction (travel direction). The visual feature point is detected based on a correction image after removing areas R1, R2 in a direction significantly different from the visual line direction, from the image G, and the visual feature point can be detected with restriction to a risk object.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a side-view state determination apparatus.

  A driver's side-view is not preferable for safe driving. Patent Document 1 discloses a method for determining a side look when the traveling direction is not viewed from the amount of movement per jumping motion that occurs when the line of sight moves.

  In the sideways, in addition to the general sideways that do not look at the direction of travel of the vehicle, the sideways direction is seen, but the sideways when the object to be watched is not being looked at (defined as “advancing direction sideways”) ) That is, when the driver's attention is distracted, etc., even if the driver's line of sight is facing the front of the vehicle (the line of sight is the front direction of the vehicle), a dangerous object or the like existing in front of the vehicle is visually recognized. Since the object to be visually recognized is not sufficiently viewed, there is a case where the object is actually a side view. Patent Document 2 discloses a method for determining a sidewalk in a state where the driver's line-of-sight direction is directed forward based on the driver's brain waves.

JP 2003-99899 A JP 2012-85746 A

  It is not common to use a driver's brain wave as described in Patent Document 2 in order to determine whether or not the above-described traveling direction side-viewing, that is, the gaze direction side-seeing (the brain wave detection device is quite large and costly. ), It is desired to be able to make a determination by a simpler method. Further, in order to determine whether or not it is a side-viewing, it is necessary to specify an object to be watched, but how to accurately specify the object to be watched is also a problem.

  The present invention has been made in view of the circumstances as described above, and the purpose thereof is a side-view state determination device that can determine a side-of-sight direction look by a simple method while accurately identifying an object to be watched. Is to provide.

In order to achieve the above object, the following solution is adopted in the present invention. That is, as described in claim 1,
Imaging means for imaging a landscape in the direction of the line of sight of the person to be judged by the side,
Visual feature point detection means for detecting a visual feature point that is a visual recognition object to be watched by a determination target person from a captured image acquired by the imaging means;
Gazing point detection means for detecting the gazing point of the determination target person;
Based on the visual feature point detected by the visual feature point detection unit and the gazing point detected by the gazing point detection unit, it is determined whether or not the determination target person is a side view that does not see the visual feature point. A side look determination means;
It is supposed to be equipped with. The target object to be watched by the determination target person can be accurately specified by the visual feature point detection means. And it can be determined whether it is a gaze direction side look by the simple method of being based on the specified visual feature point and the gaze point of a judgment subject.

A preferred mode based on the above solution is as described in claim 2 and the following. That is,
The side-by-side determination unit determines that it is side-by-side when the line-of-sight movement frequency of the gazing point to the visual feature point is a predetermined value or less (corresponding to claim 2). In this case, when visually recognizing a visual feature point without looking aside, the frequency of line-of-sight movement with respect to the visual feature point increases, so it is accurately determined whether or not it is a side-by-side based on the frequency of line-of-sight movement. can do.

  The visual feature point detection means detects visual feature points based on at least luminance, color, edge direction, and movement in the captured image (corresponding to claim 3). In this case, it is necessary to pay attention to things that are easy for humans to recognize intuitively, such as brightness, color, edge direction, and movement, as visual objects that should not be overlooked. This is preferable for more accurate detection.

From the captured image, comprising an area excluding means for excluding an area that does not need to be visually recognized,
The visual feature point detection means detects a visual feature point based on a captured image after the area is excluded by the area exclusion means;
(Corresponding to claim 4). In this case, it is possible to prevent a situation in which a visual feature point or the like in a direction unrelated to the visual recognition direction of the determination target person is unnecessarily determined as a determination target, and to erroneously determine that it is a control system load reduction or side effect. This is also preferable in preventing such a situation.

  The visual feature point detecting means detects a visual feature point limited to a dangerous object (corresponding to claim 5). In this case, it is preferable to accurately limit the visual target object to be a side determination target and further enhance the effect corresponding to the fourth aspect.

The determination target person is a person who drives or steers the moving body,
The side-view determination unit performs side-by-side determination on the condition that the line-of-sight direction of the determination target person substantially matches the moving direction of the moving body.
(Corresponding to claim 6). In this case, it is preferable to accurately determine the traveling direction side effect by preventing a situation in which an object or the like in a direction greatly deviating from the traveling direction of the moving body is specified as a visual recognition target for the side performance determination. .

  An alarm unit is provided to perform an alarm when the determination target person determines that the determination target person is looking aside (corresponding to claim 7). In this case, it is preferable in returning from the side view state to the normal state in which the object to be viewed is properly viewed.

  According to the present invention, it is preferable to improve the safe driving by simply determining that the visual feature point to be watched is accurately determined while looking at the line of sight.

The figure which shows the example of a control system of this invention. The figure which shows the example of the visual feature point which exists in front of a vehicle. The figure which shows the example which sets an exclusion area | region from the captured image which imaged the front scenery shown in FIG. The flowchart which shows the example of control of this invention.

Hereinafter, embodiments of the present invention will be described. In the embodiment, the determination target person is a driver of a vehicle (automobile in the embodiment) as a moving body. First, FIG. 1 is a block diagram showing an example of a control system according to the present invention, which is mounted on a vehicle (own vehicle) as a moving body. ing. In the figure, U is a controller (control unit) configured using a microcomputer. Signals from various sensors and devices 1 to 4 are input to the controller U. That is, reference numeral 1 denotes a camera that images the front of the host vehicle, and the camera 1 acquires a surrounding situation in front of the host vehicle. In addition, the distance and color to a visual recognition target object can also be detected by making the camera 1 into a color-type stereo camera. In addition, a radar can also be used separately for detecting the distance to the visually recognized object.

  In FIG. 1, reference numeral 2 denotes an eye camera mounted in the passenger compartment, and particularly detects the driver's line-of-sight direction (gaze direction). 3 is a vehicle state detection means using various sensors, for example, vehicle speed, acceleration, yaw rate, engine speed, rudder angle, wiper operating status (for detecting rainy weather, snowfall, etc.), light operating status (daytime, Night detection) and the like are detected. Reference numeral 4 denotes a navigation device, which is information on the current road conditions and the road conditions ahead (for example, the presence of highways, general roads, straight roads, curved roads, intersections, various road signs, traffic signals, etc.) In addition to the vehicle position information and the like.

  As will be described later, the controller U detects a visual feature point indicating an object to be visually recognized, and a traveling direction based on a comparison result between the detected visual feature point and the driver's line of sight (gaze point). Side-by-side determination for determining whether or not side-by-side is being performed is performed. Then, when it is determined that the traveling direction is a side view, the projection mapping device 11, the head-up display 12, or the speaker 13 is controlled to alert the driver (warning).

  The controller U has a model system of the saliency model 21, as will be described later, in order to determine (extract) visual feature points, that is, visual objects to be visually recognized by the driver. The saliency model 21 is for determining what is visually recognized (conspicuous) at the moment of viewing as an object to be viewed, for example, large, bright (high brightness), and high contrast ( Color), edge direction different from the surroundings, moving objects, and the like. The saliency model 21 stores a large amount of information on particularly dangerous visual objects (for example, forward vehicles, oncoming vehicles, traffic lights, pedestrians, etc.) among many visual objects existing in front of the vehicle. Yes.

  The controller U has a database 22, and the database 22 stores the frequency of line-of-sight movement (number of line-of-sight movements per unit time) with respect to the visual target object when the visual target object is reliably watched. doing. That is, in the database 22, when the object to be viewed is, for example, a traffic light at an intersection, if it is in a gaze state (if it is in a state of looking at a traffic light without looking aside), how many times per unit time for this traffic light Information regarding whether or not to move the line of sight is stored in association with each other. The database 22 corresponds to the number of line-of-sight movements as well as the traffic conditions, the road conditions, the weather, and the like for many visual objects such as traffic lights, pedestrians, vehicles ahead, and oncoming vehicles. It is remembered.

  Next, based on FIG. 2, the detection method of the visual feature point used as a visual recognition target object is demonstrated. In FIG. 2, reference numeral 31 denotes a front window glass. An upper edge portion (upper frame portion) of the front window glass 31 is defined by a front edge portion of the roof panel 32, and a lower edge of the front window glass is defined by a front edge portion of the instrument panel 33. Edge portions (lower frame portions) are defined, and left and right edge portions (left and right frame portions) of the front window glass 31 are defined by the left and right front pillars 34R and 34L. 35R and 35L are left and right side mirrors, 36 is a rearview mirror, 37 is a meter panel, and 38 is a steering handle (in FIG. 2, it is a left-hand drive vehicle).

  In FIG. 2, as a road condition ahead that is visually recognized by the driver through the front windshield 31, a traveling lane in which the host vehicle is traveling is denoted by reference numeral 41, an opposite lane is denoted by reference numeral 42, and a central boundary thereof A line is indicated at 43. A forward vehicle 44 is present in the lane 41, a traffic light 45 is present in front of the forward vehicle 44, and a lighting signboard 46 is present along the shoulder of the opposite lane 42.

  In FIG. 2, in the saliency model 21, an object that is conspicuous at the moment of viewing is determined as an object to be viewed, so that a traffic light 45 and a shining signboard 46 are included in addition to the forward vehicle 44 that is traveling immediately ahead. It is determined as a visual recognition object. In particular, when the brake lamp of the preceding vehicle 44 is lit, the preceding vehicle 44 whose brake lamp is lit is regarded as a dangerous object. Is done.

  In the captured image captured by the camera 1, at least the front vehicle 44, the traffic light 45, and the sign 46 are displayed as visual feature points indicating the visual recognition object. In reality, there are many visual feature points that are not subject to side-viewing among all the visual feature points included in the captured image. For example, the signboard 46 at a position greatly deviated to the left and right with respect to the traveling direction of the vehicle is not subject to sidewalk determination, and the upper left area of the front window glass 31 is also not subject to sidewalk determination. Furthermore, the sky scenery in the vicinity of the upper edge of the front window glass 31 is also a non-problem area that is not subject to determination. For this reason, in order to extract (narrow down) the visual feature points that are the targets of sidewalk determination, for example, as shown in FIG. 3, processing is performed so that the regions R1 and R2 in the captured image G are excluded from the sidewalk determination regions. Is done.

  The controller U, as a determination execution condition for determining a side-by-side traveling direction, is when the driver's line-of-sight direction detected by the eye camera 2 is substantially coincident with the vehicle traveling direction, that is, the line-of-sight direction relative to the vehicle traveling direction. This is when the offset amount is equal to or less than a predetermined amount (predetermined angle).

  The controller U determines the driver's line-of-sight direction (that is, the gazing point) with respect to each of the front vehicle 44 and the traffic light 45 that are visual feature points detected from the captured image G as shown in FIG. When the movement frequency of the line of sight is equal to or less than a predetermined value, it is determined that the vehicle is in a side-viewing state (a traveling direction side-viewing state) in which the preceding vehicle 44 or the traffic light 45 is not viewed.

  Next, details of the control contents of the controller U will be described with reference to the flowchart shown in FIG. In the following description, Q indicates a step.

  First, in Q1, nest states such as the vehicle speed, acceleration, and yaw rate of the vehicle are read. Thereafter, in Q2, a captured image showing the front situation (frontal landscape) captured by the camera 1 is acquired. Thereafter, in Q3, preprocessing for setting an area to be excluded from the captured image for extracting visual feature points is performed according to the vehicle state. In other words, for example, when the vehicle speed is high or acceleration, the left and right regions in the captured image are set as excluded regions, and when the yaw rate is occurring, the region on the side opposite to the direction of the curve is the curve to be excluded. Excluded. In addition, such setting of the exclusion area is an example. In addition, the exclusion area can be appropriately set according to the vehicle state and the driver state (particularly, the fatigue state).

  In Q4, a captured image after the exclusion region set in Q3 is excluded (corrected) is acquired. Thereafter, in Q5, the saliency indicating the visual feature point (region) is calculated (extracted) from the captured image excluding the excluded region (for example, the captured image excluding the regions R1 and R2 as shown in FIG. 3) ( In the case of FIG. 3, the forward vehicle 44 and the traffic light 45 are extracted).

  After Q5, in Q6, a dangerous area is set among the visual feature points extracted in Q5. That is, as the visual feature point, for example, the forward vehicle 44 and the traffic light 45 are extracted as shown in FIG. 3, for example, the forward vehicle 44 does not move peculiarly, and the inter-vehicle distance with respect to the forward vehicle 44 If there is enough, the forward vehicle 44 does not need to be a particularly dangerous object. On the other hand, when the inter-vehicle distance from the preceding vehicle 44 is too small, or when the brake lamp is turned on, the existence region of the preceding vehicle 44 is set as a dangerous region. On the other hand, when the traffic light 45 is yellow or red, the traffic light 45 is regarded as a dangerous object (when it is blue, it is not regarded as a dangerous object). As described above, when there are a plurality of visual feature points, dangerous objects to be particularly noted are narrowed down.

  After Q6, the driver's gaze behavior is detected by the eye camera 2 at Q7. Thereafter, in Q8, the frequency of the driver's line-of-sight movement performed on the dangerous object (for example, the traffic light 45) narrowed down in Q6 is calculated.

  After Q8, in Q9, the line-of-sight movement frequency acquired in Q8 is compared with the reference line-of-sight frequency stored in the database 22. The reference line-of-sight movement frequency corresponds to, for example, the current situation of the vehicle and the display state of the traffic light among the frequency of line-of-sight movement with respect to the traffic light stored in the database 22 when the dangerous object is the traffic light 45, for example. The frequency of gaze movement that has been selected is selected (extracted). The comparison in Q9 is a comparison as to whether or not the frequency of eye movement acquired in Q8 is equal to or higher than the reference eye movement frequency extracted from the database 22.

  Thereafter, in Q10, the attention state is determined. In this determination at Q10, when the comparison result at Q9 shows that the frequency of the line-of-sight movement acquired at Q8 is equal to or higher than the reference line-of-sight movement frequency extracted from the database 22, the traffic light 45 is watched closely. It is said that it is a normal state that is not a decline in attention. Conversely, if the frequency of the line-of-sight movement acquired in Q8 is less than the reference line-of-sight movement frequency extracted from the database 22, the signal 45 is overlooked or the display content of the signal 45 is fully recognized. It is said that it is in a state of reduced attention (a side-viewing state).

  In Q11, based on the comparison result in Q10, it is determined whether or not the state is a state of reduced attention (a state of looking aside in the traveling direction). If the determination in Q11 is NO, the process returns as it is. When the determination in Q11 is YES, in Q12, after a driver alerting method is calculated, in Q13, alerting is executed by the alerting method calculated in Q12.

  The processes of Q12 and Q13 are processes for changing the content to be noted in accordance with, for example, the situation of the vehicle, the road situation, etc., in a state where it is determined that the traveling direction is a sidewalk. For example, when viewing the traffic light 45 when it is displayed in yellow or red, it is dangerous, so a loudspeaker alerts (severe alert) is given by the speaker 13 (for example, “the traffic light ahead is red”). Like voice attention). Further, when the distance to the preceding vehicle 44 is small and when looking at the preceding vehicle 44, for example, the head-up display 12 is displayed by the projection mapping 11, for example, “Let's secure more distance between vehicles”. An alert is issued (mild alert). The contents of the alert (warning) can be selected in various ways. For example, the projection mapping device 11 performs lighting display (or blinking display) at a position corresponding to the position of the visual recognition object (particularly the dangerous object), and pays attention. It is also possible to notify the driver of the position to be performed.

  Although the embodiments have been described above, the present invention is not limited to the embodiments, and appropriate modifications can be made within the scope of the claims. The person to be judged is not limited to the driver of the vehicle, but can be appropriately selected as long as it is a problem such as a ship or aircraft operator, an operator such as a crane, etc. Can be a person to be determined, and is not limited to a driver or a driver of a moving body, but may be a person who performs monitoring at a fixed position, for example. It can also be determined for general sidewalks where the line-of-sight direction deviates by more than a predetermined amount from the moving direction of the moving body. In this case, the content of the sidewalk alert (alarm) to the driver And a side view in the viewing direction (advancing direction in the case of a moving body). The steps (or steps) shown in FIG. 4 can be expressed as functions (or means) of the controller U. Of course, the object of the present invention is not limited to what is explicitly stated, but also implicitly includes providing what is substantially preferred or expressed as an advantage.

  The present invention is preferable in improving the safe driving by accurately determining the visual feature point to be watched while easily determining, for example, that the driver of the vehicle travels in the traveling direction.

U: Controller 1: Camera (front view)
2: Eye camera 3: Sensor (vehicle state detection)
4: Navigation device 11: Projection mapping (for alerting)
12: Head-up display (for alerting)
13: Speaker (for alerting)

Claims (7)

Imaging means for imaging a landscape in the direction of the line of sight of the person to be judged by the side,
Visual feature point detection means for detecting a visual feature point that is a visual recognition object to be watched by a determination target person from a captured image acquired by the imaging means;
Gazing point detection means for detecting the gazing point of the determination target person;
Based on the visual feature point detected by the visual feature point detection unit and the gazing point detected by the gazing point detection unit, it is determined whether or not the determination target person is a side view that does not see the visual feature point. A side look determination means;
A side-view state determination device characterized by comprising: In claim 1,
The side-view state determination device, wherein the side-view determination unit determines that it is a side view when a gaze movement frequency of the gazing point to the visual feature point is equal to or less than a predetermined value set in advance. In claim 1 or claim 2,
The visual feature point detection unit detects a visual feature point based on at least luminance, color, edge direction, and motion in the captured image. In any one of Claims 1 thru | or 3,
From the captured image, comprising an area excluding means for excluding an area that does not need to be visually recognized,
The visual feature point detection means detects a visual feature point based on a captured image after the area is excluded by the area exclusion means;
A side-view state determination device characterized by that. In any one of Claims 1 thru | or 4,
The side-view state determination device characterized in that the visual feature point detection means detects visual feature points limited to dangerous objects. In any one of Claims 1 thru | or 5,
The determination target person is a person who drives or steers the moving body,
The side-view state determination device characterized in that the side-view determination unit performs a side-view determination on the condition that a line-of-sight direction of the determination target person substantially matches a moving direction of the moving body. In any one of Claims 1 thru | or 6,
A side-view state determination apparatus, comprising: a warning unit that issues a warning when the determination target person determines that the determination target person is watching a side-by-side.

Images