JP2006268189A - Visual inspection action determining device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To positively distinguish whether change of eye coordinates obtained by a face image of a subject is due to a visual inspection action of the subject, or it is due to an action of changing a direction of a face of the subject. <P>SOLUTION: A visual inspection action determining device 12 is provided with a face image acquiring part 20 acquiring the face image of the subject, an eye position detecting part 21, and an eye position tracking part 22 detecting the eye coordinates indicating an eye position of the subject, and carrying out tracking of the eye coordinates on the basis of the face image, and an opened and closed eye learning part 23, and an opened and closed eye determining part 24 calculating an eye opening of the subject on the basis of the face image and the eye coordinates, and determining an opened/closed state of the eyes of the subject from the eye opening. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a visual behavior determination device that determines visual behavior of a subject such as a driver driving a vehicle.

  Conventionally, driver monitoring devices such as an armpit detection device, a snooze detection device, a casual driving detection device, and a safety confirmation detection device are known as techniques for photographing a driver's face and determining the state of the driver.

  In this armpit detection device, in Patent Document 1, a driver's face image is captured as a moving image, eye coordinates are sequentially detected from successive face images by image processing, and the armpit look is determined from the movement of the eye coordinates. is doing.

In addition, as another technique for detecting the side-view driving, the technique for detecting the driver's line-of-sight direction described in Patent Document 2 and the reflection of the pupil of the eye described in Patent Document 3 are used. Using the technology for detecting the coordinates of the vehicle and the technology for estimating the direction of the driver's face based on the difference image described in Patent Document 4, the direction that the driver should visually recognize is compared with the actual viewing direction. It is proposed to determine whether the user is looking in the direction to be viewed.
JP-A-6-262959 JP 2002-83400 A JP-A-7-61257 JP 2000-113164 A

  The gaze direction detection process described in Patent Document 1 described above separates the visual behavior when facing the front and the visual behavior that is looking aside by estimating the visual direction from the eye coordinates. Judgment. However, the driver's head consists of three types: a roll motion that is a rotational motion, a pitch motion that is a vertical motion, and a yaw motion that is a left-right motion. When combined with three types of head offset movements by vertical movement, the movement becomes complicated.

  For this reason, if the driver performs a yaw movement of the face to look sideways and a roll movement of the face caused by vibration in the passenger compartment is detected only by a change in the coordinate position of the eye, The behavior due to the roll and the behavior due to the yaw of the face are detected as the same movement amount. Therefore, conventionally, there has been a problem that it is difficult to distinguish whether the driver has changed the eye coordinates when looking sideways or whether the eye coordinates have changed due to vibration.

  On the other hand, not only eye coordinates but also other parts of the face are tracked together with nose coordinates and face contour coordinates, and the distance ratio between face parts is used, the yaw motion and roll motion However, in this case, not only the eye coordinates but also the coordinates of other parts of the face must be tracked, resulting in a problem that the calculation load increases. Furthermore, although the face direction can be estimated, there is a problem that it cannot be determined whether or not the driver has changed the direction of the line of sight.

  Therefore, the present invention has been proposed in view of the above-described circumstances, and whether the change in eye coordinates obtained from the face image of the subject is due to the visual behavior of the subject or the behavior of changing the orientation of the subject's face. It is an object of the present invention to provide a visual action determination device capable of reliably distinguishing whether or not.

  A visual recognition behavior determination apparatus according to the present invention detects a face image acquisition unit that acquires a face image of a subject included in a video signal output from an imaging unit, and detects eye coordinates that indicate the eye position of the subject based on the face image In addition, the eye coordinate acquisition means for tracking the eye coordinates, the eye opening of the subject is calculated based on the face image and the eye coordinates, and the opening / closing eye for determining the opening / closing state of the subject's eye from the eye opening In order to solve the above-mentioned problem, a time-series change of the eye coordinates tracked by the eye coordinate acquisition means and a time-series change of the eye open / closed state determined by the open / close eye determination means The visual behavior determination means for determining whether the time-series change of the eye coordinates is due to the visual behavior of the subject, and the visual recognition result output means for outputting the determination result of the visual behavior determination means as a signal. Prepare.

  According to the visual recognition behavior determination device according to the present invention, whether or not the time-series change of the eye coordinates is due to the visual behavior of the subject based on the time-series changes of the eye coordinates and the time-series changes of the eye open / closed state. Therefore, in addition to the change in the position of the eye coordinate of the subject, the visual behavior is determined by detecting the physiological phenomenon of blinking in order to correct the focus of the eye when the subject reciprocates the eye direction. It is possible to reliably avoid the case where the face orientation is changed, and to determine that it is a visual action, and whether the change in eye coordinates obtained from the face image of the subject is due to the visual behavior of the subject. Therefore, it is possible to reliably distinguish whether it is due to an action that changes the orientation of the face of the subject.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
The present invention is applied to the visual recognition behavior determination device 12 of the visual behavior determination system according to the first embodiment configured as shown in FIG. 1, for example. In the present embodiment, it is assumed that the visual recognition behavior determination system is mounted on a vehicle for the purpose of supporting the driving operation of a driver (subject) driving the vehicle. The visual behavior determination device in this visual behavior determination system is a method in which a person (driver) performs a visual behavior, for example, when returning the line-of-sight direction like an action of returning the front of the vehicle to the visual recognition direction from a state of looking at the side of the vehicle. By detecting the physiological phenomenon of blinking in order to correct the focus of the eye, the time series change of the eye coordinates as the eye state and the time series change of the open / close state are detected, and the driver's It is distinguished whether the action is “visual action” or “action in which the face direction is changed instead of the visual action”.

[Configuration of visual recognition judgment system]
As shown in FIG. 1, the visual behavior determination system determines a video camera 10 as an imaging unit that captures a driver's face image, an illumination device 11 that illuminates the periphery of the driver's face, and a driver's visual behavior. And a visual recognition behavior determination device 12 that performs the visual recognition behavior determination.

  The video camera 10 has a light receiving element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor, a driver circuit that controls the light receiving element, a mirror head on which an optical filter, a lens, and the like are mounted. The captured image (driver's face image) is output as a video signal to the visual recognition action determination device 12.

  The illuminating device 11 includes a light source such as an LED (Light Emitting Diode) or a halogen lamp, a driver circuit that controls the light source, an optical element such as an optical filter, and a lens, and applies near infrared light to the driver's face. Irradiate and illuminate.

  Next, the visual recognition behavior determination device 12 will be described with reference to FIG. As shown in FIG. 2, the visual recognition behavior determination device 12 according to the first embodiment includes a face image acquisition unit 20, an eye position detection unit 21, an eye position tracking unit 22, an open / close eye learning unit 23, and an open / close eye. It has the determination part 24, the visual recognition action determination part 25, and the visual recognition result output part 26.

  The face image acquisition unit 20 receives the video signal output from the video camera 10 and uses one or a plurality of image frames in a series of image frames included in the video signal, thereby using the face image of the driver. Extract and get. At this time, the face image acquisition unit 20 performs digital processing on the video signal input from the video camera 10, captures the face image as a digital image, and stores the digital image in the video memory. Here, the position in the image at which the driver's face is imaged is determined from the mounting position and angle of view of the video camera 10, and the face image acquisition unit 20 captures a digital image of the position in the image. Become.

  The eye coordinate output unit 21 acquires, for example, the face image captured by the face image acquisition unit 20 every predetermined period, detects eye coordinates indicating the driver's eye position based on the face image, and detects the eye position. Output to the tracking unit 22.

  The eye position tracking unit 22 tracks eye coordinates by comparing a series of facial images that are temporally changed (comparison between a plurality of image frames). Then, the eye position tracking unit 22 outputs the current eye coordinates as the tracking result to the open / close eye learning unit 23, the open / close eye determination unit 24, and the visual recognition behavior determination unit 25.

  Based on the face image acquired by the face image acquisition unit 20 and the current eye coordinates that are the tracking result of the eye coordinates detected by the eye coordinate output unit 21, the open / close eye learning unit 23 Learn opening and closing states. At this time, the open / close eye learning unit 23 acquires the eye opening from the eye coordinates, obtains a learning result, and outputs the learning result to the open / close eye determination unit 24.

  The open / close eye determination unit 24 determines the open / closed state based on the driver's eye opening, and outputs the eye opening to the visual action determination unit 25.

  The visual recognition behavior determination unit 25 determines the visual behavior of the driver from the eye opening degree from the open / close eye determination unit 24 and the current eye coordinates that are the tracking result from the eye position tracking unit 22. Specifically, the visual recognition behavior determination unit 25 determines whether the driver's behavior is visual behavior based on a change in the eye opening and a time-series change in the current eye coordinates, a change in face position and a change in face direction. It is determined whether the behavior is other than visual behavior such as.

  The visual recognition result output unit 26 outputs the result determined by the visual recognition behavior determination unit 25 as a signal.

"Eye position detection process"
Next, the eye position detection process in the eye position detection unit 21 will be described.

  When the eye position detection process is started, the eye position detection unit 21 determines whether the current detection mode is a one-side candidate detection process or a two-sided determination process as shown in FIG. The detected detection mode is confirmed (step S10).

  Then, the eye position detection unit 21 detects a candidate for one eye from the entire screen of the face image according to the detection mode confirmed in step S1 (step S11), the coordinates of the one eye candidate The process proceeds to one of the binocular confirmation processing (step S12) for detecting both eyes from the local region and confirming both eyes. Further, when the process in step S11 or step S12 is completed, the eye position detection unit 21 ends the eye position detection process.

  When the one-eye candidate detection process is started in step S2, the eye position detection unit 21 sets a processing area for detecting one-eye candidates (step S20), as shown in the flowchart of FIG. Whether to perform continuous data detection of density change points existing in the processing area (step S21), narrowing down one eye candidate (step S22), and detecting one eye candidate in the next image frame Then, the process of determining whether to perform the binocular determination process (step S23) is sequentially performed.

  Here, the processing area set in step S20 is set as a right half region or a left half region of the face image as shown in FIG. Note that FIG. 5 shows an example when the face image is set to the right half region. Then, in the vicinity of the eyes in the right half or left half of the face image, for example, as shown in FIG. 6, continuous data (group A) that is a density change point group adjacent in the horizontal direction indicating the outline of the eye, Continuous data (group B) that is a density change point group that is not an eye such as an eyebrow is detected.

  In step S21, the eye position detection unit 21 detects a change in the light amount density for each vertical line (FIG. 7A) in the processing area set in step S20 (FIG. 7B). By detecting a density change point group in which the detected density change points changing from white (light) to black (dark) in the horizontal direction are detected, the density change point group is detected as continuous data. (FIG. 7 (c)). In FIG. 7A, only one of the plurality of vertical lines set in the processing area is shown, and the change in the light amount density for the single vertical line is shown in FIG. Shown in FIG. 7C shows a state in which both end points are arranged for a plurality of vertical lines while paying attention to the end points detected as “eye” in the change in the light intensity density shown in FIG. 7B. In FIG. 7C, both end points detected as “eye” are indicated by “◯” and “x”, respectively.

  In the process of narrowing down one eye candidate in step S22, if the continuous data detected in step S21 by the eye position detection unit 21 is an upper eyelid line, it is assumed that it has an upward convex shape. Based on the shape recognition, only the density change points that match the convex shape are left as an image showing the eye.

  In step S23, when the one-eye candidate coordinates are not detected by the eye position detection unit 21, it is determined that the one-eye candidate detection process shown in FIG. 4 is performed again in the next image frame, or one eye If the candidate coordinates are detected, it is determined that the binocular confirmation process in step S12 is performed in the next image frame.

  If it is determined that the binocular confirmation process is to be performed in step S12, the binocular confirmation process is performed with the density change point detected as continuous data having a continuously convex shape in the vicinity of the same coordinate and being detected as continuous data. Transition. In this step S12, as shown in FIG. 8, the process of setting a processing area for detecting both eyes at the coordinates of one eye candidate (step S30) and the continuous data of density change points in the processing area are detected. Processing (step S31), processing for determining both eyes by pattern recognition by comparing continuous data detected as a binocular candidate and a template prepared in advance (step S32), and one eye candidate in the next image frame The process of determining whether to return to the detection process, to continue the binocular determination process, or to perform the eye position tracking process (step S33) is sequentially performed.

  Here, as the processing area set in step S30, as shown in FIG. 9, an area including both eyes of the driver is set in the face image. In FIG. 9, a region for detecting the nose is also set, and the density change point detected from the nose is also shown in the region.

  The pattern recognition processing employed in step S32 prepares in advance image data that generates a template eye image from a plurality of eye images, and the template and the continuous data detected in step S31. It is possible to use a method using a cross-correlation method for obtaining the correlation. Alternatively, in this step S32, a plurality of eye images and non-eye face part images (glass frame, eyebrows, nose, etc.) are learned in advance by a neural network, and an image for determining whether or not the eye is present is used as the neural network. A method for determining more can be used.

"Eye position tracking process"
Next, the eye position tracking process in the eye position tracking unit 22 will be described with reference to the flowchart shown in FIG.

  When the eye position tracking process is started, the eye position tracking unit 22 uses a reference area from the vicinity of the eye coordinates detected by the previous eye position detection process or the binocular determination process as shown in FIG. Eyes are detected (step S40). Here, the detection position of the reference eye in the above-described eye position detection process refers to “eye (FIG. 7C)” detected by the eye position detection unit 21, as shown in FIG. The upper end, the lower end, the left end, and the right end are set, and the center position is detected as the reference eye position.

  Next, the eye position tracking unit 22 determines whether or not the reference eye has been detected (step S41). If the eye has been detected, as shown in FIG. 11B, the detected current reference eye is detected. The interocular distance from the coordinates to the coordinates of the eyes detected by the previous eye position detection process or the binocular confirmation process is obtained, a detection area for the opposite eye is set, and the opposite eye is detected from the detection area (step) S42).

  In step S41, if it is determined that the reference eye cannot be detected, the eye position tracking unit 22 changes the opposite eye to the reference eye (step S43) and detects the reference eye again (step S44). Next, the eye position tracking unit 22 determines whether or not the opposite eye can be detected as the reference eye (step S45). If it is determined that the eye cannot be detected, it is considered that tracking in the image frame has failed. (Step S46), the eye position tracking process is terminated. In step S45, when it is determined that the eye position tracking unit 22 has detected the opposite eye as the reference eye, the process proceeds to step S42.

  After detecting the opposite eye in step S42, the eye position tracking unit 22 determines whether or not the opposite eye has been detected (step S47), and if detected, the coordinates of the detected reference eye and the opposite eye are detected. Is determined (step S48), and the eye position tracking process is terminated. When it is determined in step S47 that the opposite eye cannot be detected, the eye position tracking unit 22 obtains the interocular distance from the previous eye coordinates and moves the interocular distance corresponding to the reference eye coordinates. The position is determined as the coordinates of the opposite eye (step S49), and the eye position detection process is terminated.

"Open / close eye judgment process"
Next, the opening / closing eye learning process by the opening / closing eye learning unit 23 and the opening / closing eye determination (determination) process by the opening / closing eye determination unit 24 will be described with reference to the flowchart shown in FIG.

  When the opening / closing eye learning process and the opening / closing eye determination process are started, the opening / closing eye learning unit 23 and the opening / closing eye determination unit 24 start from the image region around the eye coordinates obtained by the eye position tracking process by the eye position tracking unit 22. By determining the binarization threshold based on the gray value of the image and cutting out the eye area from the face image, the “left end”, “right end”, “upper end”, and “lower end” of the eye are acquired, and the open / close eye determination parameters are set. (Step S50). Here, the coordinates of the “left end”, “right end”, “upper end”, and “lower end” of the eye that are set as the open / closed eye determination parameters should be set so that the relative positional relationship differs between when the eye is opened and when the eye is closed. become.

  Next, the open / close eye learning unit 23 and the open / close eye determination unit 24 obtain the distance (vertical width of the eye) between the “upper end” and the “lower end” of the eye as the eye opening (step S51). Here, from the open / closed eye determination parameters set in step S50, the relative positional relationship between the “upper end” and “lower end” of the eye becomes closer as the open / closed state of the eye approaches the closed eye.

  Next, the open / close eye determination unit 24 coordinates the “upper end” with respect to the line segment connecting the “right end” and the “left end” based on the shape of the upper eyelid by the “right end”, “left end”, and “upper end” of the eye. Is determined to be above or below. Then, the open / close eye determination unit 24 determines whether the eye shape is upward, downward (concave), or horizontal based on the determination result (step S52).

  Next, the open / close eye determination unit 24 determines whether or not the eye open / close learning has been completed (step S53), and when the open / close eye learning unit 23 has not ended, the open / close eye learning is performed (step S53). Step S54). On the other hand, when the eye opening / closing learning has been completed, the detected current eye is detected based on the eye height threshold obtained from the eye shape and the eye learning result and the detected current eye height. When the vertical width is equal to or smaller than the vertical threshold of the eye, it is determined that the eye is closed and when the vertical width is equal to or larger than the vertical threshold (step S55). Specifically, as shown in FIG. 13, when the current eye vertical width (opening opening value) is equal to or greater than a predetermined threshold, the open / close eye determination unit 24 opens the driver's eyes. It is determined that the driver's eyes are in a closed state when the current eye height drops below a threshold value.

  The open / close eye learning unit 23 and the open / close eye determination unit 24 end the series of processes after the process of step S54 or step S55.

  The eye opening / closing learning process by the opening / closing eye learning unit 23 in step S54 is performed as follows. That is, the opening degree value when the eye shape is upwardly convex from the vertical width of the eye and the eye shape obtained by the series of processes shown in the flowchart of FIG. The counter memory array prepared for each value is counted, and the total value of the left and right counter memories for each opening value is equal to or greater than a preset setting value (in this embodiment, “20”). The opening value is learned as an eye opening value. Further, a predetermined ratio (in this embodiment, “30%”) set in advance from the eye opening degree value is set as a threshold value for distinguishing between opening and closing as described with reference to FIG.

"Visibility action determination process"
Next, processing in the visual recognition behavior determination unit 25 will be described.

  The visual behavior determination unit 25 inputs eye coordinates obtained by eye position tracking from the eye position tracking unit 22 to determine the visual behavior, and opens and closes the eye open / close state obtained by the open / close eye determination. Input from 24. Then, the visual recognition behavior determination unit 25 determines whether the driver's behavior is visual behavior or behavior other than visual behavior such as face position variation and face orientation variation, based on a time-series change in both detection results. Determine.

  FIG. 15 shows in time series the left and right eye coordinates (X-axis direction and Y-axis direction) obtained by the eye position tracking process and the eye open / closed state obtained by the open / closed eye determination process. As shown in FIG. 15 (a), a change in eye coordinates on the X axis, that is, a change in gaze direction, is detected at frame numbers 1300 to 1400. → Detects blinking action that occurs in a short period of time when the eye opening changes. FIG. 15B shows a change in eye coordinates in the direction opposite to that in FIG. 15A. Even in this case, the change in eye coordinates on the X axis, that is, the change in the line-of-sight direction is frame numbers 700 to 800. The eyeblink action in which the change from eye-opening to eye-closing to eye-opening is performed in a short period at the start and end of the line-of-sight direction change is detected. From such a result, when the driver (subject) is visually recognizing when the eye coordinates are changed, when the eye coordinates are moved from a predetermined eye coordinate to another eye coordinate, and when the eye coordinate is returned from the other eye coordinate to the predetermined eye coordinate. It turns out that the physiological phenomenon accompanying a blink action is detected.

  On the other hand, the visual recognition behavior determination unit 25 detects the change of the eye coordinate in time series by inputting the movement of the eye coordinate from the eye position tracking unit 22 every predetermined period, and the movement of the eye coordinate is constant. When the eye coordinate movement is less than a certain distance after that, that is, when the variance value of the eye coordinates in a certain time interval is greater than or equal to the set value, and the variance value of the eye coordinates in the next certain time interval Detects a section where is less than the set value. Further, the visual recognition behavior determination unit 25 opens / closes from the open / closed eye determination unit 24 in the vicinity between a section where the variance value of the eye coordinates is greater than or equal to a set value and a section where the variance value of the eye coordinates is less than the set value. Enter the state determination result, and determine whether or not the blink action has been performed in a short period of time when the change from opening to closing to closing to opening has been performed. It can be determined that an action with a variance value greater than or equal to a set value is a visual action, and if there is no blink action, it can be determined that an action with an eye coordinate variance value greater than or equal to a set value is not a visual action .

  Therefore, when the eye movement is detected when the eye coordinate moves from the predetermined eye coordinate to the other eye coordinate and when the eye coordinate moves from the other eye coordinate to the predetermined eye coordinate, It can be determined that the visual recognition action has been performed, and when the blink action is not detected, it can be determined that the visual recognition action has not been performed.

  Alternatively, the visual recognition behavior determination unit 25 obtains an average value of the variance value of the eye coordinates between the blink actions, and when the variance value of the eye coordinates between the blink actions is equal to or larger than the set value, If it is determined that there is a visual action between eye actions, and the variance of eye coordinates between blink actions is not equal to or greater than a set value, it may be determined that there is no visual action between the blink actions. .

  Furthermore, in order to determine the presence / absence of the visual behavior, the visual behavior determination unit 25 described the possibility of the visual behavior based on the relationship between the movement amount of the eye coordinates and the presence / absence of the blink action as shown in FIG. Table data may be used. According to FIG. 16, (1) the eye coordinate movement is large and there is a high possibility that there is a visual action when the blink action is detected either before or after the eye coordinate movement. When there is little coordinate movement and no blink action is detected before or after the eye coordinate movement, it is unlikely that there was a visual action. (3) Although the eye coordinate movement is large, before and after the eye coordinate movement If the blinking action is not detected or if the eye coordinate movement is small but the blinking action is detected before and after the eye coordinate movement, there is a moderate possibility that there was a visual action. The three conditions (1) to (3) are used. And the visual recognition action determination part 25 compares such three conditions with the movement amount of an eye coordinate, and the presence or absence of a blink action, and there is a high possibility that the movement of the eye coordinate is a visual action, Medium or low can be judged.

[Effect of the first embodiment]
As described above in detail, according to the visual recognition behavior determination device to which the present invention is applied, the time-series change of the eye coordinate is determined by the visual recognition of the subject from the time-series change of the eye coordinate and the time-series change of the eye open / closed state. Since it is determined whether or not it is due to behavior, in addition to the change in the position of the driver's eye coordinates, when the driver reciprocates in the direction of the line of sight, he / she acts to blink to correct the focus of the eyes The visual behavior can be determined by detecting a physiological phenomenon, and it can be surely avoided that the visual behavior is determined when the face direction is changed.

  Further, according to the visual recognition behavior determination system, it is determined whether the movement amount of the eye coordinates is equal to or larger than the set value, and when it is determined that there is a blink action at the time of movement of the eye coordinates, It can be determined that the series change is due to the subject's visual recognition behavior, and using the physiological phenomenon that accompanies blinking action when moving the line of sight, Can be distinguished.

  Furthermore, according to the visual recognition behavior determination system, when it is determined that there is a blink action, an average value of the movement amount of the eye coordinates between the blink actions is obtained, and the eye coordinate between the blink actions is calculated. When the amount of movement is greater than or equal to the set value, it can be determined that the time-series change in the eye coordinates is due to the visual behavior of the subject, and the physiological phenomenon that accompanies blinking when moving the line of sight By using this, it is possible to reliably distinguish the visual action and the action such as changing the face direction.

  In addition, the visual recognition behavior determination system estimates the line-of-sight direction from the eye coordinates alone and performs complicated processing as in the case of separately determining the visual behavior when facing the front and the visual behavior that is looking aside. It is possible to reliably determine whether the change in the eye coordinates is due to a visual recognition action or a change in the face orientation.

  That is, according to this visual recognition behavior determination system, the driver's head consists of three types of roll motion, which is rotational motion, pitch motion, which is vertical motion, and yaw motion, which is lateral motion. Combining movement with the three types of head offset movements by moving the entire body back and forth, left and right, and up and down results in complex movements. Such complex movements are determined only by changes in eye coordinates. There is no need. Further, according to the visual recognition behavior determination system, it is not necessary to track a plurality of face parts as in the case of using a ratio of distances between face parts by tracking eye coordinates, nose coordinates and face contour coordinates together. Only by tracking the eye coordinates, it is possible to reliably determine whether the visual action or the face orientation has been changed.

  Furthermore, according to the visual recognition behavior determination system, in order to acquire the eye coordinates, a processing area for detecting one eye candidate from the right half or the left half of the entire face image is set, and the one eye candidate coordinates are determined from the processing area. When one eye candidate coordinate is detected, a region for detecting both eyes is set with reference to the one eye candidate coordinate, and both eyes are determined for the one eye candidate coordinate in the region. Since processing is performed, both eyes can be detected with a smaller amount of calculation than searching for both eyes from the entire face image, and the processing load can be reduced.

  Furthermore, according to the visual recognition behavior determination system, in order to acquire the eye coordinates, a plurality of vertical lines are set in the processing area for detecting eyes in the face image, and the light intensity density is set for each vertical line. Obtaining the differential value of the change to detect the density change point, and acquiring the center coordinates of the continuous data in which the density change points between the vertical lines continue in the horizontal direction as eye coordinates, for example, shadows due to the influence of sunlight, Eye coordinate candidates can be detected regardless of the influence of backlight.

  Furthermore, according to the visual recognition behavior determination system, when the current reference eye coordinate is detected from the region centered on the eye coordinate acquired in time, the time is determined from the reference eye coordinate. Since the opposite eye that exists in the vicinity of the coordinates that are separated by the interocular distance acquired previously is tracked, both eyes are tracked, eliminating the need to detect eye coordinates for each frame of the face image. The eye coordinates can be easily traced based on the obtained eye coordinates and the interocular distance.

  Furthermore, according to the visual recognition behavior determination system, it is determined whether the eye shape is upwardly convex from the left end coordinate, upper end coordinate, and right end coordinate of the eye, and the upper end coordinate of the eye when the eye shape is upwardly convex The distance from the lower end coordinate is recorded as an eye opening value, learning to set a value obtained by subtracting a predetermined ratio set in advance from the eye opening value as a threshold value for distinguishing between eye opening and closing, Since the open / closed state of the driver's eye is determined from the threshold and the eye opening, only the vertical width of the eye can be used for learning the eye opening value, and the shape of the eye (upper eyelid) is horizontal or downward convex It is possible to make an open / close determination with high accuracy by excluding the opening value when there is a possibility of eye closure such as (concave) from the learning data.

[Second Embodiment]
Next, a visual recognition behavior determination system according to the second embodiment will be described. In addition, about the part similar to the above-mentioned 1st Embodiment, the detailed description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  As shown in FIG. 17, the visual behavior determination system according to the second embodiment is connected to the vehicle signal acquisition unit 31 connected to the eye position tracking unit 22 and the visual behavior determination unit 25, and to the visual behavior determination unit 25. It differs from the visual recognition behavior determination system according to the first embodiment in that it includes an ambient environment acquisition unit 32. The vehicle signal acquisition unit 31 acquires the vehicle speed and the steering angle as vehicle signals and outputs them to the eye position tracking unit 22 and the visual action determination unit 25. In addition, the ambient environment acquisition unit 32 acquires the vehicle position based on the GPS signal, the traveling road environment information from the navigation system, the inter-vehicle distance from the inter-vehicle distance sensor, and the like as the vehicle ambient environment information, and the visual behavior determination unit 25 Output.

  When the eye position tracking unit 22 receives the vehicle speed signal and the steering angle signal, the eye position tracking unit 22 determines whether to suspend the eye coordinate tracking process based on the vehicle speed signal and the steering angle signal. That is, the eye position tracking unit 22 suspends the eye coordinate tracking process when the vehicle speed is equal to or lower than a certain speed, for example, when the vehicle is in a stopped state. Further, the eye position tracking unit 22 suspends the eye coordinate tracking process when the steering angle is equal to or greater than a certain value, for example, in a right / left turn operation state.

  As a result, the eye coordinate tracking process can be suspended when the driver may move the face greatly, such as when the vehicle is stopped or when the vehicle is turning right or left. Specifically, when there is no vehicle speed, or when the direction of the line of sight is difficult to determine in a parking lot or the like, a scene in which the face movement is large and the eye coordinates cannot be captured is specified, and the eye position tracking unit 22 detects an error. Detection can be prevented. When the vehicle speed becomes equal to or higher than the predetermined value and the steering angle becomes equal to or lower than the predetermined value, tracking of the eye coordinates by the eye position tracking unit 22 is started.

  In addition, when the visual behavior determination unit 25 receives the vehicle speed signal from the vehicle signal acquisition unit 31 and the inter-vehicle distance signal from the surrounding environment acquisition unit 32, the self-vehicle is moved forward by the time series change of the vehicle speed signal and the inter-vehicle distance signal. It is determined whether or not the vehicle has suddenly approached. When the visual behavior determination unit 25 determines that the host vehicle has suddenly approached the preceding vehicle, the visual behavior determination unit 25 acquires the current eye coordinate mark from the eye position tracking unit 22, and the driver uses the current eye coordinate. It is determined whether the front in front of the vehicle is visible. As a result, when the driver does not visually recognize the front side of the vehicle, the visual recognition behavior determination unit 25 outputs the result from the visual recognition result output unit 26 to an alarm device (not shown). Here, as described above, the visual action determination unit 25 allows the driver to visually recognize the front of the vehicle based on the eye coordinates that can be determined to have changed due to the visual action based on the movement amount of the eye coordinates and the presence or absence of the blink action. It will be determined whether or not.

  This makes it a major challenge to determine whether or not the vehicle is ahead when suddenly approaching the vehicle ahead. Therefore, whether or not the eye is positioned from the coordinate position of the eye without using the result of the open / closed eye determination. Can be determined and an alarm can be issued as necessary.

  Further, when an obstacle sensor that acquires obstacle information in front of the vehicle is used by the surrounding environment acquisition unit 32 and a front vehicle and a pedestrian are detected as shown in FIG. 18, according to the eye coordinates, The visual recognition behavior determination unit 25 can determine that the current visual recognition area is a forward vehicle and the non-visual recognition area is a pedestrian. Here, as described above, the visual behavior determination unit 25 allows the driver to visually recognize the pedestrian based on the eye coordinates that can be determined to have changed due to the visual behavior based on the movement amount of the eye coordinates and the presence or absence of the blinking action. It will be determined whether or not.

  Thereby, in the visual recognition behavior determination system, the current visual recognition area is a forward vehicle, and a signal indicating that the non-visual recognition area is a pedestrian is output to an alarm device (not shown) via the visual recognition result output unit 26, thereby enabling visual recognition. It can be notified that the area is a pedestrian.

  The above-described embodiment is an example of the present invention. For this reason, the present invention is not limited to the above-described embodiment, and various modifications can be made depending on the design and the like as long as the technical idea according to the present invention is not deviated from this embodiment. Of course, it is possible to change.

It is a schematic block diagram of the visual recognition action determination system containing the visual recognition action determination apparatus to which this invention is applied. It is a block diagram which shows the structure of the visual recognition action determination system which concerns on 1st Embodiment to which this invention is applied. It is a flowchart which shows an example of the eye position detection process by an eye position detection part. It is a flowchart which shows an example of a single eye candidate detection process. It is a schematic diagram which shows an example of the process area set by one eye candidate detection process. It is a figure for demonstrating acquisition of the continuous data used as the candidate of an eye coordinate. It is a schematic diagram for demonstrating a single eye candidate detection process, (a) is a figure which shows the vertical line set in a process area, (b) is the state by which the density change of the light quantity was detected in the vertical line (C) is a figure which shows the state from which the "eye" contained in the face image was detected. It is a flowchart which shows an example of a binocular confirmation process. It is a schematic diagram which shows an example of the process area set by a binocular confirmation process. It is a flowchart which shows an example of an eye position tracking process. It is a schematic diagram for explaining eye position tracking processing, (a) is a schematic diagram for explaining detection of a reference eye, (b) is a schematic diagram for explaining detection of an opposite eye, (C) is a schematic diagram for demonstrating the detection of a reference | standard eye. It is a flowchart which shows an example of the process by an opening-and-closing eye determination part. It is a schematic diagram for demonstrating determination of an open eye and a closed eye by comparison with the opening degree value of an eye, and a threshold value. It is a schematic diagram for demonstrating the learning process by an opening-and-closing eye determination part. (A) is a figure which shows the relationship between the time-sequential change of an eye coordinate, and a blink action, (b) is a time series change of the eye coordinate at the time of visually recognizing the opposite direction to (a), and a blink action. It is a figure which shows the relationship. It is a figure for demonstrating that the possibility of a visual action can be classify | categorized with the grade of high, medium, and low by the movement amount of eye coordinates, and the presence or absence of the blink action before and after eye coordinate movement. It is a block diagram which shows the structure of the visual recognition action determination system which concerns on 2nd Embodiment to which this invention is applied. It is a figure for demonstrating setting a current visual recognition area | region and a non-visual recognition area | region by a visual recognition action determination part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Video camera 11 Illuminating device 12 Visual action determination apparatus 20 Face image acquisition part 21 Eye position detection part 21 Eye coordinate output part 22 Eye position tracking part 23 Open / close eye learning part 24 Open / close eye determination part 25 Visual action determination part 26 Visual result output Section 31 Vehicle signal acquisition section 32 Ambient environment acquisition section

Claims (11)

Face image acquisition means for acquiring the face image of the subject included in the video signal output from the imaging means;
Eye coordinate acquisition means for detecting eye coordinates indicating the eye position of the subject based on the face image and tracking the eye coordinates;
Opening / closing eye determination means for calculating the eye opening of the subject based on the face image and the eye coordinates, and determining the opening / closing state of the eye of the subject from the eye opening;
From the time series change of the eye coordinates tracked by the eye coordinate acquisition means and the time series change of the eye open / closed state determined by the open / close eye determination means, the time series change of the eye coordinates indicates the visual behavior of the subject. Visual action determination means for determining whether or not it is due to being,
And a visual recognition result output unit that outputs the determination result of the visual recognition behavior determination unit as a signal.   When the eye movement determination unit determines that the movement amount of the eye coordinates acquired by the eye coordinate acquisition unit is greater than or equal to a set value, the eye movement determination unit determines whether the eye movement determination unit moves the eye coordinates greater than or equal to the set value. 2. The method according to claim 1, wherein when it is determined that there is a blink action from the determined open / closed state, it is determined that the time-series change of the eye coordinates is due to the visual behavior of the subject. Visual recognition behavior determination device.   When it is determined that there is a blink action from the open / closed state determined by the opening / closing eye determination means, the visual action determination means obtains an average value of the movement amount of the eye coordinates between the blink actions, When the movement amount of the eye coordinates between blink actions is greater than or equal to a set value, it is determined that the time-series change of the eye coordinates is due to the visual behavior of the subject. The visual recognition behavior determination device according to 1. The eye coordinate acquisition means includes
A single eye candidate detecting means for setting a processing area for detecting a single eye candidate from the right half or the left half of the entire face image, and detecting single eye candidate coordinates from the processing area;
When one-eye candidate coordinates are detected by the one-eye candidate detection means, an area for detecting both eyes is set based on the one-eye candidate coordinates, and both eyes for the one-eye candidate coordinates are set in the area. The visual-behavior determination device according to claim 1, further comprising: a binocular confirmation processing unit that performs the confirmation processing.   The eye coordinate acquisition means sets a plurality of vertical lines in a processing area for detecting an eye in a face image, and detects a density change point by obtaining a differential value of the density change of the light amount for each vertical line. 4. The visual behavior determination device according to claim 1, wherein the center coordinates of continuous data in which density change points between the vertical lines are continuous in the horizontal direction are acquired as eye coordinates. 5. .   If the current reference eye coordinates are detected from an area centered on the eye coordinates acquired in time earlier, the eye coordinate acquisition means determines whether the eye coordinates from the reference eye coordinates in time The visual behavior determination device according to any one of claims 1 to 3, wherein the opposite eye existing in the vicinity of the coordinates separated by the acquired distance between both eyes is obtained and both eyes are tracked. The opening / closing eye determination means includes
Of the eye coordinates acquired by the eye coordinate acquisition means, it is determined whether the eye shape is upwardly convex from the left edge coordinates, upper edge coordinates, and right edge coordinates of the eye, and the eye shape when the eye shape is upwardly convex Learning to record the distance between the upper and lower coordinates as the eye opening value, and to set a value obtained by subtracting a predetermined ratio from the eye opening value as a threshold for distinguishing between opening and closing The visual behavior determination device according to any one of claims 1 to 3, wherein the eye opening / closing state of the subject is determined from the threshold value and the eye opening.   When the vehicle speed of the host vehicle detected by the vehicle speed sensor is a predetermined value or less, or when the steering angle of the host vehicle detected by the steering angle sensor is a predetermined value or more, the eye coordinate acquisition means The visual behavior determination apparatus according to any one of claims 1 to 3, wherein the tracking of the tracking is suspended.   When the inter-vehicle distance between the host vehicle and the preceding vehicle detected by the inter-vehicle distance detection sensor is suddenly approached, the visual recognition determination means determines the eye coordinate from the time-series change of the eye coordinate and the time-series change of the open / closed state. The process of determining whether or not the time-series change is due to the visual behavior of the subject is stopped, and it is determined whether or not the subject is viewing the front from the time-series change of eye coordinates. The visual recognition behavior determination device according to any one of claims 1 to 3.   The visual behavior determination means indicates that the current visual recognition area and the non-visual recognition area that the subject visually recognizes among the obstacle areas in front of the vehicle detected by the obstacle sensor, and visually recognizes the non-visual recognition area. The visual behavior determination device according to any one of claims 1 to 3, wherein a signal is output to an alarm device through visual recognition result output means. A face image of the subject included in the video signal output from the imaging unit is acquired, and based on the face image, eye coordinates indicating the eye position of the subject are detected and the eye coordinates are tracked and tracked. When calculating the eye opening of the subject based on the eye coordinates and the face image, and determining the open / closed state of the subject's eyes,
From the time-series change of the eye coordinates being tracked and the time-series change of the open / closed state of the eye, it is determined whether the time-series change of the eye coordinates is due to the visual behavior of the subject, A visual recognition behavior determination device characterized by outputting a determination result as a signal.