JP2005081101A - System and methodology for detecting visual axis direction - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To detect a visual axis direction indirectly and with high precision from an image of the face which is imaged. <P>SOLUTION: A visual axis direction detecting system is composed of an imaging device 20 which images the face, a face outline extraction means 30A which extracts a face profile from the image, a face aspect information calculation means 30B which computes the area ratio of the forehead and both cheeks to the whole face based on the image and the face profile, a face turning orientation presuming means 30C which presumes the face turning orientation to the imaging orientation based on the area ratio of the forehead and both the cheeks, a both pupil coordinate sensing means 30H and a nose coordinate sensing means 30I which detect coordinates, in which both the pupils and a nose are located, from the image, respectively, a centroid calculation means 30J which computes the centroid of an inverse triangle in which the coordinates of both the pupils and the nose are connected with a segment, a visual axis orientation presuming means 30K which presumes the visual axis orientation of in the face based on the inverse triangle and its centroid, and a visual axis orientation correction means 30L which rectifies the visual axis orientation based on the face turning orientation so that it may become the visual axis orientation to the imaging orientation. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a technique for detecting a gaze direction indirectly and with high accuracy from an image of a face imaged.

Conventionally, various technologies have been developed that effectively prevent a drowsy driving or the like and raise safety during vehicle operation by calling attention when the vehicle driver is in a low arousal state. And as described in patent document 1 or patent document 2, while extracting eyes from the image which imaged the vehicle driver's face, paying attention to the blink which is the opening / closing operation | movement, the technique which determines an arousal state Proposed.
Japanese Unexamined Patent Publication No. 2000-137792 JP-A-11-147428

  However, even if the vehicle driver does not reach a drowsy state, for example, when he / she becomes distracted due to fatigue or the like, the visual observation around the vehicle is neglected, and a fixation state in which one point is unconsciously stared. Such a fixation state cannot be detected from blinking of the eyes, and it is necessary to use the line-of-sight direction of the vehicle driver in order to further improve safety during vehicle operation. An “eye camera” is known as a device for detecting the line-of-sight direction, but wearing it on the head of a vehicle driver is not only cumbersome but may cause trouble in driving the vehicle.

  Therefore, in view of the conventional problems as described above, the present invention estimates the face turning direction and the gaze direction on the face from the image obtained by imaging the face, and corrects the gaze direction on the face based on the face turning direction. An object of the present invention is to propose a technique for detecting the gaze direction indirectly and with high accuracy.

  For this reason, in the gaze direction detection device according to the first aspect of the present invention, an imaging unit that captures an image of a face, a facial contour extraction unit that extracts a facial contour from an image captured by the imaging unit, and the imaging Based on the image captured by the means and the facial contour extracted by the facial contour extracting means, the facial information calculating means for calculating the forehead and the cheek area ratio with respect to the entire face, and the facial information calculating means Based on the forehead for the entire face and the area ratio of both cheeks, the face turning direction estimating means for estimating the face turning direction with respect to the imaging direction, and the coordinates at which both pupils and nose are located from the image taken by the imaging means Coordinate detection means for detecting; centroid calculation means for calculating the centroid of an inverted triangle connecting the coordinates of both pupils and nose detected by the coordinate detection means with line segments; and both the pupils and nose Based on the inverted triangle obtained by connecting the coordinates with line segments and the centroid calculated by the centroid calculating means, the gaze direction estimating means for estimating the gaze direction on the face, and the face turning direction estimated by the face turning direction estimating means And a gaze direction correction unit that corrects the gaze direction estimated by the gaze direction estimation unit so as to be a gaze direction with respect to the imaging direction.

  According to the second aspect of the invention, based on the offset angle of the imaging direction with respect to the reference direction, the face turning direction correction that corrects the face turning direction estimated by the face turning direction estimation means to be the face turning direction with respect to the reference direction. And the gaze direction correcting unit corrects the gaze direction estimated by the gaze direction estimating unit to be a gaze direction with respect to a reference direction based on the face turning direction corrected by the face turning direction correcting unit. It is characterized by doing.

The invention according to claim 3 further comprises facial feature setting means for setting a range occupied by both eyes and nose of the face, wherein the coordinate detection means is based on an image within the range set by the facial feature setting means. It is characterized by detecting coordinates where both pupils and nose are located.
According to a fourth aspect of the present invention, there is provided face feature learning means for learning to update the range set by the face feature setting means based on the latest image.

According to a fifth aspect of the present invention, the coordinate detecting means uses a midpoint of a line segment connecting the centers of both nostrils as a nose coordinate.
In the gaze direction detection method according to the sixth aspect of the present invention, a step of extracting a facial contour from an image obtained by imaging a face by an imaging device, and a forehead and areas of both cheeks based on the image and the facial contour. A step of calculating a ratio, a step of estimating a face turning direction with respect to an imaging direction based on a forehead and an area ratio of both cheeks with respect to the whole face, and a step of detecting coordinates where both pupils and nose are located from the image Calculating a center of gravity of an inverted triangle obtained by connecting the coordinates of both eyes and nose with line segments; estimating a gaze direction on the face based on the inverted triangle and its center of gravity; and a face relative to the imaging direction Correcting the line-of-sight direction to be the line-of-sight direction with respect to the imaging direction based on the turning direction. .

  According to the invention described in claim 1 or claim 6, the forehead and the area ratio of both cheeks to the entire face are calculated based on the image obtained by capturing the face and the face contour extracted therefrom. Then, the face turning direction with respect to the imaging direction is estimated based on the forehead for the entire face and the area ratio of both cheeks. Also, the coordinates of both pupils and nose are detected from the image of the face, and the center of gravity of an inverted triangle connecting the coordinates of both pupils and nose with line segments is calculated. Thereafter, the gaze direction on the face is estimated based on the inverted triangle and its center of gravity, and the gaze direction is corrected so as to be the gaze direction with respect to the imaging direction based on the face turning direction. Therefore, it is possible to detect the gaze direction indirectly and with high accuracy from the image obtained by capturing the face without using an eye camera or the like that is directly attached to the subject.

According to the second aspect of the present invention, the face turning direction becomes the face turning direction with respect to the reference direction based on the offset angle of the image pickup direction even if the image pickup apparatus is not installed at an ideal position facing the subject. Therefore, it is possible to prevent a reduction in detection accuracy in the line-of-sight direction.
According to the invention described in claim 3, since the coordinates where both eyes and nose are located are detected from the image within the range occupied by both eyes and nose of the face, the load required for image processing is reduced, and the processing capability is reduced. Can be improved.

According to the fourth aspect of the present invention, since the range occupied by both eyes and nose of the face is updated and learned based on the latest image, for example, even if the subject changes its appearance, the processing capability Can be suppressed.
According to the fifth aspect of the invention, since the midpoint of the line segment connecting the centers of both nostrils is the nose coordinate, the coordinate where the nose is located can be specified with high accuracy.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.
FIG. 1 shows the configuration of a gaze direction detecting device constructed by applying the present invention to a cab-over type track.
The line-of-sight detection device includes an imaging device 20 (imaging means) such as a CCD (Charge Coupled Device) camera that images the face 10A of the vehicle driver 10 and a control device 30 incorporating a computer. The control device 30 detects the direction of the line of sight of the vehicle driver 10 indirectly and with high accuracy by analyzing the image captured by the imaging device 20. Then, in the control device 30, as shown in FIG. 2, a facial contour extraction unit 30 A, a facial information calculation unit 30 B, a facial turning direction estimation unit 30 C, and the like are stored by a program stored in a memory such as a ROM (Read Only Memory). Imaging direction setting means 30D, facial turning direction correction means 30E, facial feature setting means 30F, facial feature learning means 30G, binocular coordinate detection means 30H, nose coordinate detection means 30I, centroid calculation means 30J, gaze direction estimation means 30K and gaze Each of the direction correction means 30L is realized. The pupil detection unit 30H and the nose coordinate detection unit 30I constitute a coordinate detection unit.

The facial contour extraction unit 30A extracts the facial contour of the vehicle driver 10 from the image captured by the imaging device 20 using a known image processing technique.
The face information calculation means 30B calculates the forehead and the area ratio of both cheeks (hereinafter referred to as “face information”) based on the image and the face contour. That is, as shown in FIG. 3, the image is divided by a mesh of a predetermined interval, and the rectangular elements included in the range occupied by the forehead and both cheeks are counted, so that the forehead and both cheeks for the entire face demarcated by the facial contour. The area ratio (ratio) occupied by is calculated.

  The face turning direction estimation means 30C estimates the face turning direction (the direction in which the face is directed) with respect to the imaging direction based on the face information. That is, with reference to FIG. 4A showing an image of the face facing the imaging direction, the face is upward, downward, upper right, rightward as shown in FIGS. When facing the lower right, there is a characteristic that the forehead and the area ratio of both cheeks to the entire face change. For example, as shown in FIG. 5B, when the face faces upward, the area ratio of the left and right cheeks increases at a substantially constant rate, while the area ratio of the forehead decreases. Also, as shown in FIG. 4D, when the face is directed to the upper right, the area ratio of the right cheek and forehead decreases, while the area ratio of the left cheek increases. For this reason, it can be estimated which direction the face is facing with respect to the imaging direction through the change in the area ratio of the forehead and both cheeks.

  In the imaging direction setting means 30D, the imaging direction with respect to the vehicle front as the reference direction, in other words, the offset angle of the imaging device 20 with respect to the vehicle front is set. In other words, when the vehicle driver 10 is looking forward, it is ideal to take an image of the face 10A from the directly facing position. However, if the imaging device 20 is at a position directly opposite the vehicle driver 10, the forward view may be hindered, so it is practically impossible to install the imaging device 20 at such an ideal position. It is. For this reason, the structure which installs the imaging device 20 in the position which faces the vehicle driver 10 and the face 10A can image, for example, a dashboard, a ceiling, etc. must be employ | adopted. When the imaging device 20 is installed at such a position, as shown in FIGS. 5 and 6, the imaging direction is deviated from the ideal facing direction with respect to the vertical plane and the horizontal plane. Therefore, it is desirable that a function for presetting the offset angle of the imaging device 20 with respect to the front of the vehicle is provided so that the face turning direction with respect to the imaging direction can be corrected to be the face turning direction with respect to the front of the vehicle.

The face turning direction correcting unit 30E corrects the face turning direction with respect to the imaging direction based on the offset angle of the imaging device 20, and obtains the face turning direction with respect to the front of the vehicle. That is, by subtracting the offset angle of the imaging device 20 from the face turning direction with respect to the imaging direction, the face turning direction with the vehicle front as the reference direction is obtained.
In the facial feature setting means 30F, in order to limit the image processing range and improve the processing capability, a range in which both eyes and nose of the face 10A of the vehicle driver 10 will be occupied (hereinafter referred to as “facial feature”). Is set. That is, both eyes and nose of the vehicle driver 10 are at different heights depending on their physical characteristics such as height and sitting height. Although it is possible to extract both eyes and nose from the entire image, it requires a high processing capability for the control device 30 and is not realistic from the viewpoint of cost or the like. Therefore, by limiting the image processing range based on the facial features, the processing capability can be improved through a reduction in processing load.

In the facial feature learning means 30G, the facial feature of the vehicle driver 10 is appropriately updated and learned through a difference from the latest image in view of, for example, changing the facial feature by extending the beard.
In the binocular coordinate detection means 30H, the binocular region is extracted from the image for the image processing range limited by the latest facial features. Then, both pupils are further extracted from the extracted binocular region, and their center coordinates are detected as pupil coordinates. When the vehicle driver 10 is wearing sunglasses, the position of both eyes cannot be specified from the image, and therefore the lens center of the sunglasses may be set as the pupil coordinates.

In the nose coordinate detection means 30I, the nostril region is extracted from the image for the image processing range limited by the latest facial features. Then, as the nose coordinates, the center coordinates of the extracted nostrils, that is, the midpoint of the line segment connecting the left and right nostril centers are detected.
As shown in FIG. 7, the center-of-gravity calculating means 30J calculates the center of gravity of an inverted triangle obtained by connecting the coordinates of both pupils and nose with line segments.

  The line-of-sight direction estimation means 30K estimates the line-of-sight direction on the face 10A based on the inverted triangle and the change in its center of gravity. For example, as shown in FIG. 8A, when the line of sight is directed to the upper right, the line segment connecting both pupils in the inverted triangle is tilted downward toward the left, and the center of gravity is the center of the inverted triangle. Move to the upper right. Further, as shown in FIG. 5B, when the line of sight is directed to the lower right, the line segment connecting both pupils in the inverted triangle is inclined downward toward the right, and the center of gravity thereof is the center of the inverted triangle. Move to the lower right. For this reason, if the regularity corresponding to a gaze direction is used, the gaze direction in the face 10A can be estimated indirectly and with high accuracy.

The line-of-sight direction correcting means 30L corrects the line-of-sight direction on the face based on the face turning direction with respect to the front of the vehicle, and obtains the line-of-sight direction with respect to the front of the vehicle. That is, the line-of-sight direction with respect to the front of the vehicle is obtained by adding the line-of-sight direction on the face to the face turning direction with respect to the front of the vehicle.
Next, a processing procedure executed by the gaze direction detection device will be described with reference to a flowchart shown in FIG.

In step 1 (abbreviated as “S1” in the figure, the same applies hereinafter), the face 10A of the vehicle driver 10 is imaged by the imaging device 20.
In step 2, a facial contour is extracted from the image.
In step 3, facial information is calculated based on the image and facial contour.
In step 4, the face turning direction with respect to the imaging direction is estimated based on the face information.

In step 5, the face turning direction is corrected based on the offset angle of the imaging device 20, and the face turning direction with respect to the front of the vehicle is obtained.
In step 6, the image processing range (extraction range) that is considered to be occupied by both eyes and nose of the face 10A based on the latest facial features is limited.
In step 7, the right eye region is extracted from the image for the limited right eye extraction range.

In step 8, the right pupil is further extracted from the extracted right eye region, and the center coordinate is calculated as the right pupil coordinate.
In step 9, the left eye region is extracted from the image for the limited left eye extraction range.
In step 10, the left pupil is further extracted from the extracted left eye region, and the center coordinate is calculated as the left pupil coordinate.

In step 11, a nostril region is extracted from the image for the limited nose extraction range.
In step 12, the midpoint coordinates of the line segment connecting the center of the left and right nostrils are calculated as nose coordinates from the extracted nostril area.
In step 13, inverted triangles having apexes at both pupil coordinates and nose coordinates are created.
In step 14, the barycentric coordinates are calculated from the vertex coordinates of the created inverted triangle.

In step 15, the line-of-sight direction on the face is estimated based on the inverted triangle and the change in its center of gravity.
In step 16, the gaze direction on the face is corrected based on the face turning direction with respect to the imaging direction, and the gaze direction with respect to the front of the vehicle is obtained.
According to this gaze direction detection device, a facial contour is extracted from an image obtained by capturing a facial image, and the forehead and the cheek area ratio with respect to the entire facial surface are calculated based on the image and the facial contour. Here, the area ratio of the forehead and both cheeks with respect to the entire face has a characteristic that changes with regularity according to the turning direction of the face with respect to the imaging direction, as shown in FIG. Therefore, by using the regularity, it is possible to estimate the face turning direction with respect to the face imaging direction.

  By the way, it is ideal to image the face from a position facing the subject. However, when the present invention is applied to the vehicle, it is necessary to ensure the forward view of the vehicle driver 10, and thus the image is captured at the ideal position. The device 20 cannot be installed. For this reason, if the facial turning direction with respect to the imaging direction is corrected to be the facial turning direction with respect to the reference direction based on the offset angle of the imaging device 20 with respect to the vehicle front as the reference direction, the imaging device 20 is installed at an ideal position. Even if it is not done, it is possible to prevent a reduction in the detection accuracy of the line-of-sight direction due to this.

  On the other hand, coordinates where both pupils and nose are located are detected from an image of the face, and the center of gravity of an inverted triangle connecting the coordinates of both pupils and nose with line segments is calculated. At this time, in order to improve the processing capability by limiting the image processing range, coordinates where both eyes and nose are located are detected from an image within a range where both eyes and nose are supposed to occupy the face. Here, as shown in FIG. 8, the inverted triangle and its center of gravity have a characteristic that changes with regularity according to the line-of-sight direction on the face. Therefore, by using the regularity, it is possible to estimate the gaze direction on the face.

Then, by correcting the gaze direction on the face based on the face turning direction, that is, by adding the face turning direction and the gaze direction on the face, the gaze direction with respect to the reference direction can be detected indirectly and with high accuracy.
Note that the gaze direction detection device according to the present invention is not limited to the configuration for detecting the gaze direction of the vehicle driver, and can be applied to various applications in which the gaze direction is detected by, for example, an “eye camera”. Nor.

Configuration diagram of a gaze direction detecting device constructed by applying the present invention to a track Block diagram of various functions in the control device Explanatory diagram of how to calculate facial information FIGS. 9A to 9F illustrate a method for estimating a face turning direction with respect to an imaging direction, and FIGS. 6A to 6F are explanatory diagrams when the line-of-sight directions are front, upper, lower, upper right, right, and lower right, respectively. Explanatory drawing of the offset state of the imaging device in the vertical plane The offset state of the imaging device in the horizontal plane is shown, and (A) and (B) are explanatory diagrams when the imaging device is offset to the right and left, respectively. Illustration of an inverted triangle defined by both pupil coordinates and nose coordinates and its center of gravity The method of estimating the gaze direction on the face is shown, and (A) and (B) are explanatory diagrams when the gaze is directed to the upper right and lower right, respectively. The flowchart which shows the process sequence performed with a gaze direction detection apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Vehicle driver 10A Face 20 Imaging device 30 Control apparatus 30A Face outline extraction means 30B Face information calculation means 30C Face turning direction estimation means 30D Imaging direction setting means 30E Face turning direction correction means 30F Face feature setting means 30G Face feature learning means 30H Binocular coordinate detection means 30I Nose coordinate detection means 30J Center of gravity calculation means 30K Gaze direction estimation means 30L Gaze direction correction means

Claims (6)

Imaging means for capturing an image of the face;
Facial contour extraction means for extracting a facial contour from an image captured by the imaging means;
Face information calculation means for calculating the forehead and the cheek area ratio for the entire face based on the image taken by the image pickup means and the face contour extracted by the face contour extraction means;
A face turning direction estimating means for estimating the face turning direction with respect to the imaging direction based on the forehead and the area ratio of both cheeks calculated by the face information calculating means;
Coordinate detection means for detecting coordinates where both pupils and nose are located from the image captured by the imaging means;
Centroid calculating means for calculating the centroid of an inverted triangle connecting the coordinates of both pupils and nose detected by the coordinate detecting means with line segments;
Line-of-sight direction estimating means for estimating the line-of-sight direction on the face based on the inverted triangle connecting the coordinates of both eyes and nose with line segments and the center of gravity calculated by the center-of-gravity calculating means;
Gaze direction correcting means for correcting the gaze direction estimated by the gaze direction estimating means based on the face turning direction estimated by the face turning direction estimating means to be a gaze direction with respect to the imaging direction;
A line-of-sight direction detection device comprising: Based on the offset angle of the imaging direction with respect to the reference direction, the face turning direction correcting means for correcting the face turning direction estimated by the face turning direction estimating means to be the face turning direction with respect to the reference direction,
The line-of-sight direction correcting unit corrects the line-of-sight direction estimated by the line-of-sight direction estimating unit to be a line-of-sight direction with respect to a reference direction based on the face turning direction corrected by the face turning direction correcting unit. The gaze direction detecting device according to claim 1. Comprising facial feature setting means for setting a range occupied by both eyes and nose of the face;
The gaze direction detection according to claim 1, wherein the coordinate detection unit detects coordinates where both pupils and the nose are located from an image within a range set by the facial feature setting unit. apparatus.   4. The gaze direction detecting device according to claim 3, further comprising facial feature learning means for learning to update the range set by the facial feature setting means based on the latest image.   The gaze direction detecting device according to any one of claims 1 to 4, wherein the coordinate detecting means uses a midpoint of a line segment connecting the centers of both nostrils as a nose coordinate. Extracting a facial contour from an image of the face captured by the imaging device;
Calculating a forehead and cheek area ratio for the entire face based on the image and facial contour;
Estimating the face turning direction with respect to the imaging direction based on the forehead for the entire face and the area ratio of both cheeks;
Detecting the coordinates where both pupils and nose are located from the image;
Calculating the center of gravity of an inverted triangle connecting the coordinates of both pupils and nose with line segments;
Estimating a gaze direction on the face based on the inverted triangle and its center of gravity;
Correcting the line-of-sight direction to be the line-of-sight direction with respect to the imaging direction based on the face turning direction with respect to the imaging direction;
A gaze direction detecting method characterized by causing a computer to execute.

Images