JP2003271932A - Sight line direction detector - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a sight line direction detector capable of detecting the sight line direction with excellent accuracy from an image picked up from a face. <P>SOLUTION: The image of the face is picked up by a face image picking-up means CL1. The position of eyes on the image is specified by an eye position specifying means CL2 from the face image. An eyeball rotational angle detecting means CL4 detects the rotational angle of the eyeball from the face image and the eye position. A face direction detecting means CL3 detects the direction of the face from the face image. A sight line direction calculating means CL5 calculates the sight line direction from the rotational angle of the eyeballs detected by the eyeball rotational angle detecting means CL4 and the direction of the face detected by the face direction detecting means CL3. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a line-of-sight direction detecting device for detecting a line-of-sight direction from a face image obtained by photographing the face of a subject.

[0002]

2. Description of the Related Art As a device for detecting the line-of-sight direction, Japanese Patent Laid-Open No. 9-53917 "Vehicle line-of-sight direction measuring device" is known. This device includes an imaging device that captures a reflected image from an eyeball of an occupant, a first illumination that emits invisible light that is coaxial with the imaging device, and a second illumination that emits invisible light from a position different from the first illumination. It has and. Then, the position of the center of the pupil and the position of the corneal reflection image are obtained by image processing from image data obtained by alternately illuminating the face from the first and second illuminations, and the line-of-sight direction is calculated based on the both. .

[0003]

However, in the above-mentioned prior art, since the line-of-sight direction is measured from a narrow range of the face image that captures the position of the corneal reflection image in the pupil, the line-of-sight with high accuracy is obtained. There was a problem that the direction could not be detected.

Although it is possible to detect the line of sight on the assumption that the position of the eye does not move, it is more difficult to specify the direction of the line of sight in an environment where the face is moving such as while driving.

In view of the above problems, an object of the present invention is to provide a line-of-sight direction detecting device capable of accurately detecting the line-of-sight direction from a face image.

[0006]

According to the present invention, the eye position is specified from the face image picked up by the face image pick-up means, and the rotation angle of the eyeball is detected from the image specified by the eye position specifying means. A line-of-sight direction detection device that detects the direction of the face from the face image captured by the face-image capturing means and detects the direction of the line of sight from the eyeball rotation angle and the face direction.

Further, according to the present invention, the face image capturing means for capturing the face image, the eye position identifying means for identifying the eye position on the image from the face image captured by the face image capturing means, and the face image capturing means are provided. Eye rotation angle detection means for detecting the rotation angle of the eyeball from the captured face image and the specified eye position, and face orientation detection means for detecting the face orientation from the face image captured by the face image capturing means. And a gaze direction calculating means for calculating a gaze direction from the rotation angle of the eyeball detected by the eyeball rotation angle detecting means and the face direction detected by the face orientation detecting means, It is a detection device.

[0008]

According to the present invention, there is an effect that it is possible to provide a line-of-sight direction detecting device for detecting the line-of-sight direction with high accuracy.

[0009]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described in detail below with reference to the drawings.

[First Embodiment] FIG. 1 is a block diagram for explaining the arrangement of a first embodiment of a line-of-sight direction detecting device according to the present invention. In the figure, the line-of-sight direction detection device includes a face image capturing unit CL1 that captures a face image, and an eye position identifying unit CL2 that identifies the position of an eye on the image from the face image captured by the face image capturing unit CL1. Face image capturing means C
Eye rotation angle detecting means CL4 for detecting the rotation angle of the eyeball from the face image captured by L1, and face image capturing means CL
Face orientation detecting means CL3 for detecting the orientation of the face from the face image picked up by No. 1 and eye rotation angle and face orientation detecting means CL detected by the eyeball rotation angle detecting means CL4.
And a line-of-sight direction calculation means CL5 for calculating the line-of-sight direction from the face direction detected by the reference numeral 3.

The face image pickup means CL1 is a TV for picking up the face of a driver or an operator and outputting face image data.
It is an imaging means such as a camera. The eye position specifying means CL2 is
The face image data output from the face image capturing means CL1 is processed to identify the position of the eye on the image.

The face orientation detecting means CL3 processes the face image data output from the face image capturing means CL1 to detect the face orientation with the optical axis line of the face image capturing means CL1 as the front.

The eyeball rotation angle detecting means CL4 is further provided with a pupil position detecting means CL41 and a peak position detecting means CL42 of an arc line.

The pupil position detecting means CL41 detects the position of the pupil by image-processing a minute image which is a partial image of a face image centered on the eye position specified by the eye position specifying means CL2.

Peak position detecting means CL4 of the arc line
2 performs image processing of a minute image which is a partial image of a face image centered on the eye position specified by the eye position specifying means CL2, thereby making the upper eyelid, the lower eyelid, the eyebrow, the upper frame of the eyeglass, and the eyeglass The arc-shaped line that is the reference of the line of sight such as the lower frame is detected, and the peak position of the arc-shaped line is obtained from the detection.

In the present embodiment, only the case of the upper eyelid will be described, but other parts can be detected by performing similar processing.

The present apparatus can be used to detect the line-of-sight direction of an operator of a car, a railroad car, a ship, a plant, etc., but will be described when it is applied to the eyes of a driver of a car in all the embodiments of the present invention. .

[Arrangement of Equipment] FIG. 2 is a layout of the equipment of the present invention. TV camera 1 as face image capturing means CL1
Is installed on the instrument panel of the automobile at a position where the driver can be imaged substantially in front, and the driver's face is photographed. In the present embodiment, the input image of the TV camera 1 is a black-and-white image including, for example, 640 pixels in the horizontal direction (X) and 480 pixels in the vertical direction (Y). An input image captured by the TV camera 1 is input as image data to the microcomputer 2 installed inside the vehicle body such as the back side of the instrument panel.

The microcomputer 2 is a TV camera 1
An image memory for storing the image data input from is provided. Further, in the microcomputer 2, in order to process the image data and detect the line-of-sight direction, the eye position specifying unit CL2, the face direction detecting unit CL3, the eyeball rotation angle detecting unit CL4, and the line-of-sight direction calculating unit CL5. The logic to realize and is programmed.

Next, the processing contents of the system will be described.

[Processing of Entire System] FIG. 3 is a schematic flowchart for explaining the flow of the entire processing of the system. First, when the process is started, step S1 (hereinafter,
"Step S" is simply referred to as "S"), and the process of "initial value input" is executed to initialize the work area of the microcomputer and set the initial values of various parameters.

In the "face image pickup process" of S2, the TV camera 1 picks up a face image and inputs it to the microcomputer 2 as image data.

In the "eye position specifying process" of S3, the position of the eye on the face image is specified by image-processing the image data input to the microcomputer 2.

In S4, if the eyes can be identified on the face image based on the output result of S3, the process proceeds to S4 YES and S5. If the eyes cannot be identified on the face image, the process returns to S4 NO and S2 to identify the eyes. The processes from S2 to S4 are repeated until it is possible.

In the "face orientation detection process" in S5, the orientation of the face on the image with respect to the optical axis direction of the face image capturing means is detected by image processing the image data input to the microcomputer 2.

In the "arc-shaped line peak position detection process" in S6, a minute image (hereinafter also referred to as a minute area) which is a partial image of a face image centered on the eye position specified in S3 is subjected to image processing. Detects the arc-shaped line and its peak position on the micro image.

In the "pupil position detection process" of S7, the position of the pupil on the image is detected by image-processing the micro area centered on the eye position specified in S3.

In the "eyeball rotation angle detection process I" of S8,
The rotation angle of the eyeball is detected from the peak position of the arc-shaped line detected in S6 and the position of the pupil detected in S7.

In the "line-of-sight direction calculation process" of S9, S5
The line-of-sight direction is calculated from the face orientation detected in S8 and the rotation angle of the eyeball detected in S8, and the process ends.

[Eye Position Specifying Process] (S3) The "eye position specifying process" S3 of FIG. 3 will be described with reference to the detailed flowchart of FIG. First, in S31 of FIG. 4, "eye candidate position specifying processing" for specifying the eye candidate position from the face image.
To execute. In S32, an "eye determination process" for determining whether or not it is an eye from the position of each identified eye candidate
I do.

In S33, it is determined whether or not all the eye candidate points detected in the process of specifying the eye candidate positions in S31 have been determined. If it is determined that all the eye candidate points have been determined, the process proceeds to S4 of the main routine if YES in S33.
If all the eye candidate points have not been determined, S33N
O returns to the eye determination process of S32.

[Eye Candidate Position Specifying Process] (S31) The flow of the "eye candidate position specifying process" S31 will be described with reference to the detailed flowchart of FIG. 5 and FIGS.

First, in step S311 of FIG. 5, the entire face image data captured in the "face image capturing process" S2 and input to the microcomputer 2 is stored as an entire image G in the image memory.

Then, in steps S312 to S316,
The whole image G is scanned in the Y direction to extract characteristic points.
In S312, after the completion of one line in the vertical direction, the process proceeds to the processing of the next adjacent line, and it is determined whether or not the feature point extraction on all the vertical lines is completed. If it is determined in S312 that the feature point extraction has not been completed for all lines, the process proceeds to S313.

In S313, the smoothing processing of the density value (luminance value) of one line in the vertical direction (Y-axis direction) in the vertical direction (Y-axis direction) is performed by arithmetic average calculation. This process
The purpose is to eliminate small variations in changes in the density value when capturing image data, and to capture global changes in the density value.

For example, the image is a collection of pixels in a two-dimensional array, the density value of each pixel of the image before processing is A (i, j), and this pixel itself and n pixels before and after the smoothing are used for smoothing. Then, the density value B (i, j) of each pixel after the smoothing process is represented by the arithmetic mean value of the equation (1).

[0037]

## EQU1 ## B (i, j) = [1 / (2n + 1)] [A (in, j) + ... + A (i-1, j) + A (i, j) + A (i + 1, j) + + A (i + n, j)] (1) In S314, a differential operation (difference operation) in the vertical direction (Y-axis direction) of the image at the arithmetic mean value as the operation result of S313 is performed. In S314, feature point extraction is performed using the differential value that is the calculation result of S314. Here, points where the differential value changes from negative to positive are extracted as feature points. In other words, the minimum value of the vertical density change in one line is extracted as the feature point. After this processing is completed for one line, S31
At 6, the process is switched to the next line.

If it is determined in S312 that the characteristic points of all the lines have been extracted, the process proceeds to S317, in which the Y coordinate values of the extracted characteristic points of the adjacent lines are compared, and if the Y coordinate values are within a predetermined value. , As continuous data, group number of continuous data, continuous start line number, number of continuous data, average value of vertical position of each extracted feature point forming continuous data (representative Y coordinate value of the continuous data), continuous start line And the average value of the horizontal position of the end line (representative X coordinate value of the continuous data) is stored.

Since the detection target here is the eye, it can be said that the characteristic pattern is data that continues in the horizontal direction for a relatively long time, so continuous data can be selected on the condition that it continues for a predetermined value or more in the horizontal direction. .

FIG. 6 shows the feature pattern of the face thus selected as continuous data g. that's all,
The method of extracting the continuous data g has been briefly described. For details of the processing state, see Japanese Patent Laid-Open No. 10-40361.
It is also described in "JP-A-10-143669". The continuous data g is, so to speak, an eye candidate, and the representative coordinate value C of this continuous data g is the position of the eye candidate.

Next, in S318, the existence area EA including each continuous data g is set with the representative coordinate value C of each continuous data g as shown in FIG. 6 as a reference. This existence area EA
Is determined as follows.

(How to Determine Size of Presence Area EA) The size of the presence area EA is determined as shown in FIGS. FIG. 7 shows the size of the existence area EA, and FIGS. 8 and 9 show the statistical data of the horizontal and vertical lengths of the eyes obtained by examining the sizes of the eyes of several people. Here, the size of the existing area EA is preferably as small as possible in order to reduce noise (extract facial wrinkles, light and shade, etc.) and not reduce the processing speed. The size currently used in processing such as drowsiness detection is set to a size in which the size of the eyes of several people is examined and a margin (for example, × 1.5 times) is added.

As a method of statistically obtaining the size of the eye, as shown in FIGS. 8 and 9, the data of the vertical and horizontal dimensions of the eye are collected, and the size is determined, for example, by considering the margin for the size that covers 95% of the distribution. There are possible ways to do this. And the dimension that covers this 95%, that is, the horizontal dimension xa ', the vertical dimension ya'
As shown in FIG. 6, the margin (× 1.5) is taken into consideration for the determination. A method of estimating the width and height of the eye by image processing and adding a margin to the vertical and horizontal sizes is also conceivable.

(How to Determine Position of Presence Area EA) FIG. 10
Shows a method of positioning the presence area EA of the right eye, for example. Based on the eye coordinate values (x1, y1),
A reference point P that draws the existing area EA is determined at the positions of the distances x2 and y2, and the predetermined size x of the existing area EA is determined from the point P.
3 and y3 are drawn and the position is determined. x2 and y2 are x
The length is set such that the existing area EA comes to the center of the eye in advance by 1/2 of 3, y3.

The existence area EA is set for all the continuous data g found in the entire image.

[Eye determination process] (S32) Next, the "eye determination process" S32 will be described with reference to the detailed flowchart of FIG. 11 and FIGS.

First, in step S3201 of FIG. 11, the image data of the existence area EA of the eye candidate point is stored in the image memory as a minute image IG which is a partial image of the entire image G. FIG. 12 shows the states of the entire image G and the stored minute image IG.
The image data captured in the “face image capturing process” S2 and input to the microcomputer 2 is already stored in the image memory as the entire image G.

Next, in S3202, a binarization threshold value is set based on the density information of the range AR with the representative coordinate value IC of the minute image IG corresponding to the representative coordinate value C of the whole image G as a reference. This range AR is smaller than the existing area EA,
The binarization threshold can be set accurately.

An example of the method of calculating the binarization threshold value in each range AR will be described with reference to FIG. In the range AR, the density values of several lines are read out in the vertical direction. In FIG.
This indicates that there are four lines in the vertical direction. The highest (bright) density value and the lowest (dark) density value of each line are stored in memory, and when the memory of all lines is completed, the highest (bright) density value of each line , The lowest concentration value (on the skin),
Among the lowest (darkest) density values of each line, the lowest density value (eye part) is obtained, and the median value thereof is used as the binarization threshold value.

The range AR for the binarization threshold is set so that the black part of the eye and the white part of the skin around the eye are included, and in order to reduce the influence of the variation in the brightness of the image. The minimum size required. In addition, the binarization threshold is set to the median value of the lowest (dark) density value of the eye and the lowest (dark) density value of the skin part in the area, and the skin part of the face image is determined. It is a value suitable for cutting out the eye part from.

Further, the reason why the lowest (dark) density value of the density value of the skin portion is used to determine the binarization threshold value is as follows. As described above, in order to reduce the influence of the variation in the brightness around the eyes, even if the range AR for reading the density value is made as small as possible, the range AR
This is because a portion where a direct light is shining on a part of the above may enter as a disturbance, and this portion is not used for determining the binarization threshold.

In S3203, the minute image IG is binarized using the binarization threshold value thus determined, and is stored in the image memory as a binary image bG.

By detecting the binarized candidate object using such a binarization threshold value, the eye can be accurately captured and the determination using the geometrical shape of the candidate object can be performed more accurately. The position detection accuracy of can be further improved.

Next, the flow shifts to S3204, where the position bC of the binary image bG corresponding to the representative coordinate value C of the whole image G is set to the initial position.

It is determined whether or not the pixel at the set position is a black pixel S3
205, and if it is determined that the set position is a black pixel, S3205
YES, the process proceeds to S3206. If the set position is not determined to be a black pixel (in this case, a white pixel) S3205
As NO, shift the set position vertically and horizontally by one pixel,
The determination step S3205 is performed again to determine whether the set position is a black pixel, and the processing is performed until the set position becomes a black pixel.

In step S3206, a connected component that includes the black pixel is set as a candidate object. In step S3207, the geometrical shape of the candidate object is calculated, and in step S3208, the geometrical shape of the eye template to be identified is compared with the geometrical shape of the candidate object.

An example of the method of comparing the geometric shapes of the candidate object and the eye template in S3208 will be described with reference to FIG.

The binarized shape of the eye is as shown in FIG. 14 (a) if the image is a stable environment with a good light environment, but the sunlight is exposed to direct sunlight from one side. When the environment deteriorates, the shape may be as shown in FIGS. 14B and 14C.

The eye template has a width of ⅔ or more of the average value of the eye and has a curvature in a predetermined range of upward convexity, and a condition of having a concave shape on the left side of the black eye. It is set by a combination of a condition having a concave shape on the right side of the black eye. Then, in order to allow the examples of (b) and (c) of FIG. 14, the conditions of and or may be satisfied.

In step S3209, as a result of step S3208, it is determined whether the geometric shapes of the candidate object and the eye template match. If the candidate object and the eye template have the same geometrical shape, S3209 returns YES to S321.
At 0, the candidate object is determined to be an eye. If the geometrical shapes of the candidate object and the eye template do not match, in S3209 NO, it is determined in S3214 that the candidate object is not an eye.

In step S3211, the representative coordinate value C in the whole image G of the candidate object determined to be an eye is stored as the coordinate of the eye in this image frame.

In S3212, the minute image IG including the representative candidate point determined to be the eye is stored in the image memory as the eye image MGi.

[Face Orientation Detection Processing] (S5) FIG. 15 is a schematic flow chart for explaining the "face orientation detection processing" S5. First, in step S51, the outer contour of the eye is detected from the minute image that has been determined to be the eye in the "eye determination process" S32. Next, in S52, the positions of the inner and outer corners of the eye are detected. In S53, the peak position of the arc-shaped line is detected from the minute image. Then, in S54, the face direction angle θf indicating the face direction is detected from the inner corner of the eye, the outer corner of the eye, and the peak position. Note that the processing target in this description is the right eye on the image (the driver's left eye).

[Detection of Outer Edge Contour] (S51) FIG. 18A shows a minute image MGi centered on the eye position specified by the eye position specifying process. For the micro image here, the micro image MGi stored in the image memory in S3212 of the eye determination process is normally used, but depending on the situation, the size position from the whole image G stored in the image memory may be changed. The redefined micro image may be extracted and used.

The minute image of FIG. 18A is binarized so that an image smaller than the binarization threshold becomes black (density value 0) and a portion larger than the binarization threshold becomes white (density value 255). Thus, the binarized image bMGi of FIG. 18B can be obtained. The binarization threshold of the binarization process performed here may be the same as the binarization threshold used in the binarization process performed in the eye determination process.

In FIG. 18C, a black pixel having a pixel value of 0 is searched downward from the upper left of the obtained binarized image.
When the search for the bottom pixel is completed, the pixel row on the right is searched. If the black pixel found first in the pixel row and the black pixel found last in the pixel row are obtained for each pixel row, the outer peripheral contour of the eye can be obtained as shown in FIG. 18D.

FIG. 16A is a flow chart for explaining the "processing for detecting the outer peripheral contour of the eye" S51 in FIG.
First, in S5101, the “eye determination process” (S3 in FIG. 11 is performed first).
212) binarizes the minute image MGi determined to be the eye in
The binary image bMGi is saved in the image memory. Then S
At 5102, the initial position is set at the upper left of the binary image bMGi. In step S5103, the line of the binary image bMGi is searched in the Y direction, and the positions of both ends on the line of the black region are detected and stored. In S5104, it is determined whether or not all lines have been completed. If all lines have not been completed, in S5105, the line is shifted to the right (X direction) by one, the in-line position is set to the uppermost position, and the process returns to S5103. If all the lines have been completed in S5104, the detection processing of the outer peripheral contour of the eye is completed.

[Detection of the inner and outer corners of the eye] (S52) Regarding the inner and outer corners of the eye, in the case of the right eye on the image as shown in FIG. 19, the left end of the outer contour line is the inner corner and the right end is the outer corner position. Becomes When the inner corner and the outer corner of the eye are specified as the plane coordinates, the left and right ends of the lower line are used as the inner corner and the outer corner of the eye as shown in FIG. There is also a method of using the coordinates as the eye inside and outside corner positions and a method of using the coordinates between the middle points of the left and right ends of the upper line and the lower line as the inside and outside corner positions.

FIG. 16 (b) is a flow chart for explaining the "eye inner / outer eye detection processing" S52 of FIG. First, in S5201, it is determined whether the micro image to be processed is the driver's left eye or right eye. If it is the left eye in S5201,
The left end on the image of the outer peripheral contour of the eye is the inner canthus position, and the right end on the image of the outer peripheral contour of the eye is the outer canthus position. If it is the right eye in S5201, the right end on the image of the outer peripheral contour line of the eye is the inner canthus position, and the left end on the image of the outer peripheral contour line of the eye is the outer canthus position.

[Detection of peak position of arcuate line] (S5
3) When the upper eyelid is used as the arc-shaped line, the highest position of the upper line in the line of the outer peripheral contour of the eye can be set as the peak position as shown in FIG. When the face does not face the front of the camera as shown in FIG. 20, the eyes in the image are inclined as shown in FIG. 21, and the highest position in the height direction of the upper line does not necessarily become the peak position. .
In such a case as well, the peak position may be the position where the distance between the line segment connecting the inner and outer canthus position and the outer peripheral line of the eye and the upper line is the largest.

FIG. 17A is a flow chart for explaining the "peak position detection process for arcuate line" S53 of FIG. First, in step S5301, a straight line HP passing through the inner and outer canthus positions on the minute image is calculated. Then S5302
Then, the length of the perpendicular line from the arcuate line to the straight line HP is calculated. In S5303, the position of the point on the arc-shaped line where the length of the perpendicular is maximum is set as the peak position.

[Face Direction Angle θf Detection Processing] (S54) When the face is directed to the front as shown in FIGS. 12 and 19, the peak position of the arc-shaped line is almost at the center position of the eye, and the peak The distance L2 from the position to the inner corner of the eye and the distance L1 from the peak position to the outer corner of the eye are almost equal.

Next, when the face is facing left (right in the image) as shown in FIG. 20, from the peak position of the arc-shaped line of the left eye (right eye in the image) as shown in FIG. The distance L2 to the inner corner of the eye increases, and the distance L1 from the peak position to the outer corner of the eye decreases. When the relationship between L1 and L2 is represented by a parameter L2 / L1, L2 / L1 becomes 1 when the face is in front. L2 / L1 becomes gradually smaller when the face is turned to the right, and the peak position of the arc-shaped line is toward the corner of the eye when the face is facing the side of the camera (when the face is facing 90 degrees to the right). On the other hand, since the length of L2 becomes 0, L2 / L1 becomes 0.

However, in reality, after the direction of the face exceeds about 45 degrees to the right, the right eye is hidden behind the face and does not appear in the camera. Therefore, both eyes are monitored in the original process. If there is no abnormality such as strabismus in the eyes, the eyeballs move in the same way for both the left and right eyes. Therefore, when one eye cannot be seen on the whole image G which is face image data, processing is performed using the data of the other eye.

On the contrary, when the face turns to the left side, L2 / L1
Becomes gradually larger, and the peak position of the arc-shaped line is closer to the inner canthus when the face is facing directly to the camera (when the face is facing 90 degrees to the left).
2 / L1 is infinite.

A graph showing this relationship is shown in FIG.
By referring to this graph, the face direction (face direction angle) θf can be obtained from L2 / L1. Here, the direction of the face is 0 degrees when facing the front, the right direction (left direction on the image) is positive, and the left direction (right direction on the image) is negative angle.

FIG. 17B shows "face direction angle θ" of FIG.
15 is a flowchart showing "f detection processing" S54. First, in S5401, the distance L2 in the X direction from the inner corner of the eye to the peak position of the arcuate line is calculated. In S5402, the distance L1 in the X direction from the outer corner of the eye to the peak position of the arc-shaped line
To calculate. At 5403, L2 / L1 is calculated and L2 / L1 is calculated.
A map as shown in FIG. 22 is searched from L1 to obtain the direction angle θf of the face.

The face orientation detection processing illustrated here is an example, and in addition to this method, a method of detecting the face orientation from the distance between both eyes is known.

[Peak position detection processing of arcuate line]
(S6) The "peak position detection process of arcuate line" S6 of FIG.
The processing is performed in the same manner as the "peak position detection processing of arcuate line" S53 in the "face orientation detection processing" S5. When the "face direction detection process" is processed by the method described in the present embodiment, the result detected in the "face direction detection process" S5 is left in the memory and is called to generate an arc line. The peak position of can be obtained.

[Pupil position detection process I] (S7a) The first method of the "pupil position detection process" S7 of FIG. 3 will be described with reference to the flowchart of FIG. 23 (a) and FIG. 25 (a). .

As described with reference to FIG. 18D, the downward convex portion of the outer peripheral contour line of the eye is the pupil.
As shown in (a), the pupil position is the midpoint (bisecting point) of the portion where the upper line and the lower line have the widest distance when viewed from the entire eye. When detecting the pupil position as a plane coordinate, after obtaining the coordinate position of the lateral pupil position by the above method, the midpoint of the upper line and the lower line at that position is set to the coordinate of the vertical pupil position. Can be used.

First, in S71, the eye image MGi, which is a minute image centered on the eye position specified in the above "eye position specifying process" S3, is called. Next, in S72, the line of the outer peripheral contour of the eye is detected. Then, in S73,
The pupil position is the midpoint (bisecting point) of the portion with the widest distance between the upper line and the lower line of the outer peripheral contour of the eye.

When the "face orientation detection process" is processed by the method described in this embodiment, the "face orientation detection process" S5
By leaving the result detected in step 3 on the memory and calling it, the line of the outer peripheral contour can be obtained. If the face orientation detection processing is performed by a known method other than the method described in the present embodiment, "face orientation detection processing" S5
By performing the method described in the "processing for detecting the outer peripheral contour of the eye" S51, a line of the outer peripheral contour can be newly obtained.

When the pupil position is detected as plane coordinates, after the coordinate position of the pupil position in the lateral direction is obtained by the above method,
The coordinates of the pupil position in the vertical direction can be used as the midpoint between the upper line and the lower line at that position.

[Pupil Position Detection Processing II] (S7b) The second method of the "pupil position detection processing" S7 of FIG. 3 will be described with reference to the flowchart of FIG. 23 (b) and FIG. 25 (b).

In the line of the detected outer peripheral contour of the eye, the upper line is an upward convex line, and the lower line is a combination of an upward convex line and a downward convex line. Usually, it is convex on both sides of the eye and largely convex in the middle of the line. In the second method, as shown in FIG. 25 (b), two inflection points that change from an upward convex line to a downward convex line on the lower line are detected, and these detected inflection points are connected. The midpoint position of the line segment is the pupil position.

In FIG. 23B, first, similarly to the first method, the outer peripheral contour of the eye is detected in S71 and S72. Next, in S75, two inflection points that change from the upward convex line to the downward convex line on the lower side of the outer peripheral contour line of the eye are detected. Next, in S76, the midpoint position of the line segment connecting these two inflection points is set as the pupil position.

However, there are cases in which the pupil is large depending on which side, left or right, or the state of the image is not good, and either side of the inner corner of the eye or the outer corner of the eye is lost during the binarization process. In this case, the upward convex line on either the inner or outer side of the eye may disappear, and the inflection point of the disappearing upper convex line may not be extracted. In that case, the pupil position can be the midpoint position of the line segment connecting the inflection point on one side and the end on the side where the inflection point could not be extracted. When detecting the pupil position as a plane coordinate, after obtaining the coordinate position of the lateral pupil position by the above method, the midpoint of the upper line and the lower line at that position is set to the coordinate of the vertical pupil position. Can be used.

[Pupil Position Detection Processing III] (S7c) The third method of "pupil position detection processing" S7 in FIG. 3 will be described with reference to the flowchart of FIG. 24 (a) and FIG. 25 (c). .

In the third method, an ellipse component is extracted from the minute image as shown in FIG. 25 (c), and the center of the ellipse is detected as the pupil position.

In FIG. 24A, first, similarly to the first method, the outer peripheral contour of the eye is detected in S71 and S72. Next, in S77, an elliptical component is extracted from the outer peripheral contour of the eye.
In S78, the center of the ellipse is set as the pupil position. The method of extracting the ellipse component in S77 includes region division by Hough transform and image processing, approximation to the elliptic equation by the least squares method, etc. Sensing Symposium Proceedings P347 Pupil Extraction Method).

[Eyeball Rotation Angle Detection Process I] (S8) Next, the "eyeball rotation angle detection process I" S8 of FIG. 3 will be described. In this eyeball rotation angle detection processing, the eyeball rotation angle is calculated using the peak position obtained in "arc line peak position detection processing" S6 and the pupil position obtained in "pupil position detection processing" S7. θp is detected.

Here, the distance from the peak position of the arcuate line to the center of the pupil is L3. When the pupil faces the front direction of the face as shown in FIG. 19, L3 is 0 because the center of the pupil and the peak position of the arc-shaped line substantially coincide with each other.
When the eyeball is moved and the front direction of the face is not seen as shown in FIG. 26, the value (absolute value) of L3 becomes large as shown in FIG. Further, even when the face is not facing the front of the camera as shown in FIG. 20, when the pupil is facing the front of the face as shown in FIG. 21, the center of the pupil and the peak position of the arc substantially coincide with each other. Therefore, L3 becomes 0. When the eyeball is moved and the front direction of the face is not seen, as shown in FIG. 27 (b),
The value (absolute value) of L3 becomes large.

When the value of L3 when the eyeball is directed to the right side from the front of the face on the image is positive, the relationship between L3 and the rotation angle θp of the eyeball is as shown in FIG. 28 with L2 / L1 as a parameter. . Therefore, by referring to the map as shown in FIG. 28 from L3 and L2 / L1 previously obtained in the “detection process of the face direction angle θf” S54, the eye rotation angle θ
p can be obtained. In addition, the rotational movement range of the eyeball is a range from the front to the left and right about 43 degrees.

FIG. 24B is a flow chart for explaining the "eyeball rotation angle detection process" S8 of FIG. First S
At 81, the distance L3 from the peak position of the arc-shaped line to the center position of the pupil is calculated. Next, referring to the map as shown in FIG. 28, the rotation angle θ of the eyeball is calculated from L2 / L1 and L3.
Find p.

[Gaze Direction Calculation Processing] (S9) Next, the “gaze direction calculation processing” S9 of FIG.
This will be described with reference to the flowchart of (c). The line-of-sight direction is the direction angle θf of the face obtained in the “face direction detection process” S5.
And the rotation angle θp of the eyeball obtained in “Eye rotation angle detection processing I” S8 are combined. Therefore, in S91, the rotation angle θp of the eyeball and the direction angle θf of the face are added to obtain the line-of-sight direction. To do.

Next, the effect of the first embodiment will be described.

(A) The line-of-sight direction can be detected because the line-of-sight direction is detected not only by the direction of the eyeball but by the direction of the face and the angle of rotation of the eyeball. Can be further improved.

(B) As the eyeball rotation angle detecting means, arc-shaped line peak position detecting means for detecting an arc-shaped line of the face image and detecting the peak position thereof, and pupil position detecting means for detecting the position of the pupil are provided. Since the eyeball rotation angle is detected from the distance between the pupil position and the peak position, it becomes possible to detect the line-of-sight angle based on the pupil that is the basis of the line-of-sight position.
The detection accuracy can be further improved. Further, since the center position of the eye is detected from the arc-shaped line peak position, it is possible to capture the features of the human face and perform more accurate detection.

(C) When the eye position is specified from the face image, the eye candidate position specifying process for specifying the candidate position of the eye from the face image is performed, and whether or not it is the eye from the micro image centering on this part Since it is determined that the calculation processing amount can be reduced, the processing can be performed quickly.

(D) When detecting the inner and outer corners of the eye, the contour of the eye is detected, and the inner and outer corners of the eye are specified from the contour. Therefore, accurate detection can be performed according to the shape of the eye. Therefore, the accuracy of detecting the line-of-sight direction can be further improved.

(E) Since the center of the widest position in the vertical direction of the contour of the eye is the center of the pupil when detecting the pupil position, accurate detection of the pupil position should be performed according to the characteristics of the human eye. Therefore, it is possible to further improve the gaze direction detection accuracy.

(V) When detecting the pupil position, two inflection points on the lower line of the eye contour are detected, and the center position is set as the center position of the pupil. Therefore, the inflection point is simply determined from the eye contour. In addition, the pupil position can be accurately detected in consideration of the black eye element, and the gaze direction detection accuracy can be further improved.

(G) When detecting the pupil position, since the pupil position is determined from the center position of the elliptical object in the minute image, the eye position can be accurately estimated, and the eye-gaze direction detection accuracy can be improved. Can be further improved.

(H) In detecting the peak position of the arcuate line, since the position where the height of the arcuate line is the highest is the peak position, it is possible to accurately detect the eye position according to the characteristics of the human body. The gaze direction detection accuracy can be further improved.

(B) In detecting the peak position of the arc-shaped line, the peak position is the position where the length of the perpendicular line from the arc-shaped line to the line segment connecting both ends of the arc-shaped line is the longest. This can be performed accurately, and the gaze direction detection accuracy can be further improved.

(V) Since any of the upper and lower eyelids, the upper and lower eyeglass frames, and the eyebrows is used as the face part for detecting the arcuate line, it is possible to use a characteristic item as a human body or a wearable item. Can be reliably detected, and the gaze direction detection accuracy can be further improved.

(L) In the pupil diameter detection, an elliptical object is detected from a minute image, and the horizontal and vertical diameters of this elliptical object are regarded as the horizontal and vertical diameters of the pupil. There is no need to measure, and detection can be performed with a simpler configuration.

[Second Embodiment] FIG. 29 is a block diagram for explaining the configuration of the second embodiment of the gaze direction detecting device according to the present invention. In the figure, the line-of-sight direction detecting device includes a face image capturing unit CL1 that captures a face image, and an eye position identifying unit CL2 that identifies the position of an eye on the image from the face image captured by the face image capturing unit CL1. Eye rotation angle detection means CL4 for detecting the rotation angle of the eyeball from the face image captured by the face image capturing means CL1, and face image capturing means C
Face orientation detecting means CL3 for detecting the orientation of the face from the face image captured by L1, and eyeball rotation angle detecting means CL4.
Rotation angle of eyeball and face orientation detection means C detected by
And a line-of-sight direction calculating means CL5 for calculating the line-of-sight direction from the face direction detected by L3.

The face image pickup means CL1 is a TV for picking up a face of a driver or an operator and outputting face image data.
It is an imaging means such as a camera. The eye position specifying means CL2 is
The face image data output from the face image capturing means CL1 is processed to identify the position of the eye on the image.

The face orientation detecting means CL3 processes the face image data output from the face image capturing means CL1 to detect the face orientation with the optical axis line of the face image capturing means CL1 as the front.

The eyeball rotation angle detecting means CL4 is further provided with a pupil position detecting means CL41, a peak position detecting means CL42 of an arcuate line, and a pupil diameter detecting means CL43.

The pupil position detecting means CL41 detects the position of the pupil by image-processing the micro image centered on the eye position specified by the eye position specifying means CL2.

Peak position detecting means CL4 for arcuate line
Reference numeral 2 detects an arc-shaped line that serves as a reference line of sight such as the upper eyelid, the lower eyelid, the eyebrow, the upper frame of the spectacles, the lower frame of the spectacles, and obtains the peak position of the arc-shaped line from them. In the present embodiment, only the case of the upper eyelid will be described, but other parts can be detected by performing similar processing.

The pupil diameter detecting means CL43 is the eye position specifying means C.
The lateral and longitudinal diameters of the pupil on the image are detected by performing image processing on the minute image centered on the eye position specified by L2.

The eyeball rotation angle detecting means CL4 determines whether the position of the pupil detected by the pupil position detecting means CL41 or the peak position of the arc-shaped line of the arc-shaped line peak position detecting means CL42 is the left or right of the pupil relative to the peak position. It is detected whether or not it is on the side, and the position with respect to the peak position of the pupil and the pupil diameter detection means CL
The rotation angle of the eyeball is detected from the horizontal and vertical diameters of the pupil on the image detected at 43.

[Arrangement of Equipment] The arrangement of equipment according to the present invention and the TV camera 1 as the face image pickup means CL1 are the same as those in the first embodiment, and therefore the description thereof will be omitted.

The microcomputer 2 is the TV camera 1
An image memory for storing the image data input from is provided. Further, the microcomputer 2 processes the image data to detect the line-of-sight direction, the eye position specifying means CL2, the face direction detecting means CL3, the pupil position detecting means CL41, and the arc-shaped line peak position detection. Means C
Logic for implementing L42, pupil diameter detection means CL43, and line-of-sight direction calculation means CL5 is programmed.

[Processing of Entire System] FIG. 30 is a schematic flowchart for explaining the flow of the entire processing of the system. First, when the process is started, the process of "initial value input" is executed in S1, and the work area of the microcomputer is initialized and the initial values of various parameters are set.

In the "face image pickup process" in S2, the TV camera 1 picks up a face image and inputs it to the microcomputer 2 as image data.

In the "eye position specifying process" of S3, the position of the eye on the face image is specified by performing image processing on the image data input to the microcomputer 2.

In S4, if the eyes can be identified on the face image based on the output result of S3, the process proceeds to S4 YES and S5. If the eyes cannot be identified on the face image, the process returns to S4 NO and S2 to identify the eyes. The processes from S2 to S4 are repeated until it is possible.

In the "arc-shaped line peak position detection processing" in S6, a minute image (also called a minute area) which is a partial image of the face image centered on the eye position specified in S3 is subjected to image processing to obtain a minute image. The arc line and its peak position on the image are detected.

In the "pupil position detection process" of S7, the position of the pupil on the image is detected by image-processing the micro area centered on the eye position specified in S3.

In the "face orientation detection process" of S5, the orientation of the face on the image with respect to the optical axis direction of the face image capturing means is detected by image processing the image data input to the microcomputer 2.

In the "pupil diameter detection process" of S10, the lateral and vertical diameters of the pupil on the image are detected by image-processing the micro area centered on the eye position specified in S3.

In the "eyeball rotation angle detection process II" of S11, the rotation angle θp of the eyeball is detected from the horizontal and vertical diameters of the pupil on the image detected in S10.

In the "line-of-sight direction calculation process" in S9, S5
The line-of-sight direction is calculated from the face orientation detected in S11 and the rotation angle of the eyeball detected in S11, and the process ends.

[Eye Position Specifying Process] (S3) Since it is the same as the “eye position specifying process” S3 of the first embodiment, description thereof will be omitted.

[Pupil Position Detection Processing] (S7) Since it is the same as the "pupil position detection processing" S7 of the first embodiment, description thereof will be omitted.

[Peak position detection processing of arcuate line]
(S6) "Peak position detection process of arcuate line" of the first embodiment
Since it is the same as S6 and "Peak position detection process of arcuate line" in "Face orientation detection process" S53, description thereof will be omitted.

[Face Orientation Detecting Process] (S5) Since it is the same as the "face direction detecting process" S5 of the first embodiment, description thereof will be omitted.

[Pupil Diameter Detection Processing] (S10) The pupil diameter detection processing method will be described. First, the ellipse component is extracted from the minute image. When the third method is used in the pupil position detection process, the detection result at that time is stored in the memory, and the stored result is called to simplify the process. Third in the pupil position detection process
When a method other than the above method is used, the ellipse component is subjected to the micro image by the Hough transform, the area division by the image processing, the approximation to the elliptic equation by the least square method, or the like as in the third method in the pupil position detection processing. Extract from inside. Next, the vertical and horizontal diameters of the extracted ellipse component are the vertical and horizontal diameters of the pupil.

FIG. 31A is a flow chart for explaining the pupil diameter detection processing. First, in S71, the eye image MGi is called. Next, in S72, the outer peripheral contour of the eye is detected. S
At 77, an ellipse component is extracted from the outer peripheral contour of the eye. In S78, the center of the ellipse is set as the pupil position. In S101, the vertical diameter of the ellipse,
The length of the horizontal diameter is obtained, and the vertical diameter of the ellipse is the vertical diameter of the pupil, and the horizontal diameter of the ellipse is the horizontal diameter of the pupil.

[Eye Rotation Angle Detection Processing II] (S11) FIGS. 32 (a) and 32 (b) respectively show a face as shown in FIG. 20 facing the right side in the direction from the front of the optical axis of the camera. 3 shows a case where the pupil is looking at the front of the face and a case where the pupil is looking leftward on the image with respect to the front of the face.

As can be seen from this figure, the lateral diameter is widest when the pupil is in front of the camera, becomes smaller as the distance from the camera front to the left and right, and the shape of the pupil on the image becomes vertically long. . This tendency becomes larger as the face is facing the front of the camera optical axis than when the face is looking at the front of the camera. When the face is oriented to the left and right with respect to the front of the optical axis of the camera to some extent, the horizontal diameter on the pupil image becomes narrower as the eyeball is moved in the same direction as the face, and finally, It disappears from the view of the camera.

If the horizontal diameter of the pupil is LH and the vertical diameter of the pupil is LV, and the shape of the pupil on the image is represented by the vertical ratio LV / LV, then the vertical ratio LV.
A relationship as shown in FIG. 33 is obtained between / LH and the rotation angle θp of the eyeball. Thereby, the rotation angle θp of the eyeball can be obtained from the aspect ratio LV / LH of the position of the pupil with respect to the peak position of the eye and the shape of the pupil on the image.

FIG. 31B shows "Eyeball rotation angle detection processing".
II ”S11 is a detailed flowchart for explaining S11. First, in step S111, the aspect ratio L is calculated from the pupil vertical diameter LV and the pupil lateral diameter LH.
Calculate V / LH. Next, in S112, the aspect ratio LV / L
3 from H and L2 / L1 obtained by the face orientation detection process.
The rotation angle θp of the eyeball is obtained with reference to a map such as 3.

[Line-of-sight direction calculation process] (S9) Since it is the same as the "line-of-sight direction calculation process" S9 of the first embodiment, description thereof will be omitted.

Next, the effect of the second embodiment will be described.

As the eyeball rotation angle detecting means, a pupil diameter detecting means for detecting the lateral diameter and the longitudinal diameter of the pupil is provided, and the rotational angle of the eyeball is detected based on the ratio between the lateral diameter and the longitudinal diameter of the pupil. The change in the line of sight can be detected faithfully with the movement of the pupil, and the accuracy can be further improved in this respect.

[Third Embodiment] [System Block Diagram] FIG. 34 is a block diagram illustrating the configuration of a third embodiment of the gaze direction detecting device according to the present invention. In the figure, the line-of-sight direction detecting device includes a face image capturing means CL1 for capturing a face image and a face image capturing means CL
Eye position specifying means CL2 for specifying the position of the eye on the image from the face image picked up by No. 1 and face image pick-up means CL1
First eyeball rotation angle detection means CL4A for detecting the rotation angle of the eyeball from the face image captured by the, and second eyeball rotation angle for detecting the rotation angle of the eyeball from the face image captured by the face image capturing means CL1. Detecting means CL4B, face orientation detecting means CL3 for detecting the orientation of the face from the face image captured by the face image capturing means CL1, eyeball rotation angle detected by the eyeball rotation angle detecting means CL4A or CL4B, and face detection. The line-of-sight direction calculation unit CL5 calculates the line-of-sight direction from the face direction detected by the direction detection unit CL3.

The face image pickup means CL1 is a TV for picking up a face of a driver or an operator and outputting face image data.
It is an imaging means such as a camera. The eye position specifying means CL2 is
The face image data output from the face image capturing means CL1 is processed to identify the position of the eye on the image.

The face orientation detecting means CL3 processes the face image data output from the face image capturing means CL1 to detect the face orientation with the optical axis line of the face image capturing means CL1 as the front.

The first eyeball rotation angle detecting means CL4A comprises a pupil position detecting means CL41 and an arc line peak position detecting means CL42.

The second eyeball rotation angle detecting means CL4B is composed of a pupil position detecting means CL41 and an inner / outer eye / extremity position detecting means CL.
And 44.

In other words, the first and second eyeball rotation angle detecting means CL4A and CL4B are the pupil position detecting means CL4.
Share one.

Incidentally, the first and second eyeball rotation angle detecting means C
Which of L4A and CL4B is used depends on the detection result of the face orientation detection means CL3, and the second eyeball rotation angle detection means CL4B is used if the face is facing the front, and the first eye rotation angle detecting means CL4B is used if the face is not facing the front. The eyeball rotation angle detecting means CL4A is used.

The pupil position detecting means CL41 detects the position of the pupil by image-processing the micro image centered on the eye position specified by the eye position specifying means CL2.

Peak position detecting means CL4 of arcuate line
Reference numeral 2 detects an arc-shaped line that serves as a reference line of sight such as the upper eyelid, the lower eyelid, the eyebrow, the upper frame of the spectacles, the lower frame of the spectacles, and obtains the peak position of the arc-shaped line from them. In the present embodiment, only the case of the upper eyelid will be described, but other portions can be detected by performing similar processing.

The eye inside and outside corner position detecting means CL44 detects the outside of the eye and the inside of the eye by performing image processing on a minute image centered on the eye position specified by the eye position specifying means CL2.

[Arrangement of Equipment] The arrangement of equipment of the present invention and the TV camera 1 as the face image pickup means CL1 are the same as those in the first embodiment, and therefore the description thereof will be omitted.

The microcomputer 2 is the TV camera 1
An image memory for storing the image data input from is provided. Further, in the microcomputer 2, the eye position specifying means CL2, the face direction detecting means CL3, the pupil position detecting means CL41, the arc line peak position detecting means CL42, the inner and outer canthus and outer corner position detecting means CL44, and the line of sight. Logic for implementing the direction calculation means CL5 is programmed.

[Processing of Entire System] FIG. 35 is a schematic flowchart for explaining the flow of the entire processing of the system. First, when the process is started, the process of "initial value input" is executed in S1, and the work area of the microcomputer is initialized and the initial values of various parameters are set.

In the "face image pickup process" in S2, the TV camera 1 picks up a face image and inputs it to the microcomputer 2 as image data.

In the "eye position specifying process" of S3, the position of the eye on the face image is specified by performing image processing on the image data input to the microcomputer 2.

In S4, if the eyes can be specified on the face image based on the output result of S3, the process proceeds to S4 YES and S5. If the eyes cannot be specified on the face image, the process returns to S4 NO and S2 to specify the eyes. The processes from S2 to S4 are repeated until it is possible.

In the "face orientation detection process" in S5, the orientation of the face on the image with respect to the optical axis direction of the face image capturing means is detected by image processing the image data input to the microcomputer 2.

In the "pupil position detection process" of S7, the position of the pupil on the image is detected by image-processing the micro area centered on the eye position specified in S3.

In S12, it is determined based on the result of S5 whether or not the face is facing the front of the image. If the face is facing the front of the image, the process proceeds to S12 YES, and if the face is not facing the front of the image, the process proceeds to S12 NO.
Advances the processing to S6.

In S13, "inner eye / extremity position detection processing", the inner eye area / extremity position on the image is detected by image-processing the micro area centered on the eye position specified in S3.

In the "eyeball rotation angle detection process" in S14,
The position of the pupil detected in S7 and the corner of the eye detected in S13
The rotation angle (= viewing direction) of the eyeball is calculated from the eye inside corner position, and the process ends.

In the "arc-shaped line peak position detection process" in S6, the peak position of the arc-shaped line on the image is detected by image-processing the micro area centered on the eye position specified in S3.

In the "eyeball rotation angle detection process I" in S8,
The rotation angle θp of the eyeball is detected from the peak position of the arc-shaped line detected in S6 and the position of the pupil detected in S7.

In the "line-of-sight direction calculation process" in S9, S5
The process is terminated by setting the sum of the face direction θf detected in step S8 and the rotation angle θp of the eyeball detected in step S8 as the line-of-sight direction.

[Eye Position Specifying Process] (S3) Since it is the same as the “eye position specifying process” S3 of the first embodiment, description thereof will be omitted.

[Face Orientation Detecting Process] (S5) Since it is the same as the "face direction detecting process" S5 of the first embodiment, a description thereof will be omitted.

[Pupil Position Detection Processing] (S7) Since it is the same as the “pupil position detection processing” S7 of the first embodiment, description thereof will be omitted.

[Inner / Extreme Eye Position Detection Processing] (S13) Since it is the same as the “detection of inner / outer eye corners” S52 of the first embodiment, description thereof will be omitted.

[Eyeball Rotation Angle (= Gaze Direction) Detection Processing] (S14) FIG. 36 is a plan view schematically showing the horizontal rotational movement of the eyeball. The center of rotation of the eyeball is approximately 13.6 mm posterior to the apex of the cornea, and the range of movement is approximately 43 degrees to the left and right from the front. From this, when the pupils are located at the left and right end positions, it is possible to use the 12.
It is moving 6 mm.

As shown in FIGS. 37 (a) and 37 (b), when the distance from the inner canthus position to the pupil position is LS and the distance from the pupil position to the outer canthus position is LE, the rotation angle θp of the eyeball.
Then,

## EQU2 ## θp = tan ⁻¹ {12.6 / 13.6 * (LS + LE) (LS-LE)} (2) The rotation angle θp of the eyeball can be calculated by the equation (2). In this case, since the face is facing forward, the rotation angle θ of this eyeball is
p is the line-of-sight direction.

[Peak position detection processing of arcuate line]
(S6) Since it is the same as the [peak position detection process of arcuate line] of the first embodiment, description thereof will be omitted.

[Eyeball Rotation Angle Detection Process I] (S8) Since it is the same as the "eyeball rotation angle detection process I" S8 of the first embodiment, description thereof will be omitted.

[Line-of-sight direction calculation process] (S9) Since it is the same as the "line-of-sight direction calculation process" S9 of the first embodiment, description thereof will be omitted.

Next, the effect of the third embodiment will be described.

(A) In the face direction detection, an arcuate line detecting means for detecting the peak position of the arcuate line from the face image and an eye inside / outside corner position detecting means for detecting the inner canthus and outer corner positions of the face image are provided. In addition, since the ratio of the distance from the peak position of the inner corner of the eye to the peak position of the inner corner of the eye is calculated, the accuracy can be improved more than the amount of information that is captured.

(B) In the eyeball rotation angle detection, the inner and outer corners of the eye and the outer corner of the eye for detecting the inner and outer corners of the eye on the image, and the pupil position detecting means for detecting the position of the pupil on the image. When the captured image of the face is facing the front, the rotation angle of the eyeball is detected from the inner canthus position, the outer canthus position, and the pupil position on the image, and the rotation angle of this eyeball is set as the line-of-sight direction. It is possible to accurately detect the inner canthus position and the outer canthus position, thereby further improving the detection accuracy in the line-of-sight direction.

[Brief description of drawings]

FIG. 1 is a block diagram illustrating a configuration of a first embodiment of a gaze direction detecting device according to the present invention.

FIG. 2 is a layout diagram when the line-of-sight direction detection device according to the present invention is applied to a driver of a vehicle.

FIG. 3 is a schematic flowchart showing an entire gaze direction detection process in the first embodiment.

FIG. 4 is a detailed flowchart showing eye position specifying processing.

FIG. 5 is a detailed flowchart showing a process of identifying positions of eye candidates.

FIG. 6 is an explanatory diagram of eye position detection using an eye template.

FIG. 7 is an explanatory diagram showing the size of a region where an eye is present.

FIG. 8 is a statistical diagram of the lateral length of the eye.

FIG. 9 is a statistical diagram of the length of the eye in the vertical direction.

FIG. 10 is an explanatory diagram for deciding the position of an eye existence region.

FIG. 11 is a detailed flowchart showing an eye determination process.

FIG. 12 is an explanatory diagram of minute image IG extraction.

FIG. 13 is an explanatory diagram of how to obtain a binarization threshold.

FIG. 14 is an explanatory diagram of eye position detection using an eye template.

FIG. 15 is a flowchart showing face orientation detection processing.

FIG. 16A is a detailed flowchart showing a process of detecting an outer peripheral contour of an eye. (B) It is a detailed flowchart showing the process of detecting the inner and outer corners of the eye.

FIG. 17 (a) is a detailed flowchart showing a peak position detection process for an arc line. 6B is a detailed flowchart showing a process of detecting a direction angle θf of a face.

FIG. 18 is a schematic explanatory diagram of a method of extracting a contour line of an eye by image processing.

FIG. 19 is an explanatory diagram showing the mutual positional relationship between the inner and outer corners of the eye, the peak positions of arc-shaped lines, and the pupil position.

FIG. 20 is an explanatory diagram when a face faces the right side on the image.

FIG. 21 is an explanatory diagram of the mutual relationship among the peak position of the arc-shaped line, the inner and outer corners of the eyes, and the positions of the pupils when the face faces the right side of the image.

FIG. 22 is a diagram showing the relationship between the face orientation θf and the ratio L2 / L1 of the distance between the peak position of the arcuate line and the inner and outer corners of the eye.

23 (a) is a detailed flowchart showing a pupil position detection process I. FIG. 7B is a detailed flowchart showing a pupil position detection process II.

FIG. 24 (a) is a detailed flowchart showing a pupil position detection process III. (B) It is a detailed flowchart which shows the rotation angle detection process I of an eyeball. (C) It is a detailed flowchart showing the calculation process of the line-of-sight direction.

FIG. 25 is an explanatory diagram of a method of obtaining the center position of the pupil.

FIG. 26 is an explanatory diagram of a face image when an eyeball is moved.

FIG. 27 is a diagram illustrating changes in the peak positions of arc-shaped lines on a minute image when the eyeball is moved.

FIG. 28 is a diagram showing the relationship between the distance between the peak position of the arcuate line and the position of the pupil and the rotation angle θp of the eyeball.

FIG. 29 is a block diagram illustrating a configuration of a second embodiment of a visual line direction detection device according to the present invention.

FIG. 30 is a schematic flowchart showing an entire gaze direction detection process in the second embodiment.

FIG. 31 (a) is a detailed flowchart showing a pupil diameter detection process. 6B is a detailed flowchart showing the eyeball rotation angle detection processing II.

FIG. 32 is an explanatory diagram for obtaining a pupil diameter.

FIG. 33 is a diagram showing a relationship between the ratio LV / LH of the pupil diameter to the pupil diameter and the rotation angle θp of the eyeball.

FIG. 34 is a block diagram illustrating a configuration of a third embodiment of a gaze direction detecting device according to the present invention.

FIG. 35 is a schematic flowchart showing an entire gaze direction detecting process in the third embodiment.

FIG. 36 is a plan view for explaining left and right movements of the eyeball.

FIG. 37 is an explanatory diagram of the pupil position and the outer and inner corners of the eye.

[Explanation of symbols]

CL1 ... Face image capturing means CL2 ... Eye position specifying means CL3 ... Face orientation detecting means CL4 ... Eyeball rotation angle detection means CL41 ... Pupil position detecting means CL42 ... Arc line peak position detecting means CL5 ... Gaze direction calculating means

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Hiroshi Ueno             Nissan, Takaracho, Kanagawa-ku, Yokohama-shi, Kanagawa Nissan             Inside the automobile corporation F term (reference) 5B057 AA16 BA02 CC01 CE12 DA07                       DA08 DA20 DB02 DB05 DB09                       DC08 DC16                 5L096 AA06 CA02 DA02 EA06 EA27                       EA43 FA04 FA06 FA08 FA12                       FA13 FA14 FA32 FA36 FA62                       FA64 FA66 FA67 FA69 GA02                       GA51 JA09

Claims (15)

[Claims] 1. A position of an eye is specified from a face image picked up by a face image pick-up means, a rotation angle of an eyeball is detected from the image specified by the eye position specifying means, and an image is picked up by the face image pick-up means. A line-of-sight direction detecting device which detects the direction of the face from the face image and detects the direction of the line of sight from the eyeball rotation angle and the direction of the face. 2. A face image capturing means for capturing a face image, an eye position identifying means for identifying an eye position on the image from the face image captured by the face image capturing means, and an image captured by the face image capturing means. An eyeball rotation angle detecting means for detecting an eyeball rotation angle from the face image and the specified eye position; a face orientation detecting means for detecting a face orientation from the face image captured by the face image capturing means; A line-of-sight direction detection device comprising: a line-of-sight direction calculation unit that calculates a line-of-sight direction from the rotation angle of the eyeball detected by the rotation angle detection unit and the face direction detected by the face direction detection unit. 3. The eyeball rotation angle detecting means processes a minute image which is a partial image of the face image centered on the position specified by the eye position specifying means,
Arc-shaped line peak position detection means for finding an upward convex or downward convex arc-shaped line in the minute image and detecting an upward convex or downward convex peak position on the image of the arc-shaped line; And a pupil position detecting means for detecting the inner canthus position on the image and the position of the pupil on the image, wherein the peak position detected by the arc-shaped line peak position detecting means and the pupil position detecting means are detected. Based on the position of the pupil, the distance between the pupil position and the peak position is calculated,
The gaze direction detecting device according to claim 2, wherein the rotation angle of the eyeball is detected from the distance. 4. The eyeball rotation angle detecting means includes a pupil diameter detecting means for detecting a lateral diameter and a longitudinal diameter of a pupil on the image in the micro image, and an image on the image detected by the pupil diameter detecting means. The eye gaze direction detecting device according to claim 2, wherein the rotation angle of the eyeball is detected based on the ratio of the horizontal diameter to the vertical diameter of the pupil. 5. The face orientation detecting means processes a micro image which is a partial image of the face image centered on the position specified by the eye position specifying means, thereby upwardly convexing in the micro image. Or, an arc line peak position detecting means for finding a downward convex arc line and detecting an upward convex or downward convex peak position on the image of the arc line, and an inner canthus position on the image in the minute image And an inner and outer corner of the eye position detecting means for detecting the outer corner of the eye on the image, the peak position detected by the arc-shaped line peak position detecting means, and the inner corner of the eye detected by the inner corner of the eye and outer corner position detecting means, and The distance from the inner canthus position to the peak position and the distance from the outer canthus position to the peak position are respectively calculated based on the outer canthus position, and the calculated distance from the inner canthus position to the peak position. Gaze direction detecting apparatus according to claim 2, characterized in that to calculate the orientation of the face from the ratio of the distance from the corner position to the peak position. 6. The eyeball rotation angle detecting means processes an image in the minute image by processing a minute image which is a partial image of the face image centered on the position specified by the eye position specifying means. An eye inside / outside corner position detecting means for detecting an upper inside corner position and an outside corner area on the image; and a pupil position detecting means for detecting the inside corner area on the image and the position of the pupil on the image in the minute image, When the face orientation detected by the face orientation detecting means is in the front direction of the optical axis of the face image capturing means, the eyeball rotation angle detecting means detects the inner or inner corner of the eye / extremity position detecting means. The rotation angle of the eyeball is detected from the position of the inner and outer corners of the eye, the position of the outer corner of the eye, and the pupil position on the image detected by the pupil position detection means, and the gaze direction calculation means is the rotation angle of the eyeball detected by the eyeball rotation angle detection means. To be the line-of-sight direction Gaze direction detecting apparatus according to claim 2, characterized. 7. The eye position specifying means specifies the position of an eye candidate from the face image captured by the face image capturing means, and specifies the position of the eye candidate. An eye determination process of determining whether or not the position of the eye candidate is an eye by performing image processing on a micro image centered on the position of the eye candidate specified by the process, The line-of-sight direction detection device according to claim 2. 8. The eye inside and outside corner position detecting means detects the outline of the eye by performing image processing on the minute image, and sets both end positions of the detected outline of the eye as the inside and outside eye position. The line-of-sight direction detection device according to claim 5. 9. The pupil position detecting means detects the contour of the eye by performing image processing on the minute image, and sets the center of the widest position of the detected contour of the eye in the vertical direction as the center position of the pupil. The gaze direction detecting device according to claim 3. 10. The pupil position detecting means detects an eye contour by performing image processing on the minute image, and a change in the lower line of the detected eye contour is changed from upward convex to downward convex. The line-of-sight direction detecting device according to claim 3, wherein two inflection points are detected, and a midpoint position of the detected two inflection points is set as a center position of the pupil. 11. The pupil position detecting means extracts an elliptical object from the minute image by performing image processing on the minute image, and sets the center position of the elliptical object as the center position of the pupil. The line-of-sight direction detection device according to claim 3. 12. The arc-shaped line peak position detecting means sets a position where the height of the arc-shaped line is highest as the arc-shaped line peak position.
The line-of-sight direction detection device according to. 13. The arc-shaped line peak position detecting means has a longest vertical line descended from a point on the arc-shaped line with respect to a line segment connecting both end points of the arc-shaped line. Is the arc-shaped line peak position, and the line-of-sight direction detection device according to claim 3 or claim 5 is characterized in that. 14. The arc-shaped line peak position detection means sets the upper eyelid, the lower eyelid, the upper side of the spectacle frame, the lower side of the spectacle frame, or the eyebrow as the face part detected as the arc-shaped line. The line-of-sight direction detection device according to claim 3 or 5. 15. The pupil diameter detecting means extracts an elliptical object from the minute image by performing image processing on the minute image, and determines a lateral diameter and a longitudinal diameter of the elliptical object of a pupil on the image. The line-of-sight direction detection device according to claim 4, wherein the horizontal diameter and the vertical diameter are respectively set.