JPH11183127A - Eye position detector - Google Patents

Abstract

PROBLEM TO BE SOLVED: To keep precision in eye position detection even when a curtain rail, etc., is present on a background of a person whose eye position is to be detected. SOLUTION: In the detector, there are provided with an image input means CL1 for inputting an image data comprising a face, a point extraction means CL3 wherein pixel concentration is read along a vertical pixel column of an inputted image data and the pixel of peak in one direction of a concentration value is specified to be an extraction point under the condition where a differential value of concentration change before peak, in one direction, of the concentration value on the pixel column exceeds a given value, and a curved line data extraction means CL4 wherein close extraction points are connected in pixel column direction of adjoining pixel column to extract a curved line data extending in lateral direction of a face, movement state judging means CL8 for judging movement state of the curved line data, and an eye position detecting means CL11 for detecting eye position, based on a curved lien data when the curved line data is judged to be dynamic.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an eye which can be used for automatically controlling a mirror and a seat, and for specifying an eye position necessary for detecting a drowsy driving state from a driver's eye open / closed state. It relates to a position detecting device.

[0002]

2. Description of the Related Art Various types of processing are performed on image data of a face to extract a feature amount of the face, thereby identifying a person, detecting the direction of a line of sight, and judging a drowsiness / wakefulness. Equipment is being developed. Since image data is a huge amount of data expressing the brightness of each pixel in a number of gradation levels (density values, luminances), such a device narrows down the image data early to reduce the number of data, The processing speed is secured by saving the number of operations.

For example, in the dozing alarm device disclosed in Japanese Patent Application Laid-Open No. Hei 7-181012, an eye opening index, which is one of the feature values of a face, is obtained from face image data. Binarize the density value of image data in black and white, set a black mask area in unnecessary parts of the screen to reduce the number of black parts, label each island-like black part as a unit, etc. Reduces the number of data.

[0004]

Problems to be Solved by the Invention Japanese Patent Laid-Open No. 7-18101
In the processing disclosed in Japanese Patent Laid-Open No. 2 (1994), the following three steps need to be performed in order to detect the position of the eye, so that considerable time is required for image processing and calculation.

[0005] (1) Preprocessing of image data mainly on binarization of density values and noise removal (2) Face width detection (labeling, particle removal, white pixel (3) Vertical eye position detection (judgment of barycentric coordinates, area value and fillet diameter of each label)

That is, although the density value is binarized, each of the island-shaped black portions includes several hundreds to several tens of thousands of pixels, so that each of these processes requires one operation. Takes a considerable amount of time. Therefore, it is difficult to shorten the processing cycle to increase the follow-up performance and reliability of the determination, and it is also difficult to reduce the size of the arithmetic unit and interrupt processing of other arithmetic processing.

In addition, since labeling is performed in a state where the black portions on the face are individually separated, when a black portion corresponding to the eye is connected to another black portion, for example, when long hair is applied to the outer corner of the eye, On the black and white binarized image, the black portion corresponding to the eye is integrated with other black portions, and the black portion corresponding to the eye cannot be specified.

Accordingly, the applicant of the present application has developed an eye position detecting device that can reduce the amount of data to be calculated, increase the processing speed, and accurately specify the feature amount corresponding to the eye even when the long hair hangs on the outside of the eye. For example, Japanese Patent Application No. 8-10190
No. 4 is proposed.

The eye position detecting device detects the density of pixels along a vertical pixel row of the face, and determines one pixel for each local increase in density in the pixel row to determine an extraction point. In addition, extraction points adjacent to each other in the pixel column direction of adjacent pixel columns are connected to detect the position of the eye from curve data extending in the horizontal direction of the face.

The present invention is a further improvement of the above-mentioned eye position detecting device, and the detection accuracy of the eye position can be improved even if there is a curtain rail or sunroof which becomes noise in the background portion of the eye position detection target person. An object of the present invention is to provide an eye position detecting device that can be maintained.

[0011]

According to the first aspect of the present invention, there is provided image input means for inputting image data including a face, and reading out the density of pixels along a vertical pixel row of the input image data. Point extraction means for specifying the pixel of the one-way peak of the density value as an extraction point on the condition that the differential value of the density change before the one-way peak of the density value on the pixel row exceeds a predetermined value. And curve data extracting means for continuously extracting the curve data extending in the horizontal direction of the face at the extraction points adjacent to each other in the pixel row direction of the adjacent pixel row, and moving state determining means for determining the moving state of the curve data And eye position detecting means for detecting an eye position based on the curve data when the curve data is determined to be dynamic.

Accordingly, the curve data to be determined as the movement is narrowed down by the curve data extracting means, and the moving state of the corresponding curve data is determined by the moving state determining means.
When it is determined that the curve data is dynamic, the position of the eye can be detected by the eye position detecting means based on the curve data.

According to a second aspect of the present invention, there is provided the eye position detecting device according to the first aspect, wherein an eye position search area 1 for setting an eye position search area 1 including the curve data extracted by the curve data extracting means. 1 setting means, uppermost curve data detecting means for detecting curve data appearing at the uppermost end in the eye position search area 1, and an eye for setting an eye position search area 2 based on the uppermost curve data Location search area 2
Setting means; moving state determining means for curve data for determining the moving state of curve data appearing in the eye position search area 2; and eye moving when the target curve data is determined to be dynamic. An eye position search area 3 setting means for setting the position search area 3; and returning to the setting of the eye position search area 1 when the target curve data is determined to be static curve data. A means for reducing the eye position search area 1 to reduce the range of the eye position search area 1 vertically based on the position information of the curve data, and a relative position of the curve data appearing in the eye position search area 3 It is characterized by comprising eye position detecting means for detecting the position of the eye from the relationship.

Therefore, in addition to the operation of the first aspect of the present invention, first, the eye position search area 1 including the curve data extracted by the curve data extraction means is set by the eye position search area 1 setting means. The uppermost curve data detecting means detects the uppermost curve data appearing in the eye position search area 1, and the eye position search area 2 setting means sets the eye position search area 2 based on the uppermost curve data. I do. In this way, the curve data is narrowed down, the moving state of the curve data appearing in the eye position search area 2 is determined by the moving state determination means, and when the curve data is determined to be dynamic, the eye position search area 3 is determined. The eye position search area 3 is set by the setting means. If it is determined that the target curve data is static curve data, the setting is returned to the eye position search area 1 and the eye position search area 1 is set based on the position information of the corresponding curve data. The range of the eye position search area 1 is reduced vertically by the reducing means. The eye position detecting means detects the position of the eye from the relative positional relationship of the curve data appearing in the eye position search area 3. Therefore, the eye position can be detected by narrowing the curve data to the eye position search area 1, the eye position search area 2, and the eye position search area 3.

According to a third aspect of the present invention, in the eye position detecting apparatus according to the first or second aspect, the moving state determining means determines the moving state of the curve data based on a vertical movement amount of the curve data. It is characterized by the following.

Therefore, in addition to the operation of the first or second aspect of the present invention, it is possible to determine whether or not the curve data is dynamic based on the amount of vertical movement of the curve data by the movement state determination means. Therefore, it is possible to make a determination according to the movement of the occupant during driving.

According to a fourth aspect of the present invention, in the eye position detecting apparatus according to the third aspect, the moving state determining means determines a moving state of the curve data in a vertical position of each extraction point constituting the curve data. Is characterized by the average value of

Therefore, in addition to the effect of the third aspect of the invention, the moving state determining means can determine the moving state of the curve data based on the average value of the vertical positions of the respective extraction points constituting the curve data.

According to a fifth aspect of the present invention, there is provided the eye position detecting device according to any one of the first to fourth aspects, wherein the moving state determining means determines the moving state by extracting the extracted points forming curve data. Is performed at the uppermost position.

Therefore, in addition to the effect of any one of the first to fourth aspects of the present invention, the determination of the moving state can be made at the uppermost position of the extraction points constituting the curve data.

According to a sixth aspect of the present invention, there is provided the eye position detecting device according to any one of the first to fourth aspects, wherein the moving state determining means determines a position of the curve data appearing in the eye position search area 2. It is characterized by comprising a shape judging means for judging a shape, and a moving state judgment time changing means for changing a judgment time of the moving state judging means of the curve data based on an output result of the shape judging means.

Therefore, in addition to the effect of any one of the first to fourth aspects of the present invention, the determination time of the moving state determination means is changed by determining the shape of the curve data appearing in the eye position search area 2. For example, if the curve data is convex upward, there is a high possibility that the eye is the eye, and the determination time can be lengthened.

According to a seventh aspect of the present invention, in the eye position detecting apparatus according to any one of the first to fourth aspects, the moving state determining means determines the shape of the curve data appearing in the position search area 2. It is characterized by comprising a shape determining means for determining, and a moving state determining movement amount changing means for changing a determining movement amount of the moving state determining means of the curve data based on an output result of the shape determining means.

Therefore, in addition to the effect of any one of the first to fourth aspects of the present invention, the determination movement amount of the movement state determination means can be changed by determining the shape of the curve data. For this reason, if the curve data is convex upward, it is highly likely that the eye is the eye, and the determination movement amount can be reduced.

According to an eighth aspect of the present invention, in the eye position detecting device according to any one of the first to fourth aspects, the moving state determining means is configured to determine a position of the curve data appearing in the eye position search area 2. It is characterized by comprising a shape judging means for judging a shape, and a setting condition changing means for an area 3 for changing a setting condition of the eye position search area 3 based on an output result of the shape judging means.

Therefore, in addition to the effect of any one of the first to fourth aspects of the present invention, the setting condition of the eye position search area 3 can be changed by determining the shape of the curve data. For this reason, even when the curve data is under the spectacle frame, the curve data corresponding to the eye can be captured by changing the setting condition of the eye position search area 3.

According to a ninth aspect of the present invention, in the eye position detecting apparatus according to the second aspect, the moving state determining means is configured to detect face image data based on curve data appearing in the eye position search area 2. An eye position search area 4 setting means for setting a horizontally long eye position search area 4 so as to pass right and left, and whether or not there are two or more left and right curve data in the set eye position search area 4 Left and right data determination means for determining the position of the eye position search area 3 by specifying curve data based on the magnitude of the vertical step of the determined left and right data
And a setting condition changing means.

Therefore, in addition to the effect of the second aspect of the present invention, a horizontally long eye position search area 4 is set from the curve data appearing in the eye position search area 2. It is determined whether or not there are two or more left and right curve data in the set eye position search area 4, and the curve data is specified based on the magnitude of the vertical step of the determined left and right data, and the eye position search area is determined. 3 setting conditions can be changed. Therefore, the position of the eye can be detected by determining the symmetrical feature of the eyebrow, eye, and eyeglass frames.

According to a tenth aspect of the present invention, there is provided the eye position detecting apparatus according to the ninth aspect, wherein the left and right data determining means selects two curve data from the upper part in the eye position search area 4. Features.

Therefore, in addition to the operation of the ninth aspect, two curve data can be selected from the upper part in the eye position search area 4. Therefore, the left and right eyebrows, eyes, and glasses can be captured in the eye position search area.

According to an eleventh aspect of the present invention, there is provided the eye position detecting apparatus according to the ninth or tenth aspect, wherein the setting condition changing means of the area 3 is configured to perform the setting when the level difference between the left and right data is less than a predetermined value. The image processing apparatus further includes a shape determining unit that determines the shape of the curve data appearing in the eye position search area 4 and specifies the curve data.

Therefore, in addition to the operation of the ninth or tenth aspect, when the level difference between the left and right data is equal to or less than a predetermined value, the shape of the curve data appearing in the eye position search area 4 is determined. Can be specified. For this reason, it becomes possible to distinguish between the eyeglass frame and the eye.

According to a twelfth aspect of the present invention, in the eye position detecting apparatus according to the second, sixth, seventh, eighth, or eleventh aspect, the shape determining means determines the shape of the curve data. Is characterized in that the rightmost point and the leftmost point of each extraction point constituting the curve data are connected by a straight line, and the shape is recognized based on the position of the extraction point at the center of the curve data with respect to the straight line.

Therefore, in addition to the effect of the invention described in claim 2, claim 6, claim 7, claim 8, or claim 11, the rightmost point and the leftmost point of the extraction points are connected by a straight line, , The shape can be recognized based on the position of the extraction point at the center of the curve data, and whether the curve data is convex upward or downward can be recognized.

[0035]

According to the first aspect of the present invention, the movement of the corresponding curve data is determined, so that even if there is a curtain rail or sunroof which becomes noise in the background portion of the eye position detection target person, the eye position can be detected. The detection accuracy of the position can be maintained, and accurate detection can be performed.

According to the second aspect of the present invention, in addition to the effect of the first aspect of the present invention, by narrowing the eye position search area to three levels, more accurate determination can be performed, and a more accurate eye position can be determined. Detection can be performed.

According to the third aspect of the present invention, in addition to the effects of the first or second aspect of the present invention, the determination of the moving state can be accurately performed by using more reliable data, and more accurate eye position detection can be performed. Can do it.

According to the fourth aspect of the present invention, in addition to the effect of the third aspect of the present invention, the movement state of the curve data is determined by the average value of the vertical positions of the respective extraction points constituting the curve data, thereby providing more accurate. The determination can be performed, and more accurate eye position detection can be performed.

According to the fifth aspect of the present invention, in addition to the effects of any one of the first to fourth aspects, the determination of the moving state can be made at the uppermost position of the extraction points constituting the curve data. And accurate eye position detection can be performed.

According to the sixth aspect of the present invention, in addition to the effects of any one of the first to fourth aspects, when the possibility of an eye is high due to the shape determination of the curve data, the determination is made for a long period of time. Accurate determination can be performed, and more accurate eye position detection can be performed.

According to the seventh aspect of the invention, in addition to the effects of any one of the first to fourth aspects, when the possibility of the eye is high due to the shape determination, the moving amount of the determination is reduced, and more accurate determination is made. Can be performed, and more accurate eye position detection can be performed.

According to the eighth aspect of the present invention, in addition to the effect of any one of the first to fourth aspects, by changing the setting condition of the eye position search area 3 by judging the shape of the curve data, the curve data becomes, for example, Even under a spectacle frame or the like, more accurate eye position search area 3 can be set, and more accurate eye position detection can be performed.

According to the ninth aspect of the present invention, in addition to the effect of the second aspect of the present invention, by setting the horizontally long eye position search area 4, the characteristics of the eyebrows, eyes, and eyeglass frames which exist symmetrically to the left and right are provided. The determination can be performed, and the more accurate determination enables more accurate eye position detection.

According to the tenth aspect, in addition to the effect of the ninth aspect, by selecting two curve data from the upper part in the eye position search area 4, the selection is performed in order from the upper part, and the eye position search is performed. The determination in the area 4 can be performed more accurately, and more accurate eye position detection can be performed.

According to the eleventh aspect, in the ninth or tenth aspect,
In addition to the effects of the present invention, more accurate determination can be performed by determining the shape of the curve data in the eye position search area 4, and more accurate eye position detection can be performed.

According to the twelfth aspect of the present invention, in addition to the effects of the second, sixth, seventh, eighth, or eleventh aspect, the shape of the curve data can be accurately recognized by the minimum necessary information. The eye position can be detected more accurately in a short time.

[0047]

FIG. 1 is a functional block diagram of an eye position detecting device according to an embodiment of the present invention. The eye position detecting device includes image input means CL1, density detecting means CL2, point extracting means CL3, Curve data extraction means CL4, predetermined area 1 setting means CL5, and curve data detection means CL6
, Predetermined area 2 setting means CL7 and moving state determination means CL
8, setting means CL9 for the predetermined area 3, reduction means CL10 for the predetermined area 1, and eye position detecting means CL11.

The image input means CL1 inputs image data including the driver's face. The density detecting means CL2
Detects the density of the pixel row in the vertical direction of the image data input from the image input means CL1. The point extracting means CL3 extracts points based on the increase in the density in the pixel row and its change state. The curve data extracting means CL4 specifies the pixel of the one-way peak of the density value on the condition that the differential value of the density change before the one-way peak of the density value on the pixel row exceeds a predetermined value. As an extraction point. That is, the curve data extraction means CL4 continuously extracts the extraction points adjacent to each other in the pixel column direction,
Extract the curve data extending in the horizontal direction of the face. The setting means CL5 for the predetermined area 1 sets the eye position search area 1 including the extracted curve data. The uppermost curve data detecting means CL6 detects curve data appearing at the uppermost end in the eye position search area 1. The predetermined area 2 setting means CL7 sets the eye position search area 2 based on the uppermost curve data. The moving state determining means CL
8 determines the moving state of the curve data appearing in the eye position search area 2. The setting means CL9 of the predetermined area 3 includes:
When it is determined that the target curve data is dynamic, an eye position search area 3 is set. The predetermined area 1
When it is determined that the target curve data is static curve data, the reduction means CL10 returns to the setting of the eye position search area 1 and sets the eye position based on the corresponding curve data position information. The range of the position search area 1 is reduced vertically. The eye position detecting means CL11 detects the position of the eye from the relative positional relationship of the curve data appearing in the eye position search area 3.

FIG. 2 is a configuration block diagram according to an embodiment of the present invention. As shown in FIG. 2, the camera 21 is provided as an image input unit CL1 that is installed on the instrument and captures the face of the driver from the front, and the input image from the camera 21 is, as shown in FIG. It consists of 512 pixels in the horizontal (X) direction and 480 pixels in the vertical (Y) direction. An input image captured by the camera 21 is converted into a digital amount of input image data via an AD converter 22.
It is stored in the image memory 23. An image data operation circuit 24 is connected to the image memory 23. The image data calculation circuit 24 includes the density detecting means CL2, point extracting means CL3, curve data extracting means CL4, predetermined area 1 setting means CL5, curve data detecting means CL6,
It comprises a predetermined area 2 setting means CL7, a curve data moving state determination means CL8, a predetermined area 3 setting means CL9, and a predetermined area 1 reduction means CL10.

The image data calculation circuit 24 is connected to an eye position detection circuit 25 as eye position detection means CL11 for detecting an eye position in the eye position search area 3.

Next, the flow of operation in the above configuration will be described with reference to the flowcharts of FIGS. 3 to 5 and the explanatory diagrams of FIGS. 6 to 14.

First, in step S301, an eye position search area 1 is set. This eye position search area 1
Is an area that can cover the entire image as shown in FIG. This is to allow a change in the eye position due to a difference in physique or the like.

In step S302, pixel data in the eye search area is read out, and in step S303, eye candidate data serving as curve data (in the transition step, the curve data is referred to as continuous data). Step S
The processing contents of 302 and S303 will be described with reference to the flowchart of FIG. 5 and the explanatory diagrams of FIGS.

First, in step S501, the face of the driver is photographed by the camera, and the input image for one frame is converted by a digital signal and stored in the image memory. In step S502, point extraction processing is performed on the eye position search area as shown in FIG.
After the end of the line, move on to the processing of the next line,
It is determined whether or not point extraction has been completed for all lines in a predetermined direction. If it is determined in step S502 that point extraction has not been performed on all lines, the process proceeds to step S503.

In step S503, arithmetic operations are performed on the density values of one line in a predetermined direction, for example, from top to bottom. This processing is intended to eliminate small variations in the change in density value at the time of capturing image data, and to capture a global change in density value. FIG.
3 shows the processing result of the arithmetic averaging operation of the line data of Xa.

In step S504 of FIG.
A differential operation is performed on the arithmetic mean value that is the operation result of step 503. FIG. 7B shows the processing result. In step S505 in FIG. 5, point extraction is performed using the differential value that is the result of the operation in step S504. The method of extracting the point is a point (p1) where the differential value changes from negative to positive.
~ P3), referring to FIG. 7 (a), extract points where the graph becomes convex downward.

Next, whether or not the change in the density value (q1 to q3) until the point is reached is equal to or less than a predetermined value.
It is determined whether or not the differential value of (b) is equal to or less than a stable reference value, and Y coordinates (A1 to A3) are extracted for points having a change in density value equal to or less than a predetermined value. This process is 1
After the end of the line, the process is switched to the processing of the next line in step S506.

If it is determined in step S502 that point extraction for all lines has been completed, points as shown in FIG. 8 are extracted. That is, on the Xa line in FIG.
This means that three points 1, 1, and A3 have been extracted.

Thereafter, the flow shifts to step S507, where Y of the extraction points (A1, A2, A3,.
The coordinate values are compared, and if the Y coordinate value is within a predetermined value, the continuous data group number, continuous start line number, and continuous data number are stored as continuous data.

In this case, since the detection target is an eye, the feature amount thereof can be data that continues relatively long in the horizontal direction. Therefore, continuous data is extracted on the condition that the data continues for a predetermined value or more in the horizontal direction. I do. Therefore, step S5
As a result of processing at 07, continuous data of G1 to G6 as shown in FIG. 9 can be recognized as curve data.

After extracting the continuous data in step S303,
Move to step S304. In step S304,
The judgment value of each continuous data is read. The judgment value of each continuous data is defined as the representative X coordinate of the continuous data by calculating the median value of the continuous start line and the end line of the continuous data, or the average value of the Y coordinate value of each extraction point of the continuous data. Refers to a representative Y coordinate.

In step S305, it is determined whether or not the eye position search area 3 has been set. At the stage immediately after the start, since the eye position search area 3 has not been set, the process proceeds to step S306. In step S306, it is determined whether or not the eye position search area 2 has been set. Also in step S306, similarly to step S305, at the stage immediately after the start, since the eye position search area 2 has not been set, the process proceeds to step S307.

In step S307, step S30
Since it is determined in S306 that neither eye position search area 3 nor eye position 2 is set, continuous data is detected in eye position search area 1. In this step, as shown in FIG. 10, the uppermost continuous data appearing in the eye position search area 1 is detected. In this process, instead of suddenly patterning a large number of continuous data appearing in the entire image, the continuous data appearing at the uppermost end is first focused on, and the corresponding continuous data is used as a face feature amount (continuous eyebrow and eye data). ) So that the eye search method can be performed as easily as possible.

In step S308, as shown in FIG. 10, the eye position search area 2 is set based on the representative coordinate value of the uppermost continuous data. The eye position search area 2 repeatedly determines whether or not the corresponding continuous data is a facial feature such as eyebrows or eyes over a predetermined time, and obtains the change from the appearance state of the corresponding continuous data. Therefore, it is desirable that the number of continuous data appearing in the eye position search area 2 is one, and the extraction points at both ends of the corresponding continuous data are the appearance points of the point when the column density to be searched is set roughly. Is significantly shaken in the horizontal direction (X coordinate direction) depending on the presence / absence of, it is important to set the width of the eye position search area 2 in the horizontal direction as small as possible.

Thereafter, in step S309, a timer for measuring the predetermined time is operated to activate step S302.
Return to At this time, the area in step S302 is performed only within the eye position search area 2 as shown in FIG. Thereafter, the processing of steps S303 to S305 is performed in the same manner.

In step S306, since the eye position search area 2 has been set, the flow shifts to step S401 in FIG. In step S401, it is determined whether the timer started in step S309 has passed a predetermined time. If it is determined in step S401 that the predetermined time has not elapsed, the process proceeds to step S402.

In step S402, the maximum value of the average value of the Y coordinate values (y1 to y9) of each extraction point of the continuous data appearing for each frame as shown in FIG. 12 is updated and stored until a predetermined time is reached. . In step S403, the minimum value of the average of the Y coordinate values is updated and stored until a predetermined time is reached. If it is desired to further increase the sensitivity of the movement in updating the minimum and maximum values of the Y coordinate value, the point at the uppermost end (y2 in FIG. 12) of each extraction point of the continuous data can be used as a representative value. .

In step S404, the motion of each frame appearing in the eye position detection area 2 with the passage of time is represented by the difference between the maximum value and the minimum value of the Y coordinate value stored in step S402 and step S403. Is determined in the vertical direction. At this time, the vertical direction is used to determine the movement of the continuous data, as described in the description of the size of the eye position search area 2 in the horizontal direction (X coordinate direction). This is because the connection in the horizontal direction, in other words, the number of continuous data varies from frame to frame, and it is further disadvantageous when the column density is set to be coarse in order to increase the processing speed. If the moving amount of the continuous data does not exceed the determination moving amount in step S404, step S3
02 and the same processing is repeated.

At this time, if the uppermost continuous data as shown in FIG. 10 is a static object of a vehicle interior structure such as a curtain rail, the movement is not determined in step S404, and step S401 is performed. It is determined that the predetermined time has elapsed. At this time, the process moves to step S406,
Clear the timer. Thereafter, as shown in FIG. 13 in step S407, when setting the eye position search area 2,
An eye position search area 1 reduced in the vertical direction so as not to include the reference coordinate value A is set.

In step S408, eye position search area 1
It is determined whether or not the reduced area has become infinitely smaller in the vertical direction. When a face is present in the detection target area, it is extremely unlikely to be NO. NO in this step S408
Is a loop in the case where only a static object is detected when there is no person to be detected and dynamic continuous data cannot be detected even when the detection is performed to the lowermost end.
1, the same process is repeated after initializing the eye position search area 1.

After the eye position search area 1 is reduced in step S407, the continuous data corresponding to the uppermost end of the eye position search area 1 captures the face feature amount (the eyebrows in this example) shown in FIG. 13B. In this case, it is determined in step S404 that the change in the Y coordinate of the face movement exceeds the determination movement amount within a predetermined time after the start of the timer. At this time, step S40
Move to 5.

In step S405, an eye search area 3 including the corresponding continuous data as shown in FIG. 14 is set. The eye position search area 3 has a size that includes one eye and eyebrows. This is because, although it is known that the continuous data whose motion has been recognized in the eye position search area 2 is a facial feature, it has not been possible to specify the eyebrow, eye, or eyeglass frame. After setting the eye position search area 3 in step S405, the process returns to step S302 to detect continuous data in the area.

At this time, in step S305, since it is determined that the eye position search area 3 has been set, the flow shifts to step S310 to correspond to the eyes from the continuous data appearing in the eye position search area 3. Select continuous data. The selection method at this time is determined based on the relative positional relationship between the respective continuous data. The selection type can be configured very easily because the target of the eye is narrowed down to a certain extent, and a higher eye detection accuracy can be ensured accordingly. Then, in step S311, the eye position coordinates are output, and the eye position detection process from the entire image is completed.

Next, a method for surely capturing the eye by adding the shape determination of the continuous data will be described with reference to the flowcharts of FIGS. 15 and 16 and FIG.

The flowchart of FIG. 15 is obtained by adding step S1504 to the flowchart of FIG.

The method of determining the shape will be described using an example of continuous data appearing in the eye position search area 2. In the method of determining the shape of continuous data, as shown in FIG. 17, a continuous start point ys and a continuous end point ye are connected by a straight line, and a Y coordinate value yc on an X coordinate having an extraction point near the center is calculated. At this time, by determining the positional relationship between the extraction point and yc on the same line, it is determined whether the continuous data has an upward convex shape, a downward convex shape, or a substantially linear shape. In the case of continuous data in which a curtain rail, a sunroof, and the like are detected by this determination, the data is often almost horizontal as shown in FIG. Often becomes.

In the above-described method for determining the shape of continuous data, the shape characteristics of the object to be detected can be reliably grasped with the minimum necessary information, and the processing is simple. Therefore, the shape determination time can be shortened. There are advantages.

The flowchart of FIG. 16 is obtained by adding steps S1602 to S1605 to the flowchart of FIG.

The above-mentioned steps are intended to use the shape information of the continuous data to more surely catch the eyes.
That is, when the shape of the continuous data is convex upward, the possibility of the eye is high, and the determination time is lengthened or the determination movement amount is reduced. If the shape is the same straight line, there is a high possibility that the curtain rail or the sunroof is behind the driver, and the determination time is shortened or the determination movement amount is increased. Thereby, the detection accuracy is improved.

In step S1602, it is determined whether the shape of the continuous data appearing in the eye position search area 2 is upwardly convex. At this time, if it is convex upward, the process proceeds to step S1603, and the upward convex counter is counted up. This process focuses on the fact that when the continuous data to be detected is eyebrows or eyes, most of the data is convex upward except when the eyes are closed. .

Accordingly, in step S1604, it is determined whether or not the number of upwardly convex counters has reached a predetermined number in the processing repeated within the predetermined time A in step S1601. At this time, as described above, if the data is continuous data for the eyes, eyebrows, etc., the upward convex counter is counted up at an early stage, and the process proceeds to step S1605.

In step S1605, the predetermined time A is extended, or the determination movement amount B is reduced. The extension of the predetermined time A and the reduction of the determination movement amount B are to cope with a case where the movement of the detection target person is extremely small. The upward convex counter is cleared when the eye position search area 3 is set or when the predetermined time A is exceeded.

Next, a method for surely catching the eye even when the movement under the frame of the glasses is to be determined by adding the shape determination of the continuous data with reference to FIGS. 18 to 21 will be described.

In the detection target person wearing glasses as shown in FIG. 18, the face movement happens to be small and the eye position search area 1 is reduced as shown in FIG. When the continuous data at the upper end is detected under the eyeglass frame, the eye position search area 2 is set, and the process proceeds to the motion determination. When it is determined that the corresponding continuous data has moved, the eye position search is performed according to FIG. If the region 3 is set, the eye above the eyeglass frame is outside the region and cannot be detected. In addition, in the measure for simply expanding the eye position search area 3 upward, continuous data that becomes noise, such as hair, may appear and the eye detection accuracy may be reduced.

Therefore, in this embodiment, the eye position search area 2
This problem is solved by determining the shape of the continuous data appearing in. As shown in FIG. 20, when it is below the spectacle frame, continuous data having a downward convex is often obtained. By determining the appearance frequency of the continuous data having a downward convex, as shown in FIG. The setting position of the eye position search area 3 can be changed. By determining the shape of the continuous data, continuous data of the eye can be captured in the eye position search area 3 even if the determination in the eye position search area 2 is performed under the spectacle frame. Detection is possible.

Next, the eye position search area 4 shown in FIG.
A method for further surely correcting the set position of the eye position search area 3 by adding the shape determination of the continuous data in the above will be described.

The details of this embodiment will be described with reference to the flowcharts of FIGS. 22 to 24 and FIGS. 25 to 36.

FIG. 22 is obtained by adding step S2207 to the flowchart of FIG. In the above-described embodiment, the moving amount of continuous data appearing in the eye position search area 2 is determined, and the eye search area 3 is set when the condition is satisfied. FIG.
The eye position search area 4 is set as shown in step S2309 of the flowchart of No. 3, and the relative positional relationship and shape of two continuous data appearing in the area are determined. Therefore, in step S2309 of FIG.
Is set, the process moves from step S2207 in FIG. 22 to step S2401 in FIG.

Now, the details of the determination in the eye search area 4 will be described with reference to the flowchart of FIG.

FIG. 25 shows a case where the continuous data appearing at the uppermost end in the eye position search area 1 is regarded as G1, and the movement of the continuous data in the eye position search area 2 satisfies the determination condition. Is shown.

At this time, the eye position search area 4 in step S2309 of FIG. 23 is set as a horizontally long area having the same width in the left, right, up, and down directions with reference to the center coordinates of the continuous data G1 as shown in FIG. . In step S2401 of FIG. 24, it is determined whether or not there is two or more continuous data that satisfies the shape and the position condition in the eye position search area 4, and there is two or more continuous data that satisfies the condition. Shifts to step S2402.

The continuous data G2, which is described somewhat lightly in FIG. 25, indicates that the detection of the continuous data is unstable, and in the processing for each frame, the continuous data G2 does not appear. . Therefore, two continuous data appearing in the eye position search area 4 in FIG. 26 may be G1 and G4. When three or more continuous data appear in this area, two continuous data are selected from the upper part of the area.

At the stage when two symmetrical continuous data are detected, it is not yet known whether the corresponding continuous data is an eyebrow or an eye. Therefore, FIG.
Is set, and the final eye continuous data is detected.

However, a problem may occur when setting the eye search area 3. Although one of the countermeasures has been described in the previous embodiment, this is a case where continuous data can be recognized only under the frame of the glasses and the eye position search area 3 is set. The eye position search area 3 is in a state where it is not yet known whether the corresponding continuous data is an eyebrow or an eye. Therefore, the area is enlarged and set only downward with reference to the continuous data, and appears in the area. Eye continuous data is detected from the continuous data.

As described in the previous embodiment, the reason why the image is enlarged only in the downward direction is that when the continuous data of the eyebrows is used as a reference and continuous data of hair is included or a hat is worn. This is because continuous data of the hat may appear, and the determination in the eye search area 3 may be complicated, and the detection accuracy of the eye may deteriorate.

Therefore, it is necessary to shift the eye position search area 3 upward as in the previous embodiment shown in FIG. Even in the method based on the shape determination in the eye position search area 2 as in the previous embodiment, it is possible to determine to some extent that the data is continuous data under the frame of the eyeglasses, and it is not possible to correct and set the eye search area 3. However, continuous data under the frame of the glasses may be detected as horizontal continuous data, and it may not be possible to recognize that the data is under the frame of the glasses.

In the present embodiment, in order to more reliably determine that the data is continuous data under the frame of the glasses, two determinations on two continuous data appearing in the eye position search area 4 are performed. I do.

One of the determinations is step S2402
Then, it is determined whether the continuous data corresponding to the right side of the two corresponding data is the eye or under the frame of the glasses. The determination of the presence or absence of the appearance of the two continuous data is based on the fact that the two feature amounts exist at positions where the eyes are symmetrical in the horizontal direction.

The two symmetrical continuous data appearing in the eye existence area 4 are composed of the left eyebrow (which may be on the frame of the eyeglasses when wearing glasses) and the right eye, May be left eye and right eye. Further, FIG.
As shown in FIG. 6, there is also a case where detection is performed at different levels, such as a right eyebrow and a left eye. This is because the appearance of continuous data corresponding to the eyebrows becomes unstable in the case of a person who has bangs down and is on the eyebrows.

In step S2402, it is first determined whether the data on the right side of the two continuous data is below the eye or the frame of the glasses. The determination method here is shown in FIG.
As shown in FIG. 6, the level difference L between the left and right continuous data G1 and G4
It is determined whether 1 is large. When the position of the continuous data on the right side is lower than that on the left side,
The case where the left eyebrow and the right eye as shown in FIG. 6 are targeted and the case where the left eye and the right glasses under the frame as shown in FIG. 29 are targeted.

In a situation where two continuous data appear in a combination as shown in FIG. 29, both eyebrows are difficult to catch as continuous data due to the influence of the bangs as shown in FIG. This is the case when detection is unstable due to factors such as the following. By determining such a combination of continuous data based on the steps L1 and L2 of both continuous data, the search area 3 of the eye is set to be corrected upward as shown in FIGS. In the case of FIG. 27, since the corresponding continuous data is the eyebrows, there is no need to correct the set position of the eye position search area 3. However, in this case, the eye position is equivalent to the case where the left and right symmetry of the eyebrows is captured and set. There is no problem in the detection of. Therefore, if there is a step in the two continuous data in step S2402, the process proceeds to step S2403,
The set position of the eye position search area 3 is corrected. When there is no step between the two continuous data, the eye position search area 3 shown in FIG.

Next, the reason why the eye position search area 4 is set to be wider on both sides with reference to the continuous data specified in the eye position search area 1 will be described. Since the continuous data specified in the eye position search area 1 captures the uppermost continuous data appearing in the same area, it cannot be specified which of the left and right eyebrows, eyes, and eyeglass frames. Therefore, it is necessary to look at both sides in order to capture another continuous data having a symmetrical positional relationship between the eyebrow, eye, and eyeglass frames.

The merit of this region setting is that, in addition to the improvement of the eye detection accuracy by judging the bilaterally symmetrical features of the eyebrow, eye, and eyeglass frames, the improvement of the ability to respond to changes in the light environment in the vehicle. There is also. Specifically, in the case of a right-hand drive vehicle, the right side of the driver's face may be exposed to direct light from the side window, and it may be difficult to capture the shape of the eye on the image. Therefore, in order to accurately grasp the shape of the eye, it can be said that the eye inside the vehicle, that is, the left eye of the driver is more preferable. Add eye position search area 4,
Capturing the left and right eyebrows, eyes, and eyeglass frames can reliably detect the driver's left eye, which is relatively stable in the light environment, and is therefore strong against changes in the light environment in the vehicle interior.

Next, a description will be given of a method of judging another eyeglass frame. This determination is made in step S2404. In step S2402, the shape is determined for two continuous data appearing in the eye position search area 4, thereby recognizing that it is under the frame of the glasses. The flow of the processing is shown in FIGS.
This will be described with reference to FIG.

When the eye cannot be completely caught by the eyebrows and the eyeglass lens due to the effect of the forelock, continuous data of the eyeglass frame G6 may be detected as shown in FIG.
In such a case, the appearance of two continuous data in a left-right symmetric relationship in the eye position search area 4 is caused by the left and right steps L
Since 3 is small, it cannot be recognized in step S2402 that it is continuous data under the frame of glasses. Thus, in step S2404, it is determined whether or not a downward convex shape, which is a feature of the continuous data below the eyeglass frame, as described with reference to FIG.

In this embodiment, since the shape can be determined by using two continuous data on the left and right sides, the shape is determined to be convex downward along with the continuous data on both sides shown in FIG. 36, the continuous data only on the left side has a downward convex shape as shown in FIG. 36, and the continuous data only on the right side has a downward convex shape as shown in FIG. 36. Therefore, the recognition accuracy can be greatly improved compared to the previous embodiment.

If the above condition is satisfied in step S2404, the flow shifts to step S2405, where the eye search area 3
Correct the setting position of. If the conditions of both steps S2402 and S2404 are not satisfied,
The normal eye position search area 3 is set in step S2406, and the process returns to step S2202. Step S24
If there is no continuous data that satisfies the shape and position conditions in the eye position search area 4 in step 01, step S2407
Then, the reduced eye position search area 1 is set to an area not including the setting criterion of the eye position search area 2 and a step S24 is performed.
08. In step S2408, the reduction limit of the eye position search area 1 is determined. If the reduction limit is exceeded, the detection area is returned to the initial state. Step S240
The processing of steps S1, S2407, and S2408 determines the appearance of two symmetrical continuous data appearing in the eye position search area 4 so that hair and nose,
Mouth continuous data can be excluded. Therefore, when the eye cannot be detected due to the temporary deterioration of the image state, the continuous data of the nose and mouth under the eye cannot be detected, and the eye position search area 1 reaches the lower end. The eye position can be detected again from the upper part of the screen.

Finally, a method of selecting continuous eye data appearing in the eye position search area 3 using the shape determination of the continuous data will be described.

The set position of the eye position search area 3 is shown in FIG.
(A) to (c). (A)
Fig. 7B shows a case where the same region is set to be expanded downward with reference to the continuous data of the eyebrows, and Fig. 10B shows a case where the same region is set to be expanded upward with the continuous data of the eye as a reference.
Indicates that the area is enlarged upward based on the continuous data below the spectacle frame. There are cases where the eyebrows and eyes are present in the same area as in (a) and (b), and cases where the eyes and below the spectacle frame are present as in (c).

In these two patterns, it is difficult to determine only the positional relationship between the continuous data. However, if the shape data of each continuous data is used, the distinction can be reliably performed. That is, the eyebrows and eyes often have a convex shape upward, and the eyebrows do not have a convex shape downward. In addition, the lower part of the spectacle frame often has a downward convex shape, and does not have an upward convex shape. As described above, by adding the shape combination determination to the positional relationship of the continuous data appearing in the same area, the eye detection accuracy can be further improved.

The determination in the eye position search area 3 is as follows.
Step S310 in FIG. 3 and step S151 in FIG.
0, the expression in the flowchart of step S2212 in FIG. 22 shows a single loop, but it is naturally possible to execute several loops and determine the frequency from the appearance frequency of the target continuous data. Therefore, FIG. 30 and FIG. 33 show a state in which continuous data corresponding to the eye does not appear, but if the process is repeated several times, the continuous data of the eye is at a recognizable level.

In this way, by accurately detecting the position of the driver's eyes irrespective of the presence of the curtain rail or sunroof behind, the driver's drowsy driving and inattentive driving can be accurately detected. Appropriate measures such as a warning and vehicle stop can be taken. Further, by accurately detecting the position of the driver's eye, the angle of the mirror, the seat height, and the like can be appropriately controlled.

In the above embodiment, the A / D converter 22
Although the following vehicles are all performed by digital calculation, the processing up to obtaining the arithmetic mean of the density values along the pixel row may be performed by an analog signal circuit. For example, set a scanning line in the vertical direction with a camera using a captured image,
After removing the high-frequency component by passing a code signal along a scanning line crossing the face in the vertical direction through a low-pass filter, a code signal for one scanning line is extracted for every ten scanning lines.
A / D conversion is performed by sampling.

[Brief description of the drawings]

FIG. 1 is a functional block diagram of an embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of one embodiment of the present invention.

FIG. 3 is a flowchart showing an operation of one embodiment.

FIG. 4 is a flowchart illustrating an operation of the embodiment.

FIG. 5 is a flowchart showing an operation of the embodiment.

FIG. 6 is an explanatory diagram relating to processing in a flowchart.

FIG. 7 is an explanatory diagram related to a process in a flowchart.

FIG. 8 is an explanatory diagram related to a process in a flowchart.

FIG. 9 is an explanatory diagram related to a process in a flowchart.

FIG. 10 is an explanatory diagram related to a process in a flowchart.

FIG. 11 is an explanatory diagram related to a process in a flowchart.

FIG. 12 is an explanatory diagram related to a process in a flowchart.

FIG. 13 is an explanatory diagram related to a process in a flowchart.

FIG. 14 is an explanatory diagram related to a process in a flowchart.

FIG. 15 is a flowchart according to one embodiment.

FIG. 16 is a flowchart showing the operation of the embodiment.

FIGS. 17A and 17B are explanatory diagrams related to the processing of the flowchart. FIG. 17A is an explanatory diagram of continuous data on a straight line, and FIG.

FIG. 18 is an explanatory diagram related to a process in a flowchart.

FIG. 19 is an explanatory diagram related to a process in a flowchart.

FIG. 20 is an explanatory diagram related to a process in a flowchart.

FIG. 21 is an explanatory diagram related to a process in a flowchart.

FIG. 22 is a flowchart according to one embodiment.

FIG. 23 is a flowchart according to one embodiment.

FIG. 24 is a flowchart according to an embodiment.

FIG. 25 is an explanatory diagram related to a process in a flowchart.

FIG. 26 is an explanatory diagram related to a process in a flowchart.

FIG. 27 is an explanatory diagram related to the processing of the flowchart.

FIG. 28 is an explanatory diagram related to the processing in the flowchart.

FIG. 29 is an explanatory diagram related to the processing in the flowchart.

FIG. 30 is an explanatory diagram related to a process in a flowchart.

FIG. 31 is an explanatory diagram related to a process in a flowchart.

FIG. 32 is an explanatory diagram related to the processing in the flowchart.

FIG. 33 is an explanatory diagram related to the processing in the flowchart.

FIG. 34 is an explanatory diagram related to a process in a flowchart.

FIG. 35 is an explanatory diagram related to the processing in the flowchart.

FIG. 36 is an explanatory diagram related to a process in a flowchart.

FIGS. 37A and 37B are explanatory diagrams relating to the processing of the flowchart. FIG. 37A is an explanatory diagram showing a state where the continuous area of the eyebrows is set as a reference and the area is enlarged downward, and FIG. Is an explanatory diagram in which the same region is enlarged and set upward, and FIG. (C) is an explanatory diagram in which the same region is enlarged upward based on continuous data below a spectacle frame.

[Explanation of symbols]

CL1 image input means CL2 density detection means CL3 point extraction means CL4 curve data extraction means CL5 predetermined area 1 setting means (eye position search area 1 setting means) CL6 curve data detection means CL7 predetermined area 2 setting means (eye position) Search area 2 setting means) CL8 Curve data moving state determination means CL9 Setting means for predetermined area 3 (eye position search area 3 setting means) CL10 Reduction means for predetermined area 1 (eye position search area 1)
CL11 Eye position detecting means 21 TV camera 22 A / D converter 23 Image memory 24 Image data arithmetic circuit 25 Eye position detecting circuit

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁶ Identification symbol FI // G08B 21/00 G06F 15/62 380

Claims (12)

[Claims] An image input means for inputting image data including a face, a density of a pixel is read out along a vertical pixel row of the input image data, and a density value on the pixel row in one direction is read out. A point extracting means for specifying a pixel of a peak in one direction of the density value as an extraction point on condition that the differential value of the density change before the peak exceeds a predetermined value; Curve data extracting means for extracting curve data extending continuously in the lateral direction of the face continuously from the adjacent extraction points; moving state determining means for determining a moving state of the curve data; and wherein the curve data is dynamic. An eye position detecting device comprising: an eye position detecting means for detecting an eye position based on the curve data when the determination is made. 2. The eye position detection apparatus according to claim 1, wherein the eye position search area for setting the eye position search area 1 including the curve data extracted by the curve data extraction means is set. A position search area 1 setting means; an uppermost curve data detecting means for detecting curve data appearing at the uppermost end in the eye position search area 1; and an eye position search area 2 based on the uppermost curve data. Eye position search area 2 setting means to be set; Curve data movement state determination means for determining the movement state of curve data appearing in the eye position search area 2; The target curve data is dynamic Eye position search area 3 for setting eye position search area 3 when it is determined that
Setting means, when it is determined that the target curve data is static curve data, returns to the setting of the eye position search area 1 and searches for the eye position based on the position information of the corresponding curve data. Eye position search area 1 reducing means for reducing the range of area 1 in the vertical direction, and eye position detecting means for detecting the eye position from the relative positional relationship of curve data appearing in eye position search area 3 An eye position detection device, comprising: 3. The eye position detecting device according to claim 1, wherein the moving state determining unit determines the moving state of the curve data based on a vertical movement amount of the curve data. Eye position detection device. 4. The eye position detecting device according to claim 3, wherein the moving state determining means determines the moving state of the curve data by an average value of vertical positions of each extraction point forming the curve data. An eye position detection device. 5. The eye position detecting device according to claim 1, wherein the moving state determining unit determines the moving state based on a position of an uppermost end position of the extraction points forming curve data. An eye position detecting device characterized in that the eye position detecting device is used. 6. The eye position detection device according to claim 1, wherein the movement state determination unit determines a shape of curve data appearing in the eye position search area 2. Determining means;
An eye position detecting device, comprising: a moving state determination time changing unit that changes a determination time of the curve data moving state determining unit based on an output result of the shape determining unit. 7. The eye position detecting device according to claim 1, wherein the moving state determining unit determines a shape of curve data appearing in the position search area 2. An eye position detection device comprising: a movement state determination movement amount changing unit that changes a determination movement amount of the curve data movement state determination unit based on an output result of the shape determination unit. 8. The eye position detection device according to claim 1, wherein the movement state determination unit determines a shape of curve data appearing in the eye position search area 2. Determining means;
An eye position detection device comprising: an area 3 setting condition changing unit that changes a setting condition of the eye position search area 3 based on an output result of the shape determining unit. 9. The eye position detection device according to claim 2, wherein the moving state determination unit crosses right and left of the face image data based on curve data appearing in the eye position search area 2. Eye position search area 4 setting means for setting a horizontally long eye position search area 4, and left and right data for determining whether there are two or more left and right curve data in the set eye position search area 4 A determination unit; and a setting condition changing unit of an area for changing a setting condition of the eye position search area by specifying curve data based on a magnitude of a vertical step of the determined left and right data. Eye position detection device. 10. The eye position detecting apparatus according to claim 9, wherein said left and right data determination means selects two curve data from an upper part in said eye position search area 4. Detection device. 11. The eye position detection device according to claim 9, wherein the setting condition changing unit of the area 3 is configured to change the eye position search area when the level difference between the left and right data is equal to or less than a predetermined value. 4. An eye position detecting apparatus, comprising: shape determining means for determining the shape of curve data appearing in 4 and specifying the curve data. 12. The eye position detecting device according to claim 2, wherein the shape determining means converts the shape of the curve data into the shape of the curve data. Connect the rightmost and leftmost points of each constituent extraction point with a straight line,
An eye position detecting device for recognizing a shape based on a position of an extraction point at the center of the curve data with respect to the straight line.