JP3500891B2 - Dozing state detection device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a doze state detecting device for calculating the doze state of a driver of a vehicle, a ship operator, an operator of a plant or the like by arithmetically processing face image data.

[0002]

2. Description of the Related Art As a conventional doze state detecting device using image processing, there is, for example, one disclosed in Japanese Patent Laid-Open No. 346094/1993. This is to binarize the grayscale image data of the driver's face and perform density projection and labeling on the resulting binarized image to detect the position of the eye and open / close the eye. The doze state is detected from the change.

In addition, Japanese Patent Application No. 8-101904 discloses a device for detecting the position of the eyes of a vehicle driver that can process grayscale images and can also be used for drowsiness detection.
Applicants have proposed such as described in the publication.
Here, the density of pixels is detected along the pixel row in the vertical direction of the face using the face image of the person to be photographed, and one pixel is determined for each local increase in the density in the pixel row, and the extraction point is determined. Then, the dozing state is detected by connecting the extraction points adjacent to each other in the pixel row direction of the adjacent pixel rows and detecting the position of the eye from the group of curves extending in the lateral direction of the face.

[0004]

However, in the former case, as shown in FIG. 36, for example, when the hair overlaps the eye, the hair has a density value of the same gradation as that of the eye. Therefore, in the image after the binarization processing, the eye is not a single label, and there is a problem that the position of the eye and the open / closed state of the eye cannot be recognized.

In the latter case proposed to solve this problem, the angle of view of the camera must be fixed in terms of cost. In this case, the position of the face may become extremely low with respect to the angle of view due to individual differences, as shown in FIG. In such a case, for example, the average density value on the pixel row Xa is (a) in FIG.
In addition to the eyebrows and eyes to be extracted, the hair portion is included. Therefore, when the eyebrow and eye points p2 and p3 are extracted based on the differential value distribution diagram (b), the points of the hair are similar to those having the differential value peaks q2 and q3 which are equal to or less than the predetermined value which is the extraction condition. p1
Has q1 and is extracted. A1, A2 and A3 represent the Y coordinates of the extraction points. As described above, in addition to the eyes, eyebrows, nose, mouth, and the like that are considered as extraction targets, even hair portions are also extraction targets, and thus there is a problem that the accuracy of eye selection decreases. The present invention, in view of the above conventional problems,
An object of the present invention is to provide a snooze state detection device capable of responding to differences in the hairstyle and face position of a driver, etc., and more reliably detecting eye position and open / closed state, and capable of constantly performing snooze state detection. I am trying.

[0006]

Therefore, according to the invention of claim 1, as shown in FIG. 1, in a doze state detecting device for processing face image data to detect a doze state,
Image input means 1 for inputting a face image, density detecting means 2 for detecting the density of pixels along a vertical pixel row of the face, and density before and after a unidirectional peak of density along the pixel row. A first point extracting means 3 for identifying a pixel having a differential value exceeding a predetermined value to extract a first point, and the first point adjacent in the pixel row direction of an adjacent pixel row are connected to connect the face A curve group extracting means 4 for extracting a laterally extending curve group, an eye position detecting means 5 for specifying an eye curve from the curve group to detect an eye position, and a vertical direction within a predetermined region including the eye. A pixel whose density differential value before and after the unidirectional peak of the density along the pixel row in the predetermined direction exceeds a predetermined value is specified.
Second point extracting means 6 for extracting the points, and continuity determining means 7 for determining the lateral continuity of the face depending on the presence / absence of the second points adjacent to each other in the pixel row direction of the adjacent pixel rows, Eye opening degree detection means 8 for detecting the eye opening degree based on the continuous extracted data of the second point, and awakening degree determination means for determining the awakening degree from the change in the open / closed state of the eye based on the eye opening degree. 9 and 9.

According to a second aspect of the present invention, the first point extracting means 3 or the second point extracting means 6 scans along the pixel row to determine a minimum differential value of a density exceeding a predetermined value as a negative differential value. The differential value is calculated from the negative value, and the pixel whose differential value is inverted from negative to positive value is stored, and the maximum differential value exceeding a predetermined value different from the above is detected from the subsequent positive differential value interval. Pixels that are positively inverted are extracted and set as the first or second points.

According to a third aspect of the present invention, the first point extracting means 3 or the second point extracting means 6 is provided.
However, in scanning along the pixel row, the minimum differential value of the density exceeding a predetermined value is obtained from the negative differential value section, and the pixel whose differential value is inverted from negative to positive is stored. By detecting a positive differential value that exceeds a predetermined value of, a pixel in which the differential value is inverted from negative to positive is extracted, and
Alternatively, the second point is set.

Further, in the first point extracting means 3 or the second point extracting means 6, between pixels having a first positive differential value exceeding a predetermined value for judging the magnitude of the minimum differential value and the positive differential value. When the distance is greater than or equal to the determination value, it is possible to perform the process of excluding the pixel whose differential value is inverted from negative to positive from the extraction of the first or second point.

Further, before the first positive differential value exceeding a predetermined value for judging the magnitude of the positive differential value is detected in the first point extracting means 3 or the second point extracting means 6, When there is another minimum differential value having a value smaller than the minimum differential value, it is possible to perform a process of excluding the pixel whose differential value is inverted from negative to positive from the extraction of the first or second point. Is.

When there are a predetermined number or more of 0 differential value pixels without density change following the pixel whose differential value is inverted from negative to positive, the differential value is inverted from negative to positive. It is desirable to exclude pixels from the extraction of the first or second points. It is desirable that the predetermined value for determining the magnitude of the maximum differential value or the positive differential value be changed in magnitude according to the minimum differential value.

[0012]

According to the first aspect of the invention, first, with respect to the face image input by the image inputting means 1, the density detecting means 2 detects the density of pixels along the vertical pixel row. Then, the first point extracting means 3 extracts the pixel of the peak of the density having the differential value equal to or more than the predetermined value before and after, and sets it as the first point. The curve group extracting means 4 connects the first points that are adjacent to each other between adjacent pixel rows, and extracts a curve group extending in the lateral direction of the face. The individual curves represent the dark areas transversely across the lighter areas of the image,
Represents eyes, eyebrows, mouth, etc. The eye position detecting means 5 identifies the curve representing the eye from the above curve group by utilizing the feature that the eyes are on both sides of the face and the eyes and the eyebrows are close to each other in the vertical direction. Detect the position.

Then, the second point extracting means 6 sets a predetermined area including the position of the eye, and extracts the second point in the predetermined area in the same manner as the first point. After confirming by the continuity determination means 7 that the second points are adjacent to each other and are continuous in the lateral direction of the face, the opening degree detection means 8 detects the eye based on the extracted data of the second points. The opening degree of is detected. Then, the awakening degree determination unit 9 determines the awakening degree from the change in the open / closed state of the eye based on the opening degree of the eye.

After extracting a group of curves extending in the lateral direction of the face, the curve representing the eyes is identified and the position of the eyes is identified by utilizing the characteristic of the general positional relationship between the eyes and other parts of the face such as the eyebrows. Therefore, the position of the eye can be detected easily and surely even if there is a difference in the physique or posture of the person to be photographed.

According to the second aspect of the invention, the pixel whose differential value is inverted from negative to positive corresponds to the density peak, and the differential value is inverted when the pixel is extracted as the first point or the second point. Since the minimum and maximum differential values are calculated from the negative and positive differential value sections before and after, and the strength is determined by a predetermined value, the extracted points are representative of dark areas with clear boundaries and correspond to eyebrows, eyes, mouths, etc. To do. Wrinkles
Dark areas where shadows and equal boundaries are not clear have small differential value strengths before and after the peak, and are therefore dropped from the extraction target, resulting in high quality extraction. The extraction point represents the peak of the density, and corresponds to the approximate center position of the eyes, eyebrows, and mouth.

According to the third aspect of the invention, the minimum differential value exceeding a predetermined value is detected from the negative differential value section, and the pixel whose differential value is inverted from negative to positive is extracted as the first point or the second point. In doing so, since the strength determination is performed using a positive differential value that exceeds a predetermined value within a predetermined section, it is not necessary to find the maximum value, and points can be extracted even if the density value distribution is distorted or deformed. .

When the distance between the pixels of the first positive differential value exceeding the predetermined value for judging the magnitude of the minimum differential value and the positive differential value is equal to or larger than the judgment value, the differential value changes from negative to positive. By excluding the inverted pixel from the extraction of the first or second point, erroneous extraction due to modification of the distribution curve of the differential value is prevented.

Further, before the first positive differential value exceeding a predetermined value for judging the magnitude of the positive differential value is detected in the first point extracting means or the second point extracting means, the minimum differential value is detected. When there is another minimum differential value having a value smaller than the value, the pixel whose differential value is inverted from negative to positive is excluded from the extraction of the first or second point, and the distribution of the differential value is obtained. Erroneous extraction due to curve deformation is prevented.

If a predetermined number or more of pixels having no change in density exist following the pixel whose differential value is inverted from negative to positive, the pixel whose differential value is inverted from negative to positive is the first pixel. Alternatively, by excluding the second point from extraction, erroneous extraction due to hair or the like is prevented. If the predetermined value for determining the magnitude of the maximum differential value or the positive differential value is changed in size according to the minimum differential value, the points are extracted with higher accuracy according to the density change characteristics of the target to be detected. There is something to do.

[0020]

BEST MODE FOR CARRYING OUT THE INVENTION Next, an embodiment in which the present invention is applied to a dozing state alarm for a driver of an automobile will be described with reference to the drawings. FIG. 2 is a diagram showing the configuration of the embodiment. On an instrument panel of a car, a TV camera 21 is installed to photograph the driver's face from the front. TV
The input image captured by the camera 21 is 51 in the horizontal (X) direction.
It consists of 2 pixels and 480 pixels in the vertical (Y) direction.
The analog voltage of each pixel is, for example, 256 by the / D converter 22
The converted digital value is stored in the image memory 23.

An image data arithmetic circuit 24 is connected to the image memory 23. The image data calculation circuit 24 detects the density of the pixel row in the vertical direction of the face based on the input image data stored in the image memory 23, and will be described later depending on the local increase of the density in the pixel row and the density change state. The first point is extracted, and the two-dimensional data indicating the position of the first point is stored in the built-in point memory. Then, after extracting the first point for each pixel row, the first points that are adjacent to each other in the pixel row direction of the adjacent pixel rows are connected to each other to extract a curve group in the lateral direction of the face.

An eye position detection circuit 25 is connected to the image data arithmetic circuit 24, and an open / closed eye detection circuit 26 is subsequently connected. The eye position detection circuit 25 detects the eye position by selecting an eye from the group of curves. The open / closed eye detection circuit 26 extracts a second point, which will be described later, according to the increase in the density in the vertical direction and the changing state of the density within a predetermined area including the position-detected eye, and indicates the position of the second point. Two-dimensional data is stored in the built-in point memory. Then, the continuity of the second point in the lateral direction of the face is determined, and the eye opening value is detected from the continuous data.

The open / closed eye detection circuit 26 includes an awakening degree determination circuit 2
7 is connected, and the awakening degree determination circuit 27 is the open / closed eye detection circuit 2
The degree of awakening is determined from the detection result of the open / closed eyes sent from S6. An alarm device 28 is connected to the awakening degree determination circuit 27, and an alarm is generated upon receiving a determination signal of a dozing state in which the awakening degree is lowered.

Next, the flow of control operation in the above configuration will be described with reference to the flowchart of FIG. In step 31, the TV camera 21 photographs the face portion of the driver. In step 32, the pixel density is converted into a digital value by the A / D converter 22, and then the image memory 23
The image data corresponding to the image of one frame is stored in.

In step 33, the image data arithmetic circuit 2
In step 4, it is checked whether or not the eye existence region is set. If it is set, the process proceeds to step 36, and if it is not set, the process proceeds to step 34. Here, the eye presence region, which will be described later with reference to FIG. 13, indicates a predetermined region 100 including the eye, and is a region where the eye opening detection process is performed. The predetermined area 100 including the eyes is also used as an eye tracking area. In step 34, the eye position detection circuit 25 detects the eye position. Regarding the details of the eye position detection, the flowchart of FIG. 4 and FIGS.
Will be described later with reference to FIG.

In step 35, the predetermined area 10 including the eyes
The width of 0 in the horizontal direction (X direction) and the width of 0 in the vertical direction (Y direction) are set, and the existence region of the eye (predetermined region including the eye) 100 is set, and the process proceeds to step 36. In step 36, the opening / closing eye detection circuit 26 detects the degree of eye opening. For details of the detection of the degree of eye opening, the flowchart of FIG. 10 and FIG.
1 and FIG. 12 will be described later.

In step 37, it is checked from the eye opening value detected in step 36 whether or not the eye is correctly tracked. If the eye is correctly tracked, the process proceeds to step 38 and it is determined that the eye is not correctly tracked. If so, the process proceeds to step 311. In step 38, it is determined whether the eye is open or closed based on the eye opening value. Then step 3
In 9, the eye tracking, that is, the area where the eye is present is changed.
Details of eye tracking will be described later with reference to FIG.

At step 310, the awakening degree is determined by the awakening degree determination circuit 27. If the vehicle is in the awake state, the process returns to step 31 to proceed to the processing of the next frame. If the vehicle is not in the awake state, the process proceeds to step 312, the alarm device 28 is operated to wake up the driver to generate a drowsiness alarm, and then the process returns to step 31. In step 311, the eye opening value and the eye existence region are cleared, the process returns to step 31, and the process for the next frame starts. It should be noted that steps 31 to 32 of the above constitute the image inputting means in the present invention.

FIG. 4 shows details of the processing operation of eye position detection in step 34. Here, as shown in FIG. 5, with respect to the data on the line (pixel column) of 480 pixels in the Y-axis direction, the density value data of each pixel column is arithmetically processed, and the pixel (Y Coordinates).
That is, a large change in the density value on the pixel array is obtained, and its local peak is set as the first point. When the point extraction process is performed and one line is completed, the process moves to the next adjacent line. Therefore, first, step 41
Then, it is checked whether or not the point extraction is completed for all lines in the predetermined direction (Y-axis direction).

If it is determined in step 41 that the first point extraction has not been completed for all lines, the process proceeds to step 42. In step 42, the arithmetic mean of the density values is calculated in one line in the predetermined direction to obtain the arithmetic mean density value on the pixel row. The purpose of this processing is to eliminate the small variation in the change in the density value at the time of photographing the image data and to catch the global change in the density value. For example, the arithmetic average density value as shown in FIG. 6A is calculated from the pixel row Xa shown in FIG.

In step 43, the one-line arithmetic mean density value in the predetermined direction calculated in step 42 is differentiated. Figure 6
From the arithmetic mean density value shown in (a) of FIG. 11, the differential value (differential data) shown in (b) of the same figure is obtained. Next step 44
Then, according to the differential value of (b) of FIG. 6 which is the calculation result of step 43, the point (Y coordinate) corresponding to the peak of darkness in which the arithmetic average density in (a) of FIG. Extract. The Y coordinate where the differential value is inverted from negative to positive corresponds to the peak of darkness, and is the coordinate indicating the position of the eyes, eyebrows, mouth, and the like. The peak point of this concentration is
The change in the density value before and after that, that is, the differential value is extracted as the first point by determining whether it is less than or equal to a predetermined value or more. The method of extracting this point will be described in detail later. In step 45, when the point extraction process for extracting the first point is completed for one line, the process is switched to the process for the next line, the process returns to step 41, and the process is repeated.

When it is determined in step 41 that the points of all the lines have been extracted, the eyes as shown in FIG.
This means that the first points representing the eyebrows, the mouth, etc. have been extracted. That is, the two first points A1 and A2 are extracted on the line Xc in FIG.
On d, the four first points A1, A2, A3, and A4 are extracted. The step 42 constitutes the concentration detecting means of the present invention, and the step 42 and the steps 43 and 44 constitute the first point extracting means.

When it is judged in step 41 that the extraction of the first point has been completed for all lines, step 46
In, the Y coordinate values A1, A2, A3, ... Of the first points of adjacent lines are compared. When the difference in Y coordinate value is within a predetermined value (for example, 10 pixels), X
Grouping is performed by connecting in the axial direction, and the group number of continuous data, the continuous start line number, and the number of continuous data are stored.

The specific processing contents of step 46 will be described with reference to FIG. Here, for simplification, description will be made assuming that there are 11 lines (pixel rows). The first point extracted in each of the lines 1 to 11 is
A group number (G), a Y coordinate value (Y value), and a continuous number (N) are given to form concatenated data. The 34 points shown in FIG. 8 are grouped into 10 groups.

Line 1 has two first points with Y coordinate values 192 and 229. Y coordinate value of line 1 is 1
Since there is no line to the left of point 92, there is no other continuous data at this stage, so the group number of continuous data is 1. For the same reason, the point of Y coordinate value 229 is the same as the above 1
Group number of continuous data,
The following 2 is set.

Next, in the line 2 on the right, the Y coordinate value is 1.
There are two extraction points, 91 and 224. Since the point of the Y coordinate value 191 of the line 2 is within 10 pixels from the Y coordinate value 192 of the line 1 on the left side, the group number of continuous data is set as 1 as an extraction point connected to the extraction point 192. At this time, the number of continuous data is 2. The determination is similarly performed for the point of the Y coordinate value 224 on the line 2, and the group number of continuous data is 2 and the number of continuous data is 2.

At the next point of the Y coordinate value 360 of the line 3, 360 is displayed on the line 2 on the left and the number of pixels in the Y axis direction is 10.
Since there is no point that falls within the range, the group number of continuous data of is 3, and the number of continuous data of is 1. The continuous start line number in step 46 is a line number having a point where the continuous data number is determined to be 1. Therefore, for example, the point of the Y coordinate value 360 is 3. In step 46,
In this way, the continuity of the points on each line is determined until all lines are completed, and a group of curves is obtained. Upon completion, the process moves to step 47.

In step 47, first, the number of continuous data in the continuous data group is evaluated. In this embodiment, it is assumed that an effective continuous data group is formed when the number of continuous data is 5 or more. Then, candidate point data of eye positions is formed for the effective continuous data group.
The average of the Y coordinate values of the points having the same continuous data group number is calculated and stored as the average value of the continuous points. This value is used as the representative Y coordinate value of the group. Further, a continuous end line is obtained from the continuous start line and the number of continuous data, and an average value of X coordinate values of the continuous start line and the continuous end line is calculated and stored. This value becomes the representative X coordinate value of the group.

In step 48, the eye position is detected based on the length of each continuous group and the positional relationship of the (X, Y) coordinates based on each continuous group data obtained in step 47. That is, considering the eye features,
It can be defined as an upwardly convex arcuate shape.
When the concatenated data is narrowed down based on this definition, the number of consecutive points is 5 due to the condition that the eyes are horizontally long.
From the condition that it continues for more points and has an arcuate shape, continuous data having a small difference in Y coordinate value between the continuous start point and the continuous end point can be narrowed down as eye candidates. If you narrow down the continuous data based on this,
Groups G1 to G6 as shown in FIG. 9A are extracted as valid data groups.

Next, considering the positions of the X and Y representative coordinate values of each group calculated in step 47, as shown in FIG. 9B, ZONE: depending on the degree of approach in the X coordinate direction. It can be classified into L, ZONE: C, and ZONE: R. This is because the left eye and the right eye are largely separated from each other in the X coordinate direction, that is, the faces are substantially symmetrically divided, and the left eye and the left eyebrow are not greatly separated from each other in the X coordinate direction. They are not greatly separated from each other in the X coordinate direction. Also,
The connected data due to the shadow under the nose and the connected data for the mouth are located near the center.

As described above, the eye position can be easily detected by further classifying the data according to the degree of approach in the X coordinate direction and narrowing the data. The connected data included in ZONE: L is the left eye and the left eyebrow, and ZONE: R
If it is determined that the connected data included in the right eye is the right eye and the right eyebrow, the eye positions are G3 and G4 in FIG.
The coordinate value can also be specified. The step 46 constitutes the curve group extracting means of the invention, and the steps 47 and 48 constitute the eye position detecting means.

FIG. 10 shows details of the processing operation for eye opening detection in step 36 shown in FIG. here,
As shown in FIG. 11A, the second point extraction process is performed on the data on the line (pixel array) in the Y-axis direction, and after the completion of one line, the process proceeds to the process for the next adjacent line. . Therefore, first, in step 51, it is checked whether or not the point extraction processing similar to the eye position detection is completed for the data on the line in the Y-axis direction within the predetermined area 100 including the eyes.

When it is determined in the check in step 51 that the extraction of the second points has not been completed for all lines, the process proceeds to step 52. In step 52,
The arithmetic mean of the density values of one line in the vertical direction is calculated. This process is intended to eliminate small variations in the change in the density value when the image data is captured, and is performed to capture the global change in the density value. FIG. 11B shows the processing result of the arithmetic mean calculation of the line data of the pixel column Xc of FIG. Then, in step 53, step 52
A differential operation is performed on the arithmetic mean value calculated in. The processing result is shown in FIG.

In the next step 54, first, step 5
The second point is extracted based on the differential value calculated in 3.
The method of extracting the second point will be described in detail later as with the first point. The point A at which the differential value in (c) of FIG. 11 changes from negative to positive is a point at which the graph is convex to the left in (b) of FIG. 11, and is a peak point of the concentration. By determining whether or not the differential value before and after the peak point is less than or equal to a predetermined value or more, that is, whether it is in the hatched portion as shown in (c) of FIG. To extract the points. Then, points B and C having differential values satisfying this condition are extracted. Point B is a negative maximum differential value, and point C is a point having a positive differential value exceeding a predetermined value, and represents a boundary point between a dark portion and a light portion along a vertical pixel row including the second point. ing. Steps 52 to 54 constitute the second point extracting means of the present invention.

In step 55, it is checked whether or not the second point exists, and if it exists, the process proceeds to step 56, and if it does not exist, the process proceeds to step 511. Step 5
In 6, first, the distance H in the Y coordinate direction between the point B and the point C, which is the maximum differential value before and after the second point A on the line, is obtained. That is, the length of the dark portion along the pixel row including the second point is obtained. Next, when points B and C are continuously present in the pixel row in the vertical direction, the intervals H of the lines are compared to update the maximum value.

In step 57, when the second points on each line are continuously present, the number of consecutive second points in the X-axis direction is counted up.
In step 58, it is determined whether or not the number of consecutive second points exceeds a predetermined value. If yes, the process proceeds to step 59, and if not, the process proceeds to step 510. In step 510, the process is switched to the next line and step 5
Returning to 1, the above processing is repeated for the switched line. In this way, the same processing is continued until the number of consecutive second points reaches a predetermined value.

During this process, if the continuity of the extraction points is interrupted and it is determined in step 55 that the second point does not exist, the process proceeds to step 511. In step 511, the maximum value of the interval between the extraction points B and C updated in step 56 and the counter are cleared, and the process proceeds to step 510. In step 510, switching to the next line is performed and the process returns to step 51.

When the continuity of the second point exceeds the predetermined value in step 58, the process proceeds to step 59. In step 59, the maximum value of the interval between points B and C is updated and stored. In step 59, the maximum value of the memory in the continuous data is also updated, and the update is performed between other continuous data exceeding the predetermined value.

Further, the positive differential value section following the pixel in which the differential value defined in claim 2 is inverted from negative to positive is set. Under the second point extraction conditions of FIGS. 15, 20, and 21, which will be described later, when a bright point enters the center of the eye as on the XC line of FIG. 11, there are two dark peak points of darkness, Since the maximum differential value becomes small, it may not enter the predetermined value that is the hatched portion. As described above, since the extraction condition of the second point is not satisfied, the continuity of the extraction points may be interrupted, and the eye data may be divided into L1 and L2 as illustrated in FIG. There is. But,
Even in such a case, by setting the predetermined number of consecutive numbers in step 58 to be a little less than half the length of the eye, L1,
Both L2 are extracted as continuous data exceeding a predetermined value, and the maximum value of the interval between point B and point C in step 59 is updated by H1max and H2max.

Of course, if the second point A and the points B and C indicating the boundary point between the eye and the surrounding light portion can be continuously extracted, even if the eyeball has a bright spot, As shown in 12 (b), continuous data L1 with one eye
And the maximum value H1max in the continuous data is stored in the memory. In FIG. 12, D
Sections 1, D2, and D3 indicate locations where there is no extraction point that satisfies the extraction condition of the second point in step 54, and sections L1 and L2 indicate locations where extraction points exist.

On the other hand, if it is determined in step 51 that the above-mentioned processing has been completed, the process proceeds to step 512,
The maximum value Hmax between the points B and C is output as the eye opening value. Here, steps 57 and 58 constitute the continuity determining means of the present invention, and steps 56, 59 and 51.
Reference numerals 1 and 512 form eye opening detection means.

Next, step 3 in the flowchart of FIG.
A method of tracking an eye in Nos. 7, 39, and 311 and a method of returning when a tracking error occurs will be described with reference to FIGS. Figure 3
Immediately after the system is started based on the flowchart of (1), naturally, in the first frame, the eye existing area (predetermined area including the eye) is not set. Therefore, it is determined in the check in step 33 that there is no eye-existing region, and after the eye position is detected in step 34, step 35
The eye presence region 100 is set with. At this time, the center coordinates of the eye and the center coordinates of the eye existence region 100 match as shown in FIG.

After the series of processes for the first frame is completed, the process proceeds to the process for the second frame, and the process returns to step 33.
When proceeding to, it is determined that there is an eye existence area because the eye existence area 100 has already been set. As a result, the process proceeds to step 36, the eye opening is detected, and then the process proceeds to step 37. At this time, when the eyes are correctly captured, for example, it becomes as shown in FIG. Here, the eye existence region 100 is located at the position set in the first frame, whereas the eye position is the eye position in the image data of the second frame captured in the second frame, Due to movement or the like, the center point of the eye is displaced from the center of the eye existence region 100.
However, as long as the eye does not come into contact with the boundary of the eye existence region 100, the eye opening detection described above can be performed.

In step 39, the existence region 10 of the eye is located at the center coordinates of the eye of FIG.
Change the 0 reference point. Thereby, it is possible to correspond to the movement of the driver's face. 13C and 13D show the positional relationship between the eye position and the eye existence region 100 in the face image data captured in the third frame and the fourth frame, respectively. Since the eye existence region 100 is changed in the processing of the previous frame, the deviation between the center coordinates of the eye and the center coordinates of the eye existence region 100 is small.

Whether or not the eye tracking is correctly performed in step 37 is performed using the eye opening value. That is, once the driver who is the photographed person is specified, the eye opening value only changes in the range from the time of opening the eyes to the time of closing the eyes, so when a value outside this range is output, It is determined that there was an eye tracking error. In this case, the process proceeds to step 311, the output value of the eye opening and the region where the eye is present are cleared, and the process proceeds to the next frame process. Then, in step 33, the process for the entire face is started again.

Next, the reference value for the determination of the open / closed eye in step 38 will be described. As described above, when the driver is specified, the output value of the eye opening changes between the eye open state and the eye closed state, so the reference value for determining the open / closed eye, that is, the threshold for determining the open / closed eye. Would be within that range. Here, since the person in the dozing state is not in the deep sleep state, the eyes may not be completely closed in some cases. Therefore, the median value of the open / closed eyes is set as the threshold.

Next, the determination of the awakening degree in step 310 of the flowchart of FIG. 3 will be described. 14
Is an open / closed eye pattern indicating the determination result output in step 38. The awakening degree is determined by checking whether the eye-closure integrated value output in the awakening degree determination section (for example, about 1 minute) on this eye opening pattern is a predetermined value (for example, 5) or more. In the example of FIG. 14, since the integrated value of closed eyes (close) is 7, it is determined to be dozing. It becomes possible to detect the dozing state by the awakening degree obtained in this way, and it is possible to accurately operate the alarm device, cancel the dozing state, and prevent a dozing accident or the like.

Finally, the first step in step 44 of FIG.
Point, and the second step 55 in FIG.
Details of the point extraction will be described with reference to a flowchart. Here, with respect to the differential value data on the specified line as shown in FIG. 15, the point A at which the differential value is inverted from negative to positive is extracted. In the negative differential value update section (negative differential value section) until reaching the extracted point A, the minimum value, which is the peak value of the differential value, is equal to or less than a predetermined value, point A
In the subsequent positive differential value update section (positive differential value section), it is determined whether or not the maximum differential value that is the peak value of the differential value is equal to or greater than a predetermined value, that is, the peak point B in the negative differential value update section and the positive differential value. When the peak point C in the update section satisfies the condition of entering the hatched portion, the extracted point A is set as the first or second point.

Therefore, in the eye position detection circuit 25 and the open / closed eye detection circuit 26, in addition to the point memory for storing the point data, the memory slmin, the memory slmax,
A memory oksld, a counter fl0, a counter m, and a flag flmax are provided. The memory slmin stores the minimum differential value in the negative differential value update section. The memory slmax stores the maximum differential value in the positive differential value update section. The counter fl0 has a differential value of 0,
In addition, the number is counted when continuously detected. The counter m counts the number of extraction points for each line. The flag flmax is set when the differential value changes from negative to positive. Memory oks
ld stores the Y coordinate of the extracted point.

First, in step 81 of FIG. 16, the line to be the point extraction target is specified and the point extraction process is started. At step 82, each memory, counter, and flag are initialized. This allows the memory s
lmin is set to 0, memory slmax is set to 1000, counter fl0 is set to 0, flag flmax is set to 0, and counter m is set to 0. In step 83, it is checked whether or not the size determination has been performed for all the differential values on one line. If the magnitude determination has not been performed for all differential values, the process proceeds to step 84.

At step 84, it is judged whether or not the differential value to be judged is negative, and if negative, the routine proceeds to step 85, where the differential value is smaller than the differential value stored in the memory slmin. Determine whether or not. If the differential value of the determination target is not smaller than the stored value, the process proceeds to step 87 in FIG. 17, and if the differential value of the determination target is smaller than the stored value, the stored value in the memory slmin is updated in step 86, and the stored value of FIG. Go to step 87.

In step 87, it is checked whether the flag flmax is set, that is, whether the flag flmax is 1. If the flag flmax is 0, the process proceeds to step 810. If the flag flmax is 1, the process proceeds to steps 88 and 89,
Go to step 810. At step 810, each memory, flag and counter are initialized. This is for deleting the data at the time of the previous detection or the memory due to noise during the detection in the process of keeping each memory, the flag, and the counter in the initial state when the differential value is negative. Step 81
In 1, the judgment target is advanced to the next differential value in the Y coordinate direction,
Returning to step 83, the above process is repeated for the differential value with the advanced Y coordinate. In this way, the same processing is continued until the differential value is not negative. During the processing during this period, the negative differential value updating section (negative differential value section) of FIG. 15 is formed, and the minimum differential value (peak point B) in the section is detected and stored in the memory slmin.

If it is detected in step 84 that the differential value to be judged is not negative, step 8 in FIG.
Proceed to 13. In step 813, it is determined whether or not the differential value to be determined is 0. When the differential value is 0, the process proceeds to step 814.

In step 814, first, the counter fl
It is checked whether or not the count value of 0 has become 6, and if the count value is less than 6, the process proceeds to step 816 and the counter f
When the count value of 10 becomes 6, in step 815,
The flag flmax and the counter fl0 are initialized, and the process proceeds to step 816. In step 816, the counter fl
It increments 0, advances the determination target to the next differential value in the Y coordinate direction in step 817, and returns to step 83.

When the differential value of 0 appears continuously, the above processing is repeated, and when the count value of the counter fl0 reaches 6 in step 814, the flag flmax and the counter fl which are the extraction conditions of points are calculated in step 815.
0 is initialized and the flag flmax and the counter fl0 are set to 0. The continuous appearance of the differential value of 0 (the above-mentioned treatment is repeated 6 times) means that a portion which is seen in the hair portion and has no change in density continues for a long time. Flag flma
By initializing x and the counter fl0, a flag flmax, which will be described later, appears at a place where the hair becomes black.
The purpose is to cancel the setting and prevent the hair from being extracted as points.

If it is detected in step 813 that the differential value is not 0, the process proceeds to step 91 in FIG. In step 91, the memory slmin is updated in the negative differential value update interval.
It is checked whether or not the condition that the minimum differential value stored in 1 is smaller than the predetermined value and the flag flmax is set to 0 is satisfied. If satisfied, the process proceeds to step 92, and if not satisfied, the process proceeds to step 93.

Since the process has proceeded to step 91, if the differential value to be judged is positive and the previous differential value is negative, the point A at which the differential value is inverted from negative to positive is caught. Become. If the differential value up to now is negative, the memory slmin of the minimum differential value is updated, and when the stored value satisfies the condition of being a predetermined value or less, the captured point is the minimum differential value. The judgment condition will be satisfied. In step 92, the Y coordinate of the differential value that is inverted from negative to positive is stored in the memory oksld, the flag flmax is set to 1, and the determination target is set in the Y coordinate direction through steps 93, 94, and 95. Advances to the differential value of.

Thereafter, if a positive differential value is input, the flag flmax has been set to 1, so step 93
Then, the differential value to be determined is the memory slma.
The stored value of x is compared, and if it is larger than that, step 9 is executed.
In step 4, the stored value is updated and the process proceeds to step 95. The memory slmax is initialized to 1000,
The differential value of 1000 or more will be updated. This reduces the replacement of data and speeds up the processing time. The differential value to be determined in step 93 is the memory sl
If it is smaller than the stored value of max, the routine proceeds to step 95. In step 95, the memory slmin and the counter f
Initialize 10

In step 96, the judgment target is advanced to the next differential value in the Y coordinate direction, the process returns to step 83, and the above processing is repeated for the differential value in which the Y coordinate is advanced. This process is repeated until the differential value becomes negative again. During the process, the positive differential value update section in FIG. 15 is formed, and the maximum differential value (peak point C) during that period is detected and stored in the memory slmax.

Then, the negative differential value appears again, and as shown in FIG.
17 is detected, the process proceeds to step 87 of FIG. 17 by the determination of step 85. Step 87
Then, it is determined whether or not the flag flmax is set to 1. If the flag flmax is not 1, it is determined that there is no extraction point and the process proceeds to step 810.
If it is 1, it is determined in step 88 whether the stored value of the memory slmax is equal to or larger than a predetermined value. If it is equal to or larger than the predetermined value, the pixel stored in the memory oksld satisfies the point extraction condition and the line number and Two-dimensional data composed of Y coordinate values is stored in the point memory. The counter m is incremented to count the extraction points on the same line, and the process proceeds to step 810. If the determination in step 88 is that the stored value of the memory slmax is not equal to or more than the predetermined value, it is determined that there is no extraction point and the process proceeds to step 810.

In step 810, the memory oksld,
Memory slmax, counter fl0, flag flmax
Is initialized and the process proceeds to step 811, where the determination target is advanced to the next differential value in the Y coordinate direction, the process returns to step 83, and the above process is performed again. Finally step 83
When it is determined that all differential values have been determined for the Y coordinate in step 812, the process proceeds to step 812 and determination is performed for the next line.

The present embodiment is configured as described above, and when a pixel representing a density peak and having a differential value that changes from negative to positive is extracted as a point, the maximum change (minimum change in density value) until the pixel is reached is extracted. Differential value) and the maximum change in density (maximum differential value) after the point are compared with a predetermined value in the negative differential value update section and the positive differential value update section, respectively, to extract points, so that the eyes, eyebrows, mouth As a result, a dark part with a clear outline is extracted. The extracted points correspond to their approximate center positions. In addition, since the minimum and maximum differential values are the extracted data of the points, the interval of the pixels that represent them when detecting the eye opening is the eye opening, and the eye opening can be detected without any special processing. it can. It is possible to sufficiently cope with the difference in the position of the face due to the individual difference, and it is possible to stably detect the dozing state.

Next, the second embodiment will be described. In this embodiment, the point detection accuracy is further improved. In the first embodiment, a fixed predetermined value is used for determining the magnitude of the minimum differential value and the maximum differential value, but from the viewpoint of improving detection accuracy, the predetermined value for determining the magnitude of the differential value is used. Is preferably set as small as possible. However, if the predetermined value is set to be small, the possibility that hairs, wrinkles, and the like that are not the extraction target will be extracted as points will increase.

Therefore, in this embodiment, the predetermined value (reference value) for determining the magnitude of the maximum differential value is changed according to the magnitude of the minimum differential value. That is, when the minimum differential value is detected within the negative differential value update section as shown in FIG. 20, the size of the predetermined value (reference value) for determining the magnitude of the positive differential value is set based on the minimum differential value to set the maximum differential value. The size of is determined. The other structure is similar to that of the first embodiment.

Therefore, for example, if the differential value of the peak pop point B is large, the predetermined value for detecting the differential value in the positive direction is set large. As a result, a point whose positive and negative differential values do not significantly change like the eye is typified by points and extracted. As shown in the hair shown in FIG. 38, the minimum differential value q
Point 1 where 1 is large and maximum differential value does not increase
Will not be extracted.

Next, the third embodiment will be described. In this embodiment, similarly to the second embodiment, the predetermined value for determining the magnitude of the positive differential value is changed according to the magnitude of the minimum differential value. The extraction of points determines whether or not the maximum change in the density value before reaching point A is a predetermined value or less based on whether the differential value of the peak point B in the minimum differential value update section is a predetermined value or less. When the minimum differential value is less than or equal to the predetermined value, the size of the predetermined value in the positive differential value update section after the point A is set on the basis of this differential value, and the positive differential value exceeding this predetermined value is detected. If so, the point A is extracted as the first or second point, and the Y coordinate value is stored in the memory.

In the first embodiment, the maximum differential value is used as the positive differential value to be compared with the predetermined value in the positive differential value update section. And since the maximum differential value is also used to detect the opening of the eye as indicating the boundary point of the eye, in the case of a subject wearing glasses, when the light of the infrared lamp for night photography is on the lens as a reflection point. , The maximum differential value may not appear at the boundary point. That is, as shown in FIG. 21, two peaks of density change from black to white due to reflection points may occur in the density change portion of the eye from black to white. In such a case, the peak D with a high differential value is used to extract the points.
However, since the peak D is used as a boundary point in the eye opening detection, perfect eye opening detection becomes difficult. In the present embodiment, in order to cope with this, the processing is performed in accordance with the flowcharts of FIGS. 22 to 25 with the pixel C near the boundary point of the eyes, which exceeds the predetermined value, as the extracted data of the point A.

As in the first embodiment, the eye position detecting circuit 23 includes a memory slmin, a memory slmax, a memory oksld, a counter fl0, a counter m, and a flag fl.
max is provided. The memory slmax is used for different purposes and stores the first positive differential value exceeding a predetermined value in scanning along the Y coordinate.

Steps 121 to 128 in FIG. 22.
Is similar to steps 81 to 86 in the first embodiment shown in FIG. 16, the line is specified and each memory, counter and flag are initialized. The minimum value (peak point B) from the negative differential value to be determined is stored in the memory slmin. In step 127, when the minimum value is detected, the memory, the flag, and the counter are kept in the initial state, and the data at the time of the previous detection and the memory due to the noise during the detection are deleted.

When it is detected in step 124 that the differential value to be judged is not negative, the process proceeds to step 1210 in FIG. Steps 1210 to 1213
18 is the same as steps 813 to 817 in the first embodiment of FIG. 18, and if the detection of the differential value of 0 continues 6 times, it is assumed that the hair portion with no density change is detected, and the density described below changes from white. The setting of the flag flmax set at the place where it becomes black is canceled to prevent erroneous detection.

If it is determined in step 1210 that the value is not 0, the process proceeds to step 131 in FIG. Step 1
If the differential value of the determination target is positive and the differential value up to now is negative, the memory slmin of the minimum differential value is updated and the stored value is equal to or less than the predetermined value. When the condition is satisfied, the point A that is inverted from negative to positive becomes the point that satisfies the minimum differential value determination condition.

In step 131, it is determined whether or not the condition that the minimum differential value stored in the memory slmin is smaller than a predetermined value and the flag flmax is not 1 is satisfied. If satisfied, go to step 132 and then step 1
If not satisfied, go to step 133.

In step 132, the Y coordinate of the differential value (point A) which is inverted from negative to positive is stored in the memory oksld. The flag flmax is set to 1. The minimum differential value stored in the memory slmin is multiplied by a coefficient and stored in the memory slmax. Thus, a predetermined value proportional to the minimum differential value is set. In step 133, it is checked whether the flag flmax is 1, and if it is 1, the process proceeds to step 134, and if it is not 1, the process proceeds to step 137.

In step 134, the differential value to be judged is a predetermined value (reference value) which is the value stored in the memory slmax.
It is determined whether or not it is larger, and if not, the process proceeds to step 137. In step 137, the memory slm
in, the counter fl0 is initialized, and the process proceeds to step 138. In step 138, the determination target is advanced to the next differential value in the Y coordinate direction, and the process returns to step 123.

Then, in step 134, the memory slm
When a differential value having a value larger than the stored value of ax is detected, the extraction condition of the point A is satisfied, and step 135
At, the line number and the memory o as two-dimensional data
The Y coordinate stored in ksld is stored in the point memory. After the counter m is incremented to count the extraction points on the same line, the process proceeds to step 136.

In step 136, the memory oksld,
Memory slmax, counter fl0, flag flmax
To initialize. When it is determined in step 123 that the above processing has been performed on all differential values, step 1
At 29, the determination is switched to the next line.

The present embodiment is configured as described above, and sets a predetermined value for judging the magnitude of the maximum differential value based on the magnitude of the minimum differential value, and the first positive differential value exceeding the set value. By using the value as the determination target of the point extraction, even if the density distribution curve is deformed, the point extraction data having the correspondence relationship with the boundary point of the eye is obtained, and the same result as that of the first embodiment is obtained. At the same time, the effect that the eye opening can be detected without including erroneous detection is obtained.

Next, a fourth embodiment will be described. In this embodiment, similarly to the third embodiment, the predetermined value for determining the magnitude of the positive differential value is changed according to the magnitude of the minimum differential value. As shown in FIG. 26, when a differential value larger than a predetermined value (reference value) is detected after the point A, the points are extracted between the peak point B of the minimum differential value and the point C reaching the predetermined value. Despite the appearance of negative differential values,
The interval between the peak points B and C in the Y coordinate direction is determined by the determination value and extraction is performed.

In the above embodiment, the detection of the positive differential value or the maximum differential value exceeding the predetermined value is performed within the positive differential value section. That is, the positive differential value section is used to cover from the eye density peak to the boundary point. However, when there is a bright point in the eyeball, the positive differential value section extends to the bright point, and the point to be detected may not be detected. In this embodiment, a detection section is not provided for a positive differential value exceeding a predetermined value, and whether to extract a point is determined based on the distance between the positive differential value and the minimum differential value. 27-
FIG. 29 is a flowchart of the point detection process. Here, in addition to the configuration shown in the third embodiment, the memory mini
i is provided. The memory minii stores the Y coordinate of the minimum differential value.

Steps 151 to 158 of FIG. 27.
In the same manner as in the above embodiment, a line is specified, each memory,
Initialize counters and flags. The minimum value (peak point B) from the negative differential value to be determined is the memory slm
In addition to being stored in in, the Y coordinate of the minimum differential value is stored in the memory minii in step 156. In step 157, the counter f is detected when the minimum value is detected.
This is a process for keeping 10 in the initial state and deleting the data at the time of the previous detection and the memory due to noise during the detection.

When it is detected in step 154 that the differential value to be judged is not negative, the routine proceeds to step 1510 in FIG. Steps 1510 to 1514
18 is the same as step 813 to step 817 in the first embodiment of FIG. 18, and if the detection of the differential value of 0 continues 6 times, it is assumed that a hair portion with no density change is detected, and the density described below changes from white to white. The setting of the flag flmax set at the place where it becomes black is canceled to prevent erroneous detection.

If it is determined in step 1510 that the value is not 0, the process proceeds to step 161 in FIG. After that, the predetermined value of the positive differential value based on the minimum value is set as in the third embodiment. When a positive differential value exceeding a predetermined value is detected after the point A at which the differential value is inverted from negative to positive is detected, in step 165, the memory mi is read from the Y coordinate.
The Y coordinate of the minimum differential value stored in nii is subtracted to obtain the distance F therebetween, and if the distance is within a predetermined distance by the judgment value (predetermined value), the point A is set to the first or second point in step 166. The line number and the Y coordinate of the point are stored in the point memory. This idea can also be applied to exclude the point C corresponding to the point A from the extraction of the first or second point when the point C does not appear within the predetermined distance from the peak point B in the Y coordinate direction. When the above-described processing is performed on all the differential values, the determination is switched to the next line in step 153.

The present embodiment is configured as described above, and sets a predetermined value for judging the magnitude of the maximum differential value based on the magnitude of the minimum differential value, and the first positive differential value exceeding the set value. The value is used as the point extraction judgment target. Since the distance distance from the minimum differential value is further determined for the determination target, the same effect as in the above-described embodiment can be obtained, and points can be extracted even when the eye density data is interrupted.

Next, a fifth embodiment will be described. In this embodiment, as an improved version of the fourth embodiment, as shown in FIG. 30, before the extraction condition at the point A is satisfied, it is possible to prevent the point A1 having a clearer shade from being excluded. To aim. That is, when there is density data including hair as shown in FIG. 30, in the fourth embodiment, the minimum differential value B exceeding a predetermined value is detected at the point where the hair changes from white to black. When the point A1 desired to be detected with the minimum differential value B is detected and the distance with the positive differential value C exceeding a predetermined value is determined,
The point A1 cannot be extracted because the distance judgment value is exceeded.

Therefore, in this embodiment, after the minimum differential value B and the point A are detected, the second minimum differential value B1 smaller than the minimum differential value B appears before the differential value C reaching the predetermined value appears. In this case, the extraction condition determination at the point A is canceled, and the point extraction is switched to the point A1 having the minimum differential value B1. Others are the same as those in the fourth embodiment.

Here, in addition to the configuration shown in the fourth embodiment, the memory slmin2 and the memory mini2, and the flag fl.
min is provided. The memory slmin2 stores the corresponding value in the minimum differential value update section. Memory mi
nii2 stores the Y coordinate of the corresponding value in the minimum differential value update section. The flag flmin is the memory slmi
It becomes 1 when the stored value update confirmation flag of n2 is set.

31 to 35 are flow charts for extracting points. In steps 181 to 1810, the line is specified and each memory, counter, and flag are initialized, as in the above-described embodiment. The minimum value B from the negative differential value to be determined is stored in the memory slmin, and the Y coordinate of the minimum differential value B is stored in the memory m in step 186.
Store inii.

In steps 187 and 188, the minimum differential value B which is the stored value of the memory slmin in the processing described later.
When the second smaller differential value B1 is detected,
This is a process for switching extraction from point A to point A1. That is, at the same time as confirming the setting of the flag flmax, which allows confirmation of the temporary extraction of the extraction target point A,
The differential value that is the current judgment target is the memory slmin
This is a process of determining whether or not it is smaller than the stored value of 2.

At this time, the differential value to be judged is the memory slm.
If it is larger than the stored value of in2, the process proceeds to step 189, and the normal determination is continued as in the fourth embodiment. If the differential value to be determined is smaller than the storage value of the memory slmin, the second minimum differential value B1 is detected, and step 188 is performed.
Then, the minimum differential value update flag flmin is set, and the differential value to be judged is multiplied by -0.5 to calculate the result in the memory sl.
Remember max. Memory slmin2, memory mini
The magnitude of the second minimum differential value B1 and the Y coordinate are respectively stored in i2 again.

Steps 1812 to 1816 are
Similar to steps 1510 to 1514 of the fourth embodiment, when the differential value of 0 continues 6 times, the setting of the flag flmax set at the place where the hair becomes black is canceled to prevent erroneous detection. . And step 181
If it is determined in 2 that it is not 0, the process proceeds to step 191 in FIG.

In step 191, it is judged whether or not the condition that the stored value of the minimum differential value B stored in the memory slmin is less than a predetermined value and the flag flmax is not 1 is satisfied. If satisfied, go to step 193 via step 192, and if not satisfied, go to step 193. Here, the point A at which the differential value is inverted from negative to positive is provisionally extracted, and the negative differential value update section 1 ends.

At step 192, the Y coordinate of the point A is stored in the memory oksld. Flag flmax
Is set to 1. The memory slma is calculated by multiplying the minimum differential value stored in the memory slmin by a coefficient.
Store in x. The stored value in the memory slmin is stored in the memory slmin2. memory m in mini2
Store the stored value of inii. In this way, the magnitude of the minimum differential value and the Y coordinate are stored in the memory for storing the corresponding value in the minimum differential value updating section, and the predetermined value proportional to the minimum differential value is set.

In step 193, the setting of the minimum differential value update flag flmin is determined. If the flag is set, a point A1 at which the minimum differential value updated in step 194 is positively inverted is detected from the negative interval. Then, the Y coordinate is re-stored in the memory oksld and the flag flmi is stored.
Initialize n. Steps 195 to 1911 are the same as steps 161 to 169 in the fourth embodiment of FIG. 29, and the predetermined value of the positive differential value based on the minimum value is set. When a positive differential value C exceeding a predetermined value is detected after the point A1 at which the differential value is inverted from negative to positive is detected, the minimum differential value B1 stored in the memory minii2 from the Y coordinate is detected in step 197. The Y coordinate is subtracted to obtain the distance F therebetween, and if the judgment is made within the predetermined distance by the judgment value (predetermined value), the point A1 is extracted as the first or second point in step 198, and the line number and the Y coordinate of the point Is stored in the point memory. When the above-described processing is performed on all differential values, the determination is switched to the next line in step 183.

The present embodiment is configured as described above, and in addition to the fourth embodiment, before the positive differential value exceeding the predetermined value is detected, the differential value is further differentiated from the previously detected minimum differential value. When a small second minimum differential value is detected, the point having the second minimum differential value is subjected to the extraction processing, so that it is possible to prevent the point from being unable to be extracted by the distance determination due to the entry of another minimum differential value.

[0105]

As described above, according to the present invention, in the doze state detecting device based on the face image data processing, the first extraction point is extracted for each local increase in the density in the pixel row in the vertical direction of the face, Detecting the eye position by extracting a curve group extending in the lateral direction of the face from adjacent first extraction points of adjacent pixel rows and identifying the curve of the eye from this curve group A second extraction point is similarly extracted along a vertical pixel row in a predetermined region including the eye, and a third boundary point indicating a local increase in density including the second extraction point is extracted.
Since the extraction point is extracted, the eye opening is detected based on the data of the third extraction point continuous in the lateral direction, and the arousal level is determined from the state change of the eye opening, Even if there is a difference in the physique or driving posture of the driver, the eye position can be detected easily and reliably.

According to the second aspect of the present invention, the extraction of points and the identification of the pixel whose differential value is inverted from negative to positive correspond to the peak of the density, and the first point or the second point.
When extracting as the point of, the minimum and maximum differential values are obtained from the negative and positive differential value sections before and after the differential value is inverted, and the strength is determined by a predetermined value, so the extracted point is in the dark part where the boundary is clear. As a representative, dark areas such as wrinkles and shadows where the boundaries are not clearly defined are shaken off from the extraction target, resulting in high quality extraction. The extraction points represent the peaks of the density and correspond to the approximate center positions of the eyes, eyebrows, and mouth. This simplifies subsequent processing.

According to the third aspect of the present invention, when a pixel whose differential value is inverted from negative to positive is extracted as the first point or the second point, the minimum value exceeding the predetermined value from the negative differential value section before the pixel is extracted. Since the strength determination is performed by detecting the differential value and the positive differential value, the same effect as that of the invention of claim 2 is obtained, and the distribution curve of the differential value is distorted or deformed, and the maximum differential value changes in position. Even in such a case, the boundary data between the dark portion and the light portion can be obtained, and the eye opening can be detected without erroneous detection.

If the distance between the pixels of the first positive differential value exceeding the predetermined value for judging the magnitude of the minimum differential value and the positive differential value is equal to or larger than the judgment value, the differential value changes from negative to positive. Excluding the inverted pixel from the extraction of the first or second point prevents erroneous extraction due to hair.

Further, in the first point extraction means or the second point extraction means, before the first positive differential value exceeding a predetermined value for judging the magnitude of the positive differential value is detected, the minimum differential value is detected. When there is another minimum differential value having a value smaller than the value, the pixel whose differential value is inverted from negative to positive is excluded from the extraction of the first or second point, and the distribution of the differential value is obtained. Erroneous extraction due to curve deformation is prevented.

If there are a predetermined number or more of pixels without density change following the pixel whose differential value is inverted from negative to positive, the pixel whose differential value is inverted from negative to positive is first Alternatively, by excluding the second point from the extraction, erroneous extraction due to deformation of the differential value distribution curve is prevented. Then, by changing the magnitude of the predetermined value for determining the magnitude of the maximum differential value or the positive differential value according to the minimum differential value, the point can be adjusted with higher accuracy according to the density change feature of the target to be detected. It is to extract.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of the present invention.

FIG. 2 is a block diagram showing a configuration of a first exemplary embodiment of the present invention.

FIG. 3 is a flowchart showing a processing operation of a doze state detection.

FIG. 4 is a flowchart showing a processing operation of eye position detection.

FIG. 5 is a diagram illustrating a pixel row for eye position detection.

FIG. 6 is a diagram illustrating a first point extraction principle.

FIG. 7 is a diagram illustrating extraction of a curve group.

FIG. 8 is a diagram illustrating a first extraction point grouping process.

FIG. 9 is a diagram illustrating a procedure for detecting an eye position from a group of curves.

FIG. 10 is a flowchart showing a processing operation of eye opening detection.

FIG. 11 is a diagram illustrating a principle of extracting a second point.

FIG. 12 is a diagram illustrating detection of an opening when a bright spot is located at the center of an eye.

FIG. 13 is a diagram illustrating an eye tracking state.

FIG. 14 is a diagram illustrating a procedure for determining an awakening degree.

FIG. 15 is a diagram illustrating a point extraction procedure.

FIG. 16 is a flowchart showing a processing operation of point extraction.

FIG. 17 is a flowchart showing a processing operation of point extraction.

FIG. 18 is a flowchart showing the processing operation of point extraction.

FIG. 19 is a flowchart showing the processing operation of point extraction.

FIG. 20 is a diagram illustrating a point extraction procedure according to the second embodiment of the present invention.

FIG. 21 is a diagram illustrating a point extraction procedure in the third embodiment of the present invention.

FIG. 22 is a flowchart showing the processing operation of point extraction.

FIG. 23 is a flowchart showing the processing operation of point extraction.

FIG. 24 is a flowchart showing the processing operation of point extraction.

FIG. 25 is a flowchart showing the processing operation of point extraction.

FIG. 26 is a diagram illustrating a point extraction procedure in the fourth embodiment of the present invention.

FIG. 27 is a flowchart showing the processing operation of point extraction.

FIG. 28 is a flowchart showing the processing operation of point extraction.

FIG. 29 is a flowchart showing the processing operation of point extraction.

FIG. 30 is a diagram illustrating a point extraction procedure in the fifth embodiment of the present invention.

FIG. 31 is a flowchart showing the processing operation of point extraction.

FIG. 32 is a flowchart showing a processing operation of point extraction.

FIG. 33 is a flowchart showing a processing operation of point extraction.

FIG. 34 is a flowchart showing the processing operation of point extraction.

FIG. 35 is a flowchart showing the processing operation of point extraction.

FIG. 36 is a diagram showing a conventional example.

FIG. 37 is a diagram showing another conventional example.

FIG. 38 is a diagram showing a change in density of pixels including a hair portion and a change in differential value thereof.

[Explanation of symbols]

1 Image input means 2 Concentration detection means 3 First point extraction means 4 Curve group extraction means 5 Eye position detection means 6 Second point extraction means 7 Continuity judgment means 8 Eye opening detection means 9 Arousal level judgment means 21 TV camera 22 A / D converter 23 Image memory 24 Image data operation circuit 25 eye position detection circuit 26 Open / closed eye detection circuit 27 Awakening level judgment circuit 28 Alarm device

Claims (7)

(57) [Claims] 1. A doze state detection device for processing face image data to detect a doze state, comprising: image input means for inputting a face image; and detecting pixel density along a vertical pixel row of the face. Adjacent to the density detection means, a first point extraction means for identifying a pixel having a density differential value before and after a unidirectional peak of the density along the pixel row exceeding a predetermined value, and extracting a first point, Curve group extracting means for connecting the first points that are close to each other in the pixel row direction of the pixel row to extract a curve group extending in the lateral direction of the face; An eye position detecting means for detecting a position, and a pixel in which a density differential value before and after a unidirectional peak of density along a vertical pixel row within a predetermined region including the eye exceeds a predetermined value is specified. Second to extract 2 points
Point extraction means, continuity determination means for determining the continuity of the face in the lateral direction depending on the presence or absence of the second points that are adjacent to each other in the pixel row direction of the adjacent pixel row, and the extracted data of the continuous second points. An eye opening detection means for detecting an eye opening based on the above, and a doze state detecting device characterized by determining an awakening degree from a change in an open / closed state of the eye based on the eye opening. 2. The first point extracting means or the second point extracting means obtains a minimum differential value of a density exceeding a predetermined value from a negative differential value section by scanning along the pixel row, and the differential value is By storing the pixel that is inverted from negative to positive and detecting the maximum differential value that exceeds a predetermined value different from the above from the subsequent positive differential value section, the pixel whose differential value is inverted from negative to positive is extracted. The dozing state detection device according to claim 1, wherein the first or second point is set. 3. The first point extracting means or the second point extracting means obtains a minimum differential value of a density exceeding a predetermined value from a negative differential value section by scanning along the pixel row, and the differential value is A pixel in which the differential value is inverted from negative to positive is extracted by storing a pixel that is inverted from negative to positive and detecting a positive differential value that exceeds a predetermined value different from the above from a predetermined interval that follows. Alternatively, the second point is set, and the dozing state detection device according to claim 1. 4. The first point extracting means or the second point extracting means is between pixels of the minimum differential value and the first positive differential value exceeding a predetermined value for judging the magnitude of the positive differential value. The doze state detection device according to claim 3, wherein when the distance is equal to or larger than a determination value, the pixel whose differential value is inverted from negative to positive is excluded from the extraction of the first or second point. 5. The first point extraction means or the second point extraction means is configured to detect the minimum differential value before a first positive differential value exceeding a predetermined value for determining the magnitude of the positive differential value is detected. 4. When there is another minimum differential value having a value smaller than the value, the pixel whose differential value is inverted from negative to positive is excluded from the extraction of the first or second point. Or the doze state detection device according to 4. 6. The first point extraction means or the second point extraction means has a predetermined number or more of 0 differential value pixels having no density change following the pixel whose differential value is inverted from negative to positive. In that case, the dozing state detection device according to claim 2, 3, 4 or 5, wherein a pixel whose differential value is inverted from negative to positive is excluded from the extraction of the first or second point. 7. The predetermined value for determining the magnitude of the maximum differential value or the positive differential value has a magnitude that changes according to the minimum differential value.
The doze state detection device according to 4 or 5.