JP4529635B2 - Open / close eye determination device - Google Patents

Description

  The present invention relates to an open / closed eye determination device.

  Conventionally, an upper eyelid portion is detected from a face image acquired by a CCD camera, a straight line connecting both end points of the upper eyelid is obtained, and based on the ratio of the area of the portion located above the straight line and the area of the eye An open / close eye determination device that determines open / close of an eye is known.

Further, this apparatus is configured to determine whether the eye is opened or closed based on whether or not the upper eyelid is convex upward. Furthermore, this device is configured to determine whether the eye is opened or closed based on the positional relationship between a straight line connecting both end points of the upper eyelid and the center of gravity of the upper eyelid (see, for example, Patent Document 1).
JP 2000-123188 A

  However, in the conventional apparatus, when there is noise (eg, wrinkles) around the eyes, the upper eyelid line obtained by image processing may be disturbed and normal curvature may not be calculated. This tendency is particularly strong when the eyes are opened, and it is easy to erroneously determine that the eyes are closed.

  The present invention has been made to solve such a conventional problem, and an object of the present invention is to provide an open / closed eye determination device capable of preventing erroneous determination that the eyes are closed when the eyes are opened. There is.

  According to the present invention, the open / close eye determination device includes an eye image acquisition unit, an upper eyelid detection unit, an upper eyelid coordinate extraction unit, a curve approximation unit, a correlation value detection unit, an open / closed eye determination threshold setting unit, an upper curvature detection unit, And an opening / closing eye determination means. The eye image acquisition means acquires an eye image including the eye of the person to be detected. The upper eyelid detection means detects the upper eyelid from the eye image acquired by the eye image acquisition means. The upper eyelid coordinate extracting means extracts the coordinate group constituting the upper eyelid from the upper eyelid detected by the upper eyelid detecting means, and the curve approximation means is the upper eyelid coordinate extracting means extracted by the upper eyelid coordinate extracting means. The upper heel is approximated to a quadratic or higher curve from the coordinate group of the heel. The correlation value detecting means detects a correlation value between the upper eyelid coordinate group extracted by the upper eyelid coordinate extracting means and a quadratic or higher curve approximated by the curve approximating means. When the correlation value detected by the correlation value detection unit is higher than a reference value, the opening / closing eye determination threshold setting unit is more open / closed eye determination threshold than when the correlation value detected by the correlation value detection unit is less than or equal to the reference value. Is set to a large value. The upper curvature detection means detects the curvature of the upper curvature from the coordinate group extracted by the upper eye coordinate extraction means. The open / close eye determination means determines that the upper eyelid curvature detected by the upper curvature detection means is equal to or greater than the open / close eye determination threshold set by the open / close eye determination threshold setting means, and satisfies the open / close eye determination threshold. If not, the eye is determined to be closed.

  According to the present invention, the open / close eye determination device obtains a correlation value between the upper eyelid coordinate group and a quadratic or higher curve approximated to the coordinate group. Here, the case where the correlation value is high indicates that it is easy to accurately determine the open / closed eye due to circumstances such as little noise around the eye. On the other hand, when the correlation value is low, it indicates that it is easy to make an erroneous determination with respect to the open / closed eye due to a lot of noise around the eye.

  In addition, when the correlation value is higher than the reference value, the open / close eye determination threshold is set larger than when the correlation value is equal to or lower than the reference value. For this reason, when the correlation value is higher than the reference value, that is, when it is easy to accurately determine the open / closed eye, the open / closed eye determination threshold value is increased. On the other hand, when the correlation value is less than or equal to the reference value, that is, when it is easy to make an erroneous determination for the open / closed eye, the open / closed eye determination threshold value is decreased.

  Further, the curvature of the upper eyelid is obtained, and it is determined that the eye is open when the curvature is larger than the open / closed eye determination threshold, and the closed eye is determined when the curvature is equal to or less than the open / closed eye determination threshold. Here, when there is a lot of noise around the eyes, the accuracy of detection of the curvature of the upper eyelid also decreases. That is, when the correlation value is small, the accuracy of detection also decreases for the curvature of the upper eyelid. For this reason, when the eye is opened, the curvature of the upper eyelid may be detected small. However, in the present invention, when the correlation value is less than or equal to the reference value, the open / close eye determination threshold is reduced. Therefore, even if the curvature of the upper eyelid is detected to be small when the eye is opened, the open / close eye determination threshold value is made small, so that the possibility of erroneously determining that the eye is closed when the eye is opened can be reduced.

  Therefore, it is possible to prevent erroneous determination that the eyes are closed when the eyes are opened.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described with reference to the drawings.

  FIG. 1 is a functional block diagram showing a configuration of an open / closed eye determination device according to an embodiment of the present invention. As shown in the figure, the open / close eye determination device 1 determines whether the eye of the person to be detected is in an open state or in a closed state, and includes an eye image acquisition unit CL1 and an upper eyelid detection unit CL2. And upper eyelid coordinate extracting means CL3. The open / close eye determination apparatus 1 includes a curve approximation means CL4, a correlation value detection means CL5, and an open / close eye determination threshold setting means CL6. Further, the open / close eye determination device 1 includes an upper curvature detection means CL7 and an open / close eye determination means CL8.

  The eye image acquisition unit CL1 acquires an eye image including the eyes of the person to be detected. The eye image acquisition means CL1 is configured to transmit the input eye image data to the upper eyelid detection means CL2.

  The upper eyelid detection means CL2 detects the upper eyelid from the eye image acquired by the eye image acquisition means CL1. The upper eyelid detection means CL2 is configured to transmit the detected upper eyelid data to the upper eyelid coordinate extraction means CL3.

  The upper eyelid coordinate extracting means CL3 extracts a coordinate group constituting the upper eyelid from the upper eyelid detected by the upper eyelid detecting means CL2. The upper eyelid coordinate extracting means CL3 is configured to transmit the data of the coordinate group constituting the detected upper eyelid to the curve approximating means CL4, the correlation value detecting means CL5, and the upper curvature detecting means CL7.

  The curve approximation means CL4 approximates the upper eyelid to a quadratic or higher curve from the upper eyelid coordinate group extracted by the upper eyelid coordinate extraction means CL3. The curve approximating means CL4 approximates a quadratic linear polynomial from the upper eyelid coordinate group extracted by the upper eyelid coordinate extracting means CL3, for example, by the least square method. Further, the curve approximating means CL4 is configured to transmit information on the approximated curve approximating line to the correlation value detecting means CL5.

  The correlation value detection means CL5 detects the correlation value between the upper eyelid coordinate group extracted by the upper eyelid coordinate extraction means CL3 and the curve approximation line approximated by the curve approximation means CL4. When the correlation value is detected, the correlation value detection means CL5 is configured to transmit this data to the open / close eye determination threshold setting means CL6.

  When the correlation value detected by the correlation value detection unit CL5 is higher than the reference value, the opening / closing eye determination threshold setting unit CL6 is an opening / closing eye determination threshold that is a reference for the opening / closing eye determination than when the correlation value is equal to or less than the reference value. Is set to a large value. The opening / closing eye determination threshold value setting means CL6 is configured to transmit information on the set opening / closing eye determination threshold value to the opening / closing eye determination means CL8.

  The upper curvature detection means CL7 detects the curvature of the upper curvature from the coordinate group extracted by the upper eyelid coordinate extraction means CL3. The upper eyelid curvature detection means CL7 is configured to transmit the detected upper eyelid curvature data to the open / close eye determination threshold value setting means CL6 and the open / close eye determination means CL8.

  The open / close eye determination means CL8 determines that the eyelid is open when the curvature of the upper eyelid detected by the upper curvature detection means CL7 is equal to or greater than the open / close eye determination threshold set by the open / close eye determination threshold setting means CL6, and the open / close eye determination When the threshold is not reached, it is determined that the eye is closed. The open / close eye determination means CL8 is connected to a dozing detection device or the like, and is configured to transmit the determined result of the open / close eye to the device.

  Here, the eye image acquisition means CL1 has the following configuration in detail. That is, as shown in FIG. 1, the eye image acquisition unit CL1 includes a face image acquisition unit CL11 and an eye detection unit CL12. The face image acquisition means CL11 is installed substantially in front of the detected person and images the entire face of the detected person. The eye detection means CL12 detects the eye of the person to be detected from an image including the entire face imaged by the face image imaging means CL11. The eye image acquisition unit CL1 is configured to extract a minute image that includes the eye detected by the eye detection unit CL12 and is smaller than the face image, from the face image, and forms the eye image.

  The opening / closing eye determination threshold value setting means CL6 is as follows in detail. That is, as shown in FIG. 1, the open / close eye determination threshold value setting means CL6 includes first calculation means CL61 and second calculation means CL62. The first calculation means CL61 calculates a first determination threshold value set from the representative value of the curvature of the upper eyelid when the eye is opened. The second calculation means CL62 calculates a second determination threshold value set from a representative value of the curvature of the upper eyelid when the eye is closed.

  Further, as shown in FIG. 1, the first calculation means CL61 has a first sampling means CL611 and a first representative value calculation means CL612. The first sampling means CL611 samples a predetermined number of curvatures of the upper eyelid. The first representative value calculation means CL612 calculates one of an average value, a median value, and a mode value from a predetermined number of curvatures sampled by the first sampling means CL611, and the calculated value is used as the first statistical value. Representative value.

  Further, as shown in FIG. 1, the second calculation means CL62 includes a second sampling means CL621 and a second representative value calculation means CL622. The second sampling means CL621 samples a predetermined number of curvatures of the upper eyelid. The second representative value calculating unit CL622 calculates one of an average value, a median value, and a mode value from a predetermined number of curvatures sampled by the second sampling unit CL621.

  The first calculation unit CL61 and the second calculation unit CL62 will be described in detail in a flowchart described later.

  FIG. 2 is a hardware configuration diagram of the open / close eye determination device 1 according to the embodiment of the present invention. As shown in the figure, a TV camera 2 is provided on an automobile instrument. The TV camera 2 is installed at a position where the driver can be imaged from substantially the front, and at least captures the entire face of the driver. That is, the TV camera 2 constitutes the face image acquisition means CL11 described with reference to FIG.

  In the present embodiment, the input image of the TV camera 2 includes, for example, a horizontal direction (X) 640 pixels and a vertical direction (Y) 480 pixels. An input image captured by the TV camera 2 is input as image data to a microcomputer 3 (hereinafter referred to as a microcomputer) installed inside the vehicle body such as the back side of the instrument.

  The microcomputer 3 includes eye detection means CL12, upper eyelid detection means CL2, upper eyelid coordinate extraction means CL3, curve approximation means CL4, correlation value detection means CL5, open / closed eye determination threshold setting means CL6, upper curvature detection means CL7, and Program logic that constitutes the open / close eye determination means CL8 is programmed. And if the microcomputer 3 inputs the data of the image containing the whole face from the TV camera 2, a program will be run and it will determine a driver | operator's opening and closing eyes.

  Further, the TV camera 2 or the like may have the configuration shown in FIG. FIG. 3 is a hardware configuration diagram showing another example of the face image acquisition means CL11. In the case of this example, the TV camera 2 as the face image acquisition means CL11 is attached to the driver's head. The TV camera 2 is configured to take an image of the eye through the mirror 4. Thus, by imaging only the peripheral part of the eye via the mirror 4, the eye detection means CL12 shown in FIG. 1 is not required, and the configuration and processing can be simplified.

  Note that the TV camera 2 may be installed at the position of the mirror 4 and directly take an image of the eye. Furthermore, the TV camera 2 may be installed on the instrument, and may be configured to image the eye while automatically tracking the viewpoint position to the driver.

  Here, an outline of the operation of the open / closed eye determination device 1 according to the present embodiment will be described with reference to FIG. First, the eye image acquisition unit CL1 acquires an eye image including the eye by imaging the face of the person to be detected. Next, the upper eyelid detection means CL2 detects the upper eyelid of the detected person.

  Then, the upper eyelid coordinate extracting means CL3 extracts a coordinate group for the upper eyelid of the detected person detected by the upper eyelid detecting means CL2. The curve approximation means CL4 approximates the upper eyelid to a quadratic or higher curve from the upper eyelid coordinate group extracted by the upper eyelid coordinate extraction means CL3. Next, the correlation value detection means CL5 detects the correlation value between the upper eyelid coordinate group and the approximated quadratic or higher curve.

  Here, the correlation value detection means CL5 will be described in detail with reference to FIG. 4A and 4B are explanatory diagrams showing the state of the upper eyelid, in which FIG. 4A shows an example in which there is little noise around the eye, and FIG. 4B shows an example in which there is much noise around the eye. . FIGS. 4A and 4B also show the state when the eye part is image-processed.

  First, as shown in FIG. 4A, when there is little noise around the eye, the upper eyelid coordinate group extracted by the upper eyelid coordinate extracting means CL3 is convex upward as is apparent from the state of image processing. It becomes an arc shape. Further, the curve approximation line approximated by the curve approximation means CL4 also has an upwardly convex arc shape, as is apparent from the description relating to the curve approximation means CL4 described later. Accordingly, when there is little noise around the eyes, the correlation value between the upper eyelid coordinate group and the curve approximation line becomes high.

  On the other hand, as shown in FIG. 4B, when there is a lot of noise around the eye, as is clear from the state of image processing, the coordinate group of the upper eyelid is not detected in an upwardly convex arc shape, Detected as a broken line. Since the curved approximate line has an upwardly convex arc shape, when there is a lot of noise around the eyes, the correlation value between the upper eyelid coordinate group and the curved approximate line is low.

  Refer to FIG. 1 again. The upper curvature detection means CL7 detects the curvature of the upper curvature from the coordinate group extracted by the upper eyelid coordinate extraction means CL3. Here, when there is little noise around the eye, the upper curvature detection means CL7 is easy to detect the curvature accurately, and when there is a lot of noise around the eye, the upper curvature detection means CL7 is difficult to detect the curvature accurately. Become.

  Here, the curvature and the correlation value have the following relationship. FIGS. 5A and 5B are explanatory diagrams showing the relationship between the curvature radius of the upper eyelid and the correlation value. FIG. 5A shows an example in which there is little noise around the eye, and FIG. 5B shows that there is much noise around the eye. An example of the case is shown. In FIG. 5, the vertical axis represents the radius of curvature, and the horizontal axis represents the correlation value.

  First, when there is little noise around the eyes, as shown in FIG. 5A, the correlation value is high and concentrated near “1”. That is, when there is little noise around the eyes, both the coordinate group of the upper eyelid and the curve approximation line can be obtained with high accuracy, so the correlation value is high and concentrated near “1”. In addition, when the noise is low, the curvature of the upper eyelid is also accurately obtained, so that the curvature values are different between when the eyes are opened and when the eyes are closed. That is, as shown in FIG. 5 (a), the radius of curvature when the eye is opened shows a value of "100" or less, and the radius of curvature when the eye is closed shows a value larger than "100". Therefore, the opening / closing eye determination device 1 can accurately determine the opening / closing eye by setting the opening / closing eye determination threshold value so that the curvature radius becomes “100”.

  On the other hand, when there is a lot of noise around the eyes, as shown in FIG. 5 (b), the correlation value varies from about “0.4” to “1”. That is, when there is a lot of noise around the eyes, the correlation value may be a low value such as “0.4” due to the influence of noise. Also, when there is a lot of noise, it is difficult to accurately determine the curvature of the upper eyelid. Therefore, the radius of curvature at the time of eye opening shows a value of “170” or less when the correlation value is higher than about “0.85”, but when the correlation value is about “0.85” or less, about “ 350 "vicinity is detected. Similarly, when the correlation value is higher than about “0.85”, the radius of curvature when the eye is closed shows a value of “170” or more, but when the correlation value is about “0.85” or less, about “ 350 "vicinity is detected.

  For this reason, even if the open / closed eye determination threshold is set so that the radius of curvature is “170”, the open / closed eye determination device 1 cannot accurately perform the open / closed eye determination, and the detected person is in the open state. Nevertheless, it may be erroneously determined to be closed.

  Therefore, when the correlation value detected by the correlation value detection unit CL5 is higher than the reference value, the opening / closing eye determination threshold setting unit CL6 sets the opening / closing eye determination threshold larger than when the correlation value is equal to or lower than the reference value. It is said.

  Specifically, in the case of the example shown in FIG. 5B, the apparatus 1 sets the reference value as shown in FIG. 6 and the opening / closing eye determination threshold as shown in FIG. FIG. 6 is an explanatory diagram of a reference value, and FIG. 7 is an explanatory diagram of an open / closed eye determination threshold value. In FIG. 6, the vertical axis indicates the appearance frequency, and the horizontal axis indicates the correlation value. In FIG. 7, the vertical axis indicates the radius of curvature, and the horizontal axis indicates the correlation value.

  First, as shown in FIG. 6, even when there is a lot of noise around the eyes, the correlation value exceeds approximately “0.85”. When the correlation value exceeds “0.85”, as shown in FIG. 5B, the eye opening and the eye closing are separated with a curvature radius of about “170” as a boundary. Therefore, when the correlation value exceeds “0.85”, it is possible to accurately determine whether the eye is open or closed by setting the open / close eye determination threshold value to have a curvature radius “170” as shown in FIG.

  On the other hand, when the correlation value is “0.85” or less, as shown in FIG. 5 (b), it is difficult to distinguish between the open eye and the closed eye, but the curvature radius of the upper eyelid exceeds “350” when the eye is opened. There is nothing. Accordingly, when the correlation value is “0.85” or less, as shown in FIG. 7, if the opening / closing eye determination threshold is set to have the curvature radius “350”, the eye opening is not erroneously determined to be closed. It becomes.

  From the above, when the correlation value exceeds “0.85”, the open / close eye determination threshold value setting means CL6 sets the open / close eye determination threshold value with the curvature radius “170”. In addition, when the correlation value is “0.85” or less, the open / close eye determination threshold value setting unit CL6 sets the open / close eye determination threshold value with the curvature radius “350”. Thereby, this apparatus 1 reduces the possibility of erroneously determining that the eyes are closed when the eyes are opened.

  The opening / closing eye determination means CL8 determines that the upper eyelid curvature detected by the upper curvature detection means CL7 is greater than or equal to the opening / closing eye determination threshold by the opening / closing eye determination threshold setting means CL6, and determines the opening / closing eye determination threshold. It is determined that the eye is closed when it is less than.

  Note that the values of the above-mentioned curvature radii “170” and “350” are suitable open / closed eye determination threshold values in the case shown in FIG. 5B, and are values when other subjects are imaged. Needless to say that is different.

  Next, a detailed operation of the open / close eye determination device 1 according to the present embodiment will be described. In addition, although this apparatus 1 can be used for detection of opening / closing eyes of a driver of a car, a railway vehicle, a ship, a plant operator, etc., in the following description, it is used for detection of opening / closing eyes of a driver of a car. The open / closed eye determination device 1 will be described as an example.

  FIG. 8 is a main flowchart showing an outline of the operation of the open / close eye determination device 1 according to the first embodiment. As shown in the figure, first, when the process is started, the microcomputer 3 executes an initial value input process (ST10). In this initial value input process, various constants such as sampling time are read.

  Thereafter, the microcomputer 3 initializes the processing frame counter “i” to “0” (ST11). After initialization, the microcomputer 3 executes an end determination process (ST12). At this time, the microcomputer 3 makes a determination based on, for example, whether the engine is activated.

  Then, the microcomputer 3 determines whether it is “STOP” (ST13). For example, if it is determined that the engine is not activated, the microcomputer 3 determines that it is “STOP” (ST13: YES), and the process ends.

  On the other hand, when it is determined that it is not “STOP” due to the engine being started and running (ST13: NO), the microcomputer 3 executes an eye image acquisition process (ST14). Thereby, in the configuration example shown in FIG. 2, the TV camera 2 captures the entire face of the driver, and the microcomputer 3 acquires an eye image from the image of the entire face. In the case of the configuration shown in FIG. 3, the TV camera 2 directly acquires an eye image. This process is performed by the microcomputer 3 executing a program corresponding to the eye image acquisition means CL1.

  FIG. 9 is a flowchart showing a detailed operation of the eye image acquisition process (ST14) shown in FIG. When the apparatus 1 employs the configuration shown in FIG. 2, an eye image is acquired by the following process.

  First, as shown in the figure, if “NO” is determined in step ST13, the TV camera 2 acquires a face image by imaging the entire face of the driver (ST30). Then, the microcomputer 3 executes a process for specifying the candidate eye position (ST31). With this process, one or a plurality of candidate eye positions are specified from the entire image. Specifically, one or more candidate positions having the possibility of being the left eye and the right eye are specified from the entire image.

  Thereafter, the microcomputer 3 executes an eye determination process (ST32). That is, it is determined whether or not one of the candidates identified in step ST31 is an eye. In this process, when the microcomputer 3 determines that the candidate is an eye, the microcomputer 3 extracts a minute image that includes the eye and is smaller than the face image from the face image, and sets the image as an eye image.

  Thereafter, the microcomputer 3 determines whether or not the candidate is determined to be an eye based on the result of the eye determination process (ST32) (ST33). If the candidate is not determined to be an eye in step ST32 and an eye image has not been acquired, the microcomputer 3 determines that the candidate is not an eye (ST33: NO). Then, the microcomputer 3 determines whether or not all of the candidates specified in step ST31 have been determined (ST34). If all the determinations are made (ST34: YES), the process proceeds to step ST15 in FIG.

  On the other hand, when it has not determined for all (ST34: NO), the process returns to step ST32. In step ST32, the microcomputer 3 selects another candidate and determines again whether or not the selected candidate is an eye.

  By the way, when it is determined in step ST32 that the candidate is an eye and an eye image is extracted, the microcomputer 3 determines that the candidate is an eye (ST33: YES). And the microcomputer 3 acquires the eye image extracted in step ST32, and a process returns to step ST15.

  As described above, the present apparatus 1 acquires an eye image. Note that the process (ST31) and the eye determination process (ST32) for identifying a candidate having the possibility of being an eye are performed as follows when the microcomputer 3 executes a program corresponding to the eye detection means CL12. .

  FIG. 10 is a flowchart showing details of the eye candidate position specifying process (ST31) shown in FIG. In the figure, first, the microcomputer 3 stores the entire data of the captured image as an entire image in the image memory (ST40).

  Next, the microcomputer 3 performs the determination in step ST41. This determination will be described later. If “NO” is determined in step ST41, the microcomputer 3 performs an arithmetic mean calculation of the density values along only one line in the pixel row in the vertical direction (Y-axis direction) of the entire image (ST42).

  This arithmetic average calculation is a process of obtaining an average density value for a predetermined number of pixels arranged in the vertical direction, for example, and setting the density value of one pixel out of the predetermined number of pixels as an average value. For example, when the predetermined number is “5”, the first to fifth pixels from the top of the screen are selected to obtain an average value, and this average value is set as the density value of the fifth pixel. Next, the second to sixth pixels from the top of the screen are selected to obtain an average value, and this average value is set as the density value of the sixth pixel. Then, this is sequentially repeated to obtain the average density value for all the pixels in one line.

  By performing the arithmetic mean calculation in this way, the present apparatus 1 can eliminate small variations in density changes during image data shooting, and can capture global changes in density values.

  After the arithmetic mean calculation, the microcomputer 3 performs differential arithmetic on the arithmetic mean value in the vertical direction (ST43). Then, the microcomputer 3 performs point extraction based on the differential value (ST44). This point extraction is a process of determining one pixel for each local increase in the arithmetic average value of the pixel density along the vertical pixel row. For example, the differential value of the arithmetic average value is negative. This process determines a pixel that changes from positive to negative.

  After determining the pixel to be a point, the microcomputer 3 switches the line from which the current point has been extracted to the next line (ST45).

  Then, the microcomputer 3 determines whether or not point extraction has been completed for all the vertical lines (ST41). When it is determined that the point extraction has not been completed for all lines (ST41: NO), the process returns to step ST41 again through the processes of steps ST42 to ST45 described above.

  On the other hand, when it is determined that the point extraction has been completed for all lines (ST41: YES), the Y coordinate values of the extraction points of the adjacent lines are compared. When the Y coordinate value is within a predetermined value, (i) the group number of continuous data, (ii) the continuous start line number, and (iii) the number of continuous data are stored as continuous data. In addition, (iv) the average value of the vertical position of each extraction point constituting the continuous data (representative vertical position of the continuous data), (v) the average value of the horizontal position of the continuous start line and end line (the continuous data) Are stored (ST36).

  In the present embodiment, since the eye is a detection target, continuous data extends relatively long in the horizontal direction. For this reason, the microcomputer 3 can select the continuous data on the condition that after the continuous data is formed, it continues in the horizontal direction for a predetermined value or more.

  Thereafter, the microcomputer 3 determines the representative coordinate value C for each continuous data, and sets the existence area EA using this as a reference (ST47). The representative coordinate value C is determined by the average value of the stored X coordinate values and the average value of the Y coordinate values in the process of step ST46 (average values shown in iv and v above).

  After the representative coordinate value C is determined and the existence area EA is set, the process proceeds to step ST32 in FIG. The above is the eye candidate position specifying process (ST31). As described above, the obtained continuous data becomes the eye candidate, and the representative coordinate value C of the continuous data becomes the position of the eye candidate point.

  FIG. 11 is an explanatory diagram showing the continuous data formed by the process of step ST46 shown in FIG. 10, and the representative coordinate value C and the existence area EA determined by the process of step ST37. The eye candidate position specifying process (ST31) specifies one or a plurality of eye candidates, but FIG. 7 illustrates an example in which a plurality of eye candidates are specified.

  As shown in the figure, the microcomputer 3 forms a plurality of continuous data G. This is because an eye is a detection target, and a feature amount (mouth, nose, eyebrows, etc.) that is similar to the eye is detected.

  As described above, the continuous data G is formed when the extraction points determined for each pixel column in the vertical direction are adjacent in the horizontal direction of the image. Then, the representative coordinate value C is determined by the average value of the X coordinate values of the pixels in the lateral direction forming the continuous data and the average value of the Y coordinate values of the pixels forming the continuous data. Further, the existence area EA is set with the representative coordinate value C as a reference.

  Next, a method for setting the existence area EA will be described. FIG. 12 is an explanatory diagram showing the size of the existence area EA shown in FIG. 11, and FIGS. 13 and 14 show statistical data on the lengths of horizontal Xa and vertical Ya, which are obtained by examining the size of several eyes. FIG. 15 is an explanatory diagram showing a method of determining the position of the existence area EA on the image.

  The existence area EA is set by first determining the size of the existence area EA and then determining the position of the existence area EA on the image.

  The size of the existence area EA is preferably as small as possible in order to reduce noise (extract facial wrinkles, brightness and darkness) and not reduce the processing speed. In this embodiment, the size of the presence area EA is determined by examining the size of the eyes of several people and adding a margin (for example, x1.5). That is, as shown in FIG. 13 and FIG. 14, the data is determined by collecting data on the vertical and horizontal dimensions of the eye and adding a margin to a dimension that covers 95% of the distribution.

  As shown in FIG. 12, the size of the existence area EA is determined by adding a margin (× 1.5) to the dimension covering 95%, that is, the horizontal dimension xa and the vertical dimension ya. Yes. The size of the existence area EA may be a size that estimates the eye width and height by image processing and adds a margin to the vertical and horizontal sizes.

  After the size of the existence area EA is determined in this way, as shown in FIG. 15, for example, the reference point P is determined based on the coordinate values (x1, y1) of the eye. The reference point P is determined at a position separated from the eye coordinate values (x1, y1) by distances x2, y2.

  Then, the microcomputer 3 draws the dimensions x3 and y3 of the existence area EA with the point P as a reference. Thereby, the position of the existence area EA is determined. Thereafter, the existence area EA is set for all the continuous data G found in the entire image.

  Note that x2 and y2 are 1/2 of x3 and y3, and it is desirable to set the length so that the existence area EA is at the center of the eye in advance.

  The eye candidate position specifying process (ST31) in FIG. 9 is performed by the processes in FIGS. Next, the eye determination process (ST32) shown in FIG. 9 will be described. FIG. 16 is a flowchart for explaining the details of the eye determination process (ST32) shown in FIG.

  First, the microcomputer 3 stores the image data of the existence area EA obtained by the processing of FIG. 10 in the image memory as a minute image IG (ST50). FIG. 17 shows the state of the entire image and the minute image IG stored in the image memory. FIG. 17 is an explanatory diagram showing a minute image. As shown in FIG. 13, the microcomputer 3 extracts an image in the existence area EA from the entire image and forms a minute image IG.

  Again, a description will be given with reference to FIG. The microcomputer 3 sets the representative coordinate value C of the entire image as the representative coordinate value IC of the minute image IG. Then, the microcomputer 3 sets a range AR based on the representative coordinate value IC of the minute image IG, and sets a binarization threshold based on the density information of the range AR (ST51).

  An example of a binarization threshold calculation method in the range AR will be described with reference to FIG. FIG. 18 is an explanatory diagram of a method of calculating a binarization threshold in the range AR. First, the microcomputer 3 reads density values of several lines in the vertical direction in the range AR.

  The microcomputer 3 stores the highest (bright) density value and the lowest (dark) density value in each line. When the memory of all lines is completed, the microcomputer 3 determines that the lowest density value (skin part) among the highest (lightest) density values of each line and the lowest (darkest) density value of each line. Then, the lowest density value (eye part) is obtained. Then, the median is set as a binarization threshold.

  Note that the above-mentioned range AR is set so that the black part of the eye and the white part of the skin around the eye enter in order to determine the binarization threshold suitably. The range AR is set to a minimum size necessary for reducing the influence of variations in image brightness.

  Furthermore, the binarization threshold is set to the median value of the lowest (dark) density value of the eye within the range AR and the lowest (dark) density value of the skin portion, so that the skin portion to the eye portion are set. The value is suitable for cutting out.

  Here, the reason why the lowest (dark) density value in the skin portion is used to determine the binarization threshold is as follows. For example, when direct light hits a part of the range AR, the skin portion tends to reflect light more strongly than the black portion of the eyeball. For this reason, this apparatus 1 will input the light which can be said to be many noises.

  In this case, even if the range AR from which the density value is read is made as small as possible, the image is affected by noise light, and the apparatus 1 cannot determine an accurate binarization threshold. For this reason, in the present embodiment, by using the lowest (dark) density value of the density value of the skin portion without using the high density value portion that may be strongly reflected, more appropriate two The value threshold can be determined.

  Again, a description will be given with reference to FIG. After determining the binarization threshold, the microcomputer 3 binarizes the minute image IG using the determined binarization threshold, and stores it in the image memory as a binary image bG (ST52).

  Next, the microcomputer 3 sets the representative coordinate value C of the entire image as the position bC of the binary image bG, and sets this position bC as the initial position (ST53). Thereafter, the microcomputer 3 determines whether or not the set position is a black pixel (ST54). Here, first, it is determined whether or not the initial position set in step ST53 is a black pixel.

  If it is determined that the set position is not a black pixel (ST54: NO), the microcomputer 3 shifts the set position up, down, left, and right by one pixel (ST55). Thereafter, the microcomputer 3 determines whether or not the setting position shifted in step ST55 is a black pixel. The microcomputer 3 repeats this process until it is determined that the set position is a black pixel.

  On the other hand, when determining that the set position is a black pixel (ST54: YES), the microcomputer 3 sets the connected component of the black pixel as a candidate object (ST56). Then, the microcomputer 3 calculates the geometric shape of the candidate object (ST57).

  After the calculation, the microcomputer 3 compares the geometric shape of the eye template stored in advance with the geometric shape of the candidate object (ST58). An example of a method for comparing the geometric shapes of the candidate object and the eye template will be described with reference to FIG.

  FIG. 19 is an explanatory diagram of a method for comparing the geometric shapes of a candidate object and an eye template. FIG. 19A shows a case where the candidate object is imaged in an optimal state, and FIG. 19B shows that the right side of the eye is missing. (C) shows a state in which the left side of the eye is missing.

  The binarized shape of the eye image is as shown in FIG. 19A if the light environment is good and the image is stable. However, when the light environment deteriorates due to direct sunlight hitting the vehicle interior from one side, as shown in FIGS. 19B and 19C, a part of the shape may be lost.

  In order to accurately determine the candidate object as described above, the microcomputer 3 performs a comparison determination based on three conditions. First, the condition (i) is that the lateral width is 2/3 or more of the market value of the eye and has an upward convex curvature. Next, the condition (ii) is that there is a concave shape on the left side of the black eye. The condition (iii) is that there is a concave shape on the right side of the black eye.

  Again, a description will be given with reference to FIG. After comparing the geometric shapes, the microcomputer 3 makes a comparison determination based on the above three conditions, and determines whether the geometric shapes of the candidate object and the eye template match (ST59). Here, considering the case where a part of the eye shape is missing as shown in FIGS. 19B and 19C, the microcomputer 3 satisfies the conditions (i) and (ii) and the condition (ii). ) And (iii) are determined to match.

  If it is determined that they do not match (ST59: NO), the microcomputer 3 determines that the candidate object is not an eye (ST60), and then the process proceeds to step ST33 in FIG.

  On the other hand, if it is determined that they match (ST59: YES), the microcomputer 3 determines that the candidate object is an eye (ST61). Then, the determined coordinate value of the candidate object (corresponding to the representative coordinate value C in the entire image) is stored as the coordinate value of the eye on the image (ST62).

Thereafter, the microcomputer 3 stores the minute image IG including the candidate object determined to be coincident as the eye image MG _i in the image memory (ST63). And a process transfers to step ST33 of FIG.

  In the process of FIG. 16, a binarized candidate object is detected using a binarization threshold. For this reason, in this embodiment, an eye part and other parts (a background or a face part other than eyes) can be clearly distinguished, and an eye can be grasped correctly. Furthermore, the determination using the geometric shape of the candidate object can be performed more accurately, and the eye position detection accuracy can be further improved.

  As described above with reference to FIGS. 11 to 19, the microcomputer 3 (eye detection means CL <b> 12) acquires an eye image from the entire input image. As described above, if an eye image has been acquired in step ST33 in FIG. 9, the determination is “YES”, and the process proceeds to step ST15 in FIG.

  FIG. 8 will be referred to again. In step ST15, the microcomputer 3 determines whether an eye image has been acquired (ST15). If it is determined that an eye image could not be acquired (ST15: NO), the process proceeds to step ST27. On the other hand, if it is determined that an eye image has been acquired (ST15: YES), the microcomputer 3 executes an upper eyelid detection process (ST16). In this process, the microcomputer 3 executes a program corresponding to the upper eyelid detection means CL2 and detects the upper eyelid from the eye image.

  Here, there are various methods for detecting the upper eyelid. The microcomputer 3 detects the upper eyelid by any one or more of these methods. As a specific example, the microcomputer 3 detects an edge of the eye image in the vertical direction of the image, and detects an edge that is convex upward in the vertical direction of the image as an upper eyelid.

  Next, the microcomputer 3 determines whether or not the upper eyelid has been detected (ST17). If it is determined that the upper eyelid has not been detected (ST17: NO), the process proceeds to step ST27. On the other hand, when it is determined that the upper eyelid has been detected (ST17: YES), the microcomputer 3 extracts the coordinates of the upper eyelid (ST18). In this process, the microcomputer 3 executes a program corresponding to the upper eyelid coordinate extracting means CL3 to extract the upper eyelid coordinate group.

  FIG. 20 is an explanatory diagram showing detailed operations of the upper eyelid detection process (ST16) and the upper eyelid coordinate extraction process (ST18) shown in FIG. 8, (a) shows an example of the acquired eye image, (B) shows an image example obtained by subjecting an eye image to edge detection processing in the longitudinal direction of the image, (c) shows an example image obtained by performing image processing on the image shown in (b), and (d) shows ( An image example is shown when image processing is performed on the image shown in c).

  First, when an eye image as shown in FIG. 20A is acquired, the microcomputer 3 detects an edge in the eye image vertical direction. Thereby, as shown in FIG. 20B, the microcomputer 3 detects the edge line of the upper part of the upper eyelid and the edge line formed by combining the lower side of the upper eyelid and the black eye.

  Here, since the edge line of the upper part of the upper eyelid is convex upward, the upper eyelid detection means CL2 detects this edge line as the upper eyelid. Next, the microcomputer 3 extracts the coordinates of the upper eyelid from the edge line of the upper part of the upper eyelid. Thereby, the microcomputer 3 will extract the coordinate group of the upper eyelid.

  Further, instead of the method of extracting the coordinates of the upper eyelid from the edge line of the upper part of the upper eyelid, the following method may be adopted. That is, the microcomputer 3 finds a discontinuous point with respect to the edge line formed by the combination of the lower side of the upper eyelid and the black eye. A discontinuous point is a point at which the coordinate value (Y value) in the image vertical direction changes from increase to decrease or decreases to increase when the edge line is scanned in the horizontal direction (X-axis direction). Say point.

  Here, there are two discontinuous points for the edge line formed by the combination of the lower side of the upper eyelid and the black eye. For this reason, the microcomputer 3 complements by connecting two discontinuous points with a line, as shown in FIG. 20C (complement line 1 in FIG. 20C). Thereby, the microcomputer 3 obtains an edge line for the lower part of the upper eyelid. Then, the microcomputer 3 extracts the coordinate group of the upper eyelid from the edge line of the lower part of the upper eyelid.

  Furthermore, as shown in FIG. 20 (c), the microcomputer 3 complements the upper edge line and the lower edge line of the upper collar (complement line 2 in FIG. 20 (c)), and FIG. 20 (d). As shown in FIG. 5, a closed space may be formed by the upper edge line and the lower edge line. In this case, the microcomputer 3 uses the closed space as the upper eyelid region and extracts points in this space as the upper eyelid coordinate group.

  As described above, the microcomputer 3 detects the upper eyelid and extracts the coordinate group. Next, the microcomputer 3 executes an upper eye curve approximation process (FIG. 8: ST19). In this process, the microcomputer 3 executes a program corresponding to the curve approximation means CL4 and obtains the upper curve approximation line.

  Next, the curve approximation process (ST19) will be described in detail. The microcomputer 3 approximates the group of coordinates (x, y) extracted in step ST18 to a curve represented by a predetermined function f (x, y) = c. Here, (x, y) is represented by a coordinate system with the upper left corner of the eye image as the origin. Further, the function f (x, y) = c is expressed by a second-order or higher polynomial expression. This polynomial is an expression composed only of even-order terms in a coordinate system transformed so that the apex of the approximated curve is the origin. This is because the left-right symmetry and single-vertexity of the curved shape are impaired in an expression including an odd-order term in the coordinate system in which the vertex is the origin.

Here, the description will be given on the assumption that the parabola y = ax2 + bx + c that can be easily calculated among the quadratic curve is assumed, but the microcomputer 3 is a conic curve such as a circle, an ellipse, or a hyperbola, or ax ² + 2hxy + by ² + 2gx + 2fy + c = 0. It goes without saying that curve approximation can be performed by following the same procedure for a general quadratic curve or a curve of quadratic or higher.

Reference is now made to FIG. FIG. 21 is a flowchart showing details of the curve approximation process (ST19). As shown in FIG. 21, the microcomputer 3 approximates the coordinate group by the method of least squares. That is, first, the microcomputer 3 obtains the total value of the coordinate values (x, y) of each point constituting the coordinate group (ST70). At this time, the microcomputer 3 obtains the sum of the coordinate value x, the coordinate value y, and the value obtained by squaring the coordinate value x. Specifically, the microcomputer 3 obtains the total value of these by the following equation.

Thereafter, the microcomputer 3 obtains an average value of the total values (ST72). Specifically, the microcomputer 3 obtains an average value of these by the following formula.

And the microcomputer 3 calculates | requires a dispersion value and a covariance value from the calculated | required average value by the following formula | equation (ST73).

Thereafter, the microcomputer 3 obtains the regression coefficients a and b and the constant term c from the obtained variance value and covariance value by the following formula (ST74).

Thereby, the microcomputer 3 approximates the edge line to a parabola y = ax ² + bx + c. In the description of FIG. 21, the microcomputer 3 approximates the edge line to a parabola y = ax ² + bx + c, but a general quadratic curve represented by a conic curve such as a circle, an ellipse, or a hyperbola, or ax ² + 2hxy + by ² + 2Gx + 2fy + c = 0. You may make it approximate to a curve.

  FIG. 8 will be referred to again. After the curve approximation process (ST19), the microcomputer 3 executes the upper eye curvature calculation process (ST20). That is, the microcomputer 3 executes a program corresponding to the upper curvature detection means CL7 and detects the curvature.

  Here, the microcomputer 3 detects the curvature of the upper eyelid from the coordinate group extracted by the upper eyelid coordinate extracting means CL3. At this time, the microcomputer 3 obtains a curve approximation line in the same manner as described above, and detects the curvature R from the curve approximation line.

Here, details of a process in which the microcomputer 3 detects the curvature R from the approximate curve line (parabola) will be described. First, when the curve approximation line is expressed by the equation y = ax ² + bx + c, the curvature R is the regression coefficient a of the quadratic term. Further, the microcomputer 3 approximates the edge line to a circle (x−a) ² + (y−b) ² = r or an ellipse (x−a) ² / rx + (y−b) ² / ry = 1. To do. At this time, the curvature R is R = 1 / r in the case of a circle. Also, if it is an ellipse,

It becomes. In addition, description is abbreviate | omitted when the microcomputer 3 approximates an edge line other than a parabola, a circle | round | yen, and an ellipse.

  Thus, after calculating | requiring a curvature, the microcomputer 3 performs the process which calculates the correlation value of the coordinate group of an upper eyelid, and a curve approximation line (ST21). In this process, the microcomputer 3 executes a program corresponding to the correlation value detection means CL5 to detect the correlation value.

Specifically, the microcomputer 3 obtains the value of the sum of squares of the Y values of the curve approximation line with respect to the sum of squares of the Y values of the points constituting the upper eye coordinate group, and detects this value as a correlation value. That is,

  The correlation value is detected from the following formula.

  Thereafter, the microcomputer 3 executes an opening / closing eye determination threshold value setting process (ST22). FIG. 22 is a flowchart showing details of the open / closed eye determination threshold value setting process (ST22) shown in FIG. First, the microcomputer 3 determines whether or not the first and second determination threshold values are set (ST80). Here, the first determination threshold is a threshold set when the correlation value is higher than the reference value. That is, with regard to FIG. 5B, the threshold value is set to “170” as the curvature radius set when the reference value is higher than “0.85”. The second determination threshold is a threshold set when the correlation value is equal to or less than the reference value. That is, with regard to FIG. 5B, the threshold value is set to “350” as the curvature radius set when the reference value is “0.85” or less.

  If it is determined that the first and second determination threshold values are not set (ST80: NO), the microcomputer 3 executes first and second determination threshold value calculation processes as shown in steps ST81 to ST86. Accordingly, if “NO” is determined in step ST80, the microcomputer 3 executes programs corresponding to the first calculation means CL61 and the second calculation means CL62.

  Next, the microcomputer 3 determines whether or not the correlation value between the upper eyelid coordinate group obtained in step ST21 and the curve approximation line is high (ST81). That is, the microcomputer 3 determines whether or not the correlation value is higher than the reference value “0.85”.

  If the correlation value is higher than the reference value (ST81: YES), the microcomputer 3 stores the value of the curvature of the upper eyelid (ST82). That is, when the correlation value is higher than the reference value, it is considered that the curvature of the upper eyelid is obtained with high accuracy, and the microcomputer 3 stores the value of the curvature of the upper eyelid. At this time, the microcomputer 3 executes programs corresponding to the first sampling means CL611 and the second sampling means CL621.

  Next, the microcomputer 3 determines whether or not the curvature value of the upper eyelid has been stored for a predetermined number of frames (ST83). If the predetermined number of frames are not stored (ST83: NO), the process proceeds to step ST23 in FIG. On the other hand, when the predetermined number of frames are stored (ST83: YES), the microcomputer 3 calculates the first and second determination thresholds (ST84). At this time, the microcomputer 3 executes programs corresponding to the first representative value calculating unit CL612 and the second representative value calculating unit CL622. Then, the microcomputer 3 performs the following processing.

  First, the microcomputer 3 obtains a representative value of the curvature of the upper eyelid when the eye is opened from the curvature of the upper eyelid stored for a predetermined number of frames in order to obtain the first determination threshold value. That is, the microcomputer 3 obtains one of the average value, the median value, and the mode value of the curvature of the curvature of the upper eyelid sampled by a predetermined number. Then, the microcomputer 3 sets the representative value of the curvature as the first statistical representative value, and recognizes the first statistical representative value as the curvature at the time of eye opening.

  Here, in general, when the eyes of the person to be detected are imaged, the person to be detected is in an open state of about 90% or more. For this reason, even if it is any of the average value, median value, or mode value of the sampled curvature, it becomes a value close to the curvature at the time of eye opening. Therefore, the microcomputer 3 can recognize the first statistical representative value as the curvature at the time of eye opening.

  Next, the microcomputer 3 determines the curvature of the upper eyelid when the eye is closed from the correlation between the curvature of the upper eyelid when the eye is opened and the curvature of the upper eyelid when the eye is opened and the maximum curvature of the upper eyelid when the eye is closed. Find the maximum curvature of the upper arm of the individual. FIG. 23 is an explanatory diagram showing the curvature of the upper eyelid when the eye is opened for each detected person, and FIG. 24 is an explanation showing the correlation between the curvature of the upper eyelid when the eye is opened and the maximum curvature of the upper eyelid when the eye is closed. FIG.

  First, as shown in FIG. 23, the curvature of the upper eyelid at the time of eye opening was detected for 25 persons to be detected A to Y. At this time, the curvature of the upper eyelid at the time of eye opening differs for each detected person, but the value of the radius of curvature is approximately in the range of “50” to “120”. Moreover, the maximum curvature (minimum curvature radius) of the upper eyelid when these eyes were closed was detected for these subjects. Then, it became clear that it has a relationship as shown in FIG. That is, as shown in FIG. 24, it has been clarified that the curvature radius of the upper eyelid when the eye is open and the minimum curvature radius of the upper eyelid when the eye is closed are in a proportional relationship.

  For this reason, the microcomputer 3 obtains the maximum curvature of the upper eyelid of the individual to be detected when the eyes are closed by applying the first statistical representative value recognized as the curvature of the upper eyelid at the time of eye opening to the correlation. Can do. The microcomputer 3 calculates this maximum curvature as the first determination threshold value.

  As described above, the microcomputer 3 stores (samples) the curvature of the upper eyelid for a predetermined number of frames (predetermined number) and obtains the maximum curvature of the upper eyelid when the individual to be detected is closed. Here, the maximum curvature of the upper eyelid when the eye is closed can be said to be the minimum curvature of the upper eyelid when the eye is opened. For this reason, it can be said that the maximum curvature of the upper eyelid when the eye is closed is a curvature in which the determination of the open eye and the closed eye is divided. Therefore, the first determination threshold value is a value indicating the boundary between open eyes and closed eyes, and is a value suitable for performing open / closed eye determination.

  In FIG. 24, the radius of curvature is illustrated for some detected persons, and the radius of curvature is omitted for other detected persons.

  Next, the microcomputer 3 calculates a second determination threshold value. In calculating the second determination threshold, first, the microcomputer 3 obtains one of an average value, a median value, and a mode value as the second statistical representative value from a predetermined number of sampled curvatures. Thereafter, the microcomputer 3 samples again the curvature of the upper eyelid.

  The microcomputer 3 then selects one of the average value, the median value, and the mode value of the curvature for the curvature smaller than the calculated second statistical representative value among the predetermined number of curvatures sampled again. Is determined as a third statistical representative value. That is, the microcomputer 3 obtains a second statistical representative value when a predetermined number of curvatures of the upper eyelid are stored, and further obtains a third statistical representative value when a predetermined number of curvatures of the upper eyelid are stored. Become. Then, the microcomputer 3 repeats the sampling and the calculation of the statistical representative value N times (N is an integer equal to or greater than 2) (that is, in order to calculate the second determination threshold value, a plurality of processes of steps ST81 to ST86 are performed. Must be repeated).

  Summarizing the above operation, the microcomputer 3 obtains one of the average value, median value, and mode value of the curvature as the second statistical representative value from the predetermined number of curvatures sampled at the first time. Then, the microcomputer 3 calculates the curvature for a curvature smaller than the nth statistical representative value obtained for the (n-1) th time out of a predetermined number of curvatures sampled at the nth time (n is an integer of 2 or more and N or less). Any one of the average value, the median value, and the mode value is obtained as the (n + 1) th statistical representative value. Thereafter, the microcomputer 3 sets the obtained N + 1-th statistical representative value as the second determination threshold value.

  Here, in general, when the eye of the person to be detected is imaged, since the person to be detected is in an open state of about 90% or more, any of the average value, median value, or mode value of the sampled curvature may be used. It becomes a value close to the curvature at the time of eye opening. Therefore, among the curvatures sampled again, for any curvature smaller than the second statistical representative value, any one of the average value, median value, or mode value of the curvature is obtained as the third statistical representative value. The average value, median value, or mode value is obtained excluding most of the curvature at the time. Then, by repeating these steps, the average value, median value, or mode value is obtained by removing much of the curvature when the eye is further opened.

  Therefore, by obtaining an average value, a median value, or a mode value from the curvature obtained by removing a large amount of curvature when the eyes are opened, these values can be made close to the curvature when the eyes are closed. That is, the second determination threshold value can be set to a small value (a high value based on the radius of curvature) so that the eyes are not erroneously determined to be closed. Moreover, since the curvature of the current driver's eye is sampled, the second determination threshold can be reduced within a range appropriate for the current driver.

  As shown in FIG. 7, in order to ensure that the second determination threshold value is a value such as a curvature radius “350” so that the eyes are not erroneously determined to be closed when the eyes are opened, The number of repetitions N may be set appropriately in accordance with the shooting angle, the system used, and the like. Further, after obtaining the second statistical representative value, the microcomputer 3 does not sample again (that is, does not repeat the processing of ST81 to ST86 N times), and has a predetermined number of upper curvatures that are already stored. For example, the calculation of the statistical representative value may be repeated to obtain the (N + 1) th statistical representative value.

  In addition, the second determination threshold value may be calculated as follows. That is, the microcomputer 3 samples a predetermined number of curvatures of the upper eyelid, and among the sampled number of curvatures sampled, for the curvature smaller than the first determination threshold value calculated earlier, the average value, median value, or mode of curvature One of the values is determined as the second statistical representative value. Then, the microcomputer 3 uses the obtained second statistical representative value as the second determination threshold value.

  Here, in order to reduce the possibility of erroneously determining that the eyes are closed when the eyes are opened, the second determination threshold must be at least smaller than the first determination threshold. That is, as shown in FIG. 7, when the curvature radius is considered as a reference, if the second determination threshold value is not at least larger than the first determination threshold value, the possibility of erroneously determining that the eye is closed at the time of opening the eye is reduced. Don't be. However, according to the above method, the second determination threshold value is always smaller than the first determination threshold value, and the possibility of erroneously determining that the eyes are closed when the eyes are opened can be reduced. Furthermore, it is not necessary to set the number of repetitions (the number of N) appropriately without sampling multiple times.

  Reference is again made to FIG. As described above, after calculating the first and second determination thresholds, the microcomputer 3 sets the first determination threshold in a memory or the like (ST85). Next, the microcomputer 3 sets the second determination threshold value in a memory or the like (ST86). And a process will transfer to step ST23 of FIG.

  By the way, when it is determined in step ST80 that the first and second determination threshold values are set (ST80: YES), the microcomputer 3 determines whether or not the correlation value between the upper eyelid coordinate group and the curve approximation line is high. Is determined (ST87). That is, the microcomputer 3 determines whether or not the correlation value is higher than the reference value.

  If the correlation value is higher than the reference value (ST81: YES), the microcomputer 3 sets the first determination threshold as the opening / closing eye determination threshold (ST88). And a process transfers to step ST23 of FIG. If the correlation value is not higher than the reference value (ST81: NO), the microcomputer 3 sets the second determination threshold as the open / close eye determination threshold (ST89), and the process proceeds to step ST23 in FIG.

  FIG. 8 will be referred to again. As described above, after the opening / closing eye determination threshold value setting process (ST22) ends, the microcomputer 3 determines whether or not the opening / closing eye determination threshold value has been determined (ST23). Here, if only the processing of steps ST81 to ST86 shown in FIG. 22 is executed and the processing of step ST88 or ST89 shown in FIG. 22 is not executed, the microcomputer 3 has not determined the opening / closing eye determination threshold value. (ST23: NO), the process proceeds to step ST27.

  On the other hand, if the process of step ST88 or ST89 shown in FIG. 22 is executed, the microcomputer 3 determines that the opening / closing eye determination threshold has been determined (ST23: YES), and the curvature of the upper eyelid determines the opening / closing eye determination. It is determined whether or not the threshold value is exceeded (ST24). At this time, the microcomputer 3 executes a program corresponding to the open / close eye determination means CL8.

  If it is determined that the curvature of the upper eyelid is greater than or equal to the opening / closing eye determination threshold (ST24: YES), the microcomputer 3 determines that the driver is in the open state (ST25). Thereafter, the process proceeds to step ST27. On the other hand, when it is determined that the curvature of the upper eyelid is not equal to or greater than the opening / closing eye determination threshold value (ST24: NO), the microcomputer 3 determines that the driver is in the closed eye state (ST26). Thereafter, the process proceeds to step ST27.

  In step ST27, the microcomputer 3 increments the processing frame counter “i” (ST27). Thereafter, the process returns to step ST12, and for example, it is determined that the engine is not started, and the above-described process is repeated until it is determined that it is “STOP”.

  In this way, the open / closed eye determination device 1 obtains a correlation value between the upper eyelid coordinate group and a quadratic or higher curve approximated to the coordinate group. Here, the case where the correlation value is high indicates that it is easy to accurately determine the open / closed eye due to circumstances such as little noise around the eye. On the other hand, when the correlation value is low, it indicates that it is easy to make an erroneous determination with respect to the open / closed eye due to a lot of noise around the eye.

  In addition, when the correlation value is higher than the reference value, the open / close eye determination threshold is set larger than when the correlation value is equal to or lower than the reference value. For this reason, when the correlation value is higher than the reference value, that is, when it is easy to accurately determine the open / closed eye, the open / closed eye determination threshold value is increased. On the other hand, when the correlation value is less than or equal to the reference value, that is, when it is easy to make an erroneous determination for the open / closed eye, the open / closed eye determination threshold value is decreased.

  Further, the curvature of the upper eyelid is obtained, and it is determined that the eye is open when the curvature is larger than the open / closed eye determination threshold, and the closed eye is determined when the curvature is equal to or less than the open / closed eye determination threshold. Here, when there is a lot of noise around the eyes, the accuracy of detection of the curvature of the upper eyelid also decreases. That is, when the correlation value is small, the accuracy of detection also decreases for the curvature of the upper eyelid. For this reason, when the eye is opened, the curvature of the upper eyelid may be detected small. However, in this embodiment, when the correlation value is less than or equal to the reference value, the open / close eye determination threshold value is reduced. Therefore, even if the curvature of the upper eyelid is detected to be small when the eye is opened, the open / close eye determination threshold value is made small, so that the possibility of erroneously determining that the eye is closed when the eye is opened can be reduced.

  Therefore, it is possible to prevent erroneous determination that the eyes are closed when the eyes are opened.

  Also, any one of the average value, median value, and mode value of the sampled curvature is obtained as the first statistical representative value, and the first statistical representative value is recognized as the curvature at the time of eye opening. Here, in general, when the eyes of the person to be detected are imaged, the person to be detected is in an open state of about 90% or more. For this reason, even if it is any of the average value, median value, or mode value of the sampled curvature, it becomes a value close to the curvature at the time of eye opening. Therefore, the curvature at the time of eye opening can be recognized.

  In addition, the upper eyelid curvature when the eyes are opened and the correlation between the curvature of the upper eyelids when the eyes are opened and the maximum curvature of the upper eyelids when the eyes are closed are determined in advance. The maximum curvature of the kite is being sought. The maximum curvature is calculated as the first determination threshold value. That is, from the curvature of the upper eyelid when the eye is open, the maximum curvature of the upper eyelid when the individual to be detected is closed is obtained.

  Thus, in this embodiment, the curvature of the upper eyelid can be sampled to obtain the maximum curvature of the upper eyelid when the individual to be detected is closed. Here, the maximum curvature of the upper eyelid when the individual to be detected is closed is, in other words, the minimum curvature of the upper eyelid when the individual to be detected is opened. That is, it can be said that the maximum curvature of the upper eyelid when the individual to be detected is closed is a curvature at which the determination of opening and closing of eyes is divided for the current detected person. Therefore, the first determination threshold value is a value suitable for the open / close eye determination.

  Therefore, an appropriate threshold value can be set for each person to be detected when determining the open / closed eye.

  Also, sampling is repeated N times, and for the curvature sampled at the first time, one of the average value, median value, or mode value of the curvature is obtained as the second statistical representative value, and the curvature sampled at the Nth time. For the curvature smaller than the Nth statistical representative value, any one of the average value, median value, or mode value of the curvature is obtained as the N + 1 statistical representative value, and this N + 1th statistical representative value is obtained. Is the second determination threshold.

  Here, in general, when the eyes of the person to be detected are imaged, the person to be detected is in an open state of about 90% or more. For this reason, even if it is any of the average value, median value, or mode value of the sampled curvature, it becomes a value close to the curvature at the time of eye opening. Therefore, among the curvatures sampled again, for any curvature smaller than the second statistical representative value, any one of the average value, median value, or mode value of the curvature is obtained as the third statistical representative value. The average value, median value, or mode value is obtained excluding most of the curvature at the time. Then, by repeating these steps, an average value, median value, or mode value is obtained by removing much of the curvature when the eye is further opened.

  Therefore, these values can be reduced by obtaining an average value, median value, or mode value from the curvature obtained by removing a large amount of curvature at the time of eye opening. That is, the second determination threshold value can be set to a small value (a high value based on the radius of curvature) so that the eyes are not erroneously determined to be closed. Moreover, since the curvature of the current driver's eye is sampled, the second determination threshold can be reduced within a range appropriate for the current driver.

  Therefore, it is possible to set an appropriate threshold value for each person to be detected when determining the open / closed eye.

  Here, in order to reduce the possibility of erroneously determining that the eyes are closed when the eyes are opened, the second determination threshold must be at least smaller than the first determination threshold. That is, when considering the radius of curvature as a reference, if the second determination threshold value is not at least larger than the first determination threshold value, the possibility that the eye is erroneously determined to be closed at the time of eye opening will not be reduced. For this reason, a predetermined number of curvatures of the upper eyelid are sampled, and among the predetermined number of curvatures sampled, any one of the average value, median value, or mode value of the curvature is smaller than the previously calculated first determination threshold value. One of them is obtained as the second statistical representative value. Then, the obtained second statistical representative value is set as a second determination threshold value.

  In this way, the second determination threshold value is always smaller than the first determination threshold value, and it is possible to reliably reduce the possibility of erroneously determining that the eyes are closed when the eyes are opened. Furthermore, it is not necessary to sample a plurality of times, and it is not necessary to appropriately set the number of repetitions (the number N).

  Therefore, it can be easily prevented from erroneously determining that the eyes are closed when the eyes are opened.

  Further, the eye of the detected person is detected from the face image obtained by capturing the entire face of the detected person, and a minute image smaller than the face image is acquired as an eye image. For this reason, even in the detection of the curvature of the upper eyelid later, a minute eye image is processed, and the opening / closing eye is determined more accurately than when detecting the upper eyelid from the entire face image. be able to. Therefore, the open / closed eye determination accuracy can be improved.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not restricted to the said embodiment, You may combine each embodiment. Moreover, you may add a change in the range which does not deviate from the meaning of this invention. Changes may be made without departing from the spirit of the present invention.

  For example, in the present embodiment, the first and second determination threshold values are obtained. However, the threshold values are not limited to two and may be set in three or more stages. Further, it may be as shown in FIG. 25 and FIG.

  FIG. 25 is an explanatory diagram showing the threshold and the distribution status of the open / closed eyes, and FIG. 26 is an explanatory diagram of the threshold. As shown in the figure, the threshold value may be expressed by a linear expression instead of a constant value in a section where the correlation value is “0.6” to “0.85”, for example. When obtaining such a linear expression, the microcomputer 3 calculates, for example, an average value, a median value, or a mode value for the curvature of the upper eyelid obtained in the section of the correlation values “0.60” to “0.61”. Seek and remember. Further, the microcomputer 3 similarly obtains and stores the average value, the median value, or the mode value at intervals of about “0.01” in the interval of the correlation value “0.61” or more. Then, a linear expression is obtained by obtaining an approximate line from the obtained 250 values.

It is a functional block diagram which shows the structure of the opening-and-closing eye determination apparatus which concerns on embodiment of this invention. It is a hardware block diagram of the opening-and-closing eye determination apparatus 1 which concerns on embodiment of this invention. It is a hardware block diagram which shows the other example of the face image acquisition means CL11. It is explanatory drawing which shows the mode of an upper eyelid, (a) shows the example when there is little noise around eyes, (b) shows the example when there is much noise around eyes. It is explanatory drawing which shows the relationship between the curvature radius of an upper eyelid, and a correlation value, (a) shows the example when there is little noise around an eye, (b) shows the example when there is much noise around an eye. Show. It is explanatory drawing of a reference value. It is explanatory drawing of an opening-and-closing eye determination threshold value. It is a main flowchart which shows the outline of operation | movement of the opening-and-closing eye determination apparatus 1 which concerns on 1st Embodiment. It is a flowchart which shows the detailed operation | movement of the eye image acquisition process (ST14) shown in FIG. 10 is a flowchart showing details of eye candidate position specifying processing (ST31) shown in FIG. It is explanatory drawing which shows the continuous data formed by the process of step ST46 shown in FIG. 10, and the representative coordinate value C and presence area EA which are defined by the process of step ST37. It is explanatory drawing which shows the magnitude | size of the presence area EA shown in FIG. It is explanatory drawing which shows the statistical data of the length of the horizontal Xa which investigated the magnitude | size of several people's eyes. It is explanatory drawing which shows the statistical data of the length of length Ya which investigated the magnitude | size of several people's eyes. It is explanatory drawing which shows the method of determining the position on the image of presence area EA. 10 is a flowchart for explaining details of an eye determination process (ST32) shown in FIG. It is explanatory drawing which shows a micro image. It is explanatory drawing of the calculation method of the binarization threshold value in the range AR. It is explanatory drawing of the comparison method of the geometric shape of a candidate object and an eye template, (a) shows the case where a candidate object was imaged in the optimal state, (b) shows the state where the right side of the eye was missing. , (C) shows a state where the left side of the eye is missing. It is explanatory drawing which shows the detailed operation | movement of the upper eyelid coordinate extraction process (ST18) shown in FIG. 8, (a) shows an example of the acquired eye image, (b) is an edge in an image vertical direction about an eye image (C) shows an image example when the image processing is performed on the image shown in (b), and (d) shows a case where the image processing is performed on the image shown in (c). An example of the image is shown. It is a flowchart which shows the detail of a curve approximation process (ST19). It is a flowchart which shows the detail of the opening-and-closing eye determination threshold value setting process (ST22) shown in FIG. It is explanatory drawing which shows the curvature of the upper eyelid at the time of eye opening for every to-be-detected person. It is explanatory drawing which shows correlation with the curvature of the upper eyelid at the time of eye opening, and the maximum curvature of the upper eyelid at the time of eyes closed. It is explanatory drawing which shows the distribution condition of a threshold value and an opening-and-closing eye. It is explanatory drawing of a threshold value.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Opening / closing eye determination apparatus CL1 ... Eye image acquisition means CL11 ... Face image acquisition means CL12 ... Eye detection means CL2 ... Upper eyelid detection means CL3 ... Upper eyelid coordinate extraction means CL4 ... Curve approximation means CL5 ... Correlation value detection means CL6 ... Opening / closing Eye determination threshold value setting means CL61 ... first calculation means CL611 ... first sampling means CL612 ... first representative value calculation means CL62 ... second calculation means CL621 ... second sampling means CL622 ... second representative value calculation means CL7 ... upper curvature Detection means CL8 ... Opening / closing eye determination means

Claims (7)

Eye image acquisition means for acquiring an eye image including the eyes of the detected person;
Upper eyelid detection means for detecting upper eyelid from the eye image acquired by the eye image acquisition means;
Upper eyelid coordinate extracting means for extracting a coordinate group constituting the upper eyelid from the upper eyelid detected by the upper eyelid detecting means;
A curve approximation means for approximating the upper eyelid to a quadratic or higher curve from the upper eyelid coordinate group extracted by the upper eyelid coordinate extracting means;
Correlation value detection means for detecting a correlation value between the upper eyelid coordinate group extracted by the upper eyelid coordinate extraction means and a quadratic or higher curve approximated by the curve approximation means;
When the correlation value detected by the correlation value detection means is higher than a reference value, the opening / closing eye determination threshold value that serves as a reference for opening / closing eye determination than when the correlation value detected by the correlation value detection means is less than or equal to the reference value Open / close eye determination threshold setting means for setting a large value;
An upper curvature detection means for detecting the curvature of the upper eyelid from the coordinate group extracted by the upper eyelid coordinate extraction means;
When the curvature of the upper eyelid detected by the upper curvature detection means is equal to or greater than the opening / closing eye determination threshold set by the opening / closing eye determination threshold setting means, the eye is determined to be open, and when the opening / closing eye determination threshold is not reached, the eye is closed Opening / closing eye determination means for determining
An open / closed eye determination device comprising: The opening / closing eye determination threshold value setting means includes:
First calculation means for calculating a first determination threshold value set from a representative value of curvature of the upper eyelid at the time of eye opening;
Second calculating means for calculating a second determination threshold value set from a representative value of the curvature of the upper eyelid when the eye is closed,
When the correlation value obtained from the correlation value detection means is higher than a predetermined reference value, the first determination threshold value calculated by the first calculation means is set as the open / close eye determination threshold value,
The second determination threshold value calculated by the second calculation unit is set as an open / closed eye determination threshold value when the correlation value obtained from the correlation value detection unit is a predetermined reference value or less. The open / close eye determination device according to claim 1. The first calculation means includes
First sampling means for sampling a predetermined number of curvatures of the upper eyelid,
From the predetermined number of curvatures sampled by the first sampling means, one of an average value, a median value, and a mode value is calculated, and the calculated value is a first representative representative value. Calculating means,
The first statistical representative value obtained by the first representative value calculating means is recognized as the curvature at the time of eye opening,
The upper eyelid curvature when the eye is closed is determined based on the correlation between the curvature of the upper eyelid when the eye is opened and the maximum curvature of the upper eyelid when the eye is closed and the curvature of the upper eyelid when the eye is closed. The open / closed eye determination device according to claim 2, wherein the maximum curvature is calculated, and the maximum curvature is calculated as the first determination threshold value. The second calculation means includes
A second sampling means for sampling a predetermined number of curvatures of the upper eyelid;
Second representative value calculating means for obtaining one of an average value, a median value, and a mode value from a predetermined number of curvatures sampled by the second sampling means;
The second sampling means repeats a predetermined number of samplings for the curvature of the upper eyelid N times (N is an integer of 2 or more),
The second representative value calculating means includes:
From the predetermined number of curvatures sampled for the first time by the second sampling means, one of the average value, median value or mode value of the curvature is obtained as the second statistical representative value,
Of the predetermined number of curvatures sampled n times (n is an integer not less than 2 and not more than N) by the second sampling means, for the curvature smaller than the nth statistical representative value, the average value of the curvature, the median value, or Find one of the mode values as the (n + 1) th statistical representative value,
The open / closed eye determination device according to claim 2, wherein the (N + 1) th statistical representative value obtained by the second representative value calculation unit is set as the second determination threshold value. The second calculation means includes
A second sampling means for sampling a predetermined number of curvatures of the upper eyelid;
Of the predetermined number of curvatures sampled by the second sampling means, any one of an average value, median value, or mode value of the curvature is smaller than the first determination threshold value calculated by the first calculation means. Second representative value calculating means for obtaining one as a second statistical representative value,
The open / closed eye determination device according to claim 3, wherein a second statistical representative value obtained by the second representative value calculation unit is used as the second determination threshold value. The eye image acquisition means includes
A face image capturing means that is installed substantially in front of the detected person and images the entire face of the detected person;
Eye detection means for detecting the eyes of the person to be detected from a face image including the entire face imaged by the face image imaging means,
6. The minute image that is smaller than the face image including the eyes detected by the eye detection means is extracted from the face image and used as an eye image. The open / close eye determination device described. Acquire eye images including the eyes of the person to be detected, detect upper eyelids, extract the coordinate group that constitutes upper eyelid from the detected upper eyelid, and extract the upper eyelid from the extracted upper eyelid coordinate group to secondary or higher To the curve
When the correlation value between the extracted upper eyelid coordinate group and the approximated quadratic or higher curve is detected, and the detected correlation value is higher than the reference value, the opening / closing eye is lower than when the correlation value is lower than the reference value. Set a large judgment threshold,
When the curvature of the upper eyelid is detected from the extracted upper eyelid coordinate group and the detected curvature of the upper eyelid is equal to or greater than the opening / closing eye determination threshold, the eye opening is determined, and when the opening / closing eye determination threshold is not reached An open / closed eye determination device characterized by determining that the eye is closed.