JP2005327072A - Eye tracking apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an eye tracking apparatus capable of satisfactorily returning to an eye tracking process when eye tracking fails. <P>SOLUTION: The eye tracking apparatus includes a position relation decision section 43d for comparing an eye-nose distance obtained when tracking by mistakenly regarding a face portion other than eyes of an object person as eye with an eye-nose distance detected from an overall image in a succeeding process. At the time of the above comparison, the position relation decision section 43d uses a frequency distribution of the eye-nose distance generated by a frequency distribution generation section 43b. The position relation decision section 43d decides that the eye detected from the overall image is incorrect when the above two eye-nose distances are at similar places in the distribution, and that the eye detected from the overall image is correctly detected when the two distances are at different places in the distribution. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an eye tracking device.

  Conventionally, there is known a device that detects an eye from a whole face image obtained by imaging the entire face of the driver, and detects and tracks the eye from a local region where the eye is likely to exist after the detection. ing. This apparatus has an advantage that the processing speed can be improved because the eye is detected and tracked from the local area after the eye is detected once. In addition, even if the driver is wearing spectacles, the eye is detected and tracked from the local area around the eye, making it difficult for the frame portion of the spectacle to enter the local area, reducing the effects of mistracking by spectacles. There exists an advantage that it can do (refer patent document 1).

Further, a face image processing device that obtains a face inclination angle by inputting a driver's face image by a face image input means and extracting facial feature points such as eyebrows, eyes, nose, and mouth from the face image. It has been known. In this apparatus, the eyeball presence region can be set with high accuracy by obtaining the tilt angle of the face. (See Patent Document 2)
Also, an eye image tracking device that accurately tracks an eye image by recognizing not only an eye image but also an eye periphery image to recognize the eye from the eye periphery image when the eye is lost is known. ing. Further, in this apparatus, the tracking accuracy of the eye image is improved by using the relative position with respect to the nose as well as the eye (see Patent Document 3).
Japanese Patent Laid-Open No. 7-181012 JP-A-9-35070 Japanese Patent Laid-Open No. 11-66320

  However, in the apparatus described in Patent Document 1, there is a possibility that the eyebrows are mistakenly tracked as being eyes. In other words, in the image of the local region, the eyes and the eyebrows have very similar characteristics and are difficult to distinguish, leading to erroneous tracking.

  Further, in the apparatus described in Patent Document 2, since it is necessary to simultaneously detect a plurality of face parts such as eyebrows, eyes, nose, and mouth, the current computer processing capability tracks the eyes in real time. Is difficult. For this reason, in this apparatus, when the eye is to be tracked in real time, the processing must be performed roughly, and this may cause erroneous eye tracking.

  Furthermore, in the apparatus described in Patent Document 3, the relative position between the eyes and the nose is used. However, the relative position between the eyes and the nose has a large individual difference, and the influence of the individual difference is reduced by recording personal information. Failure to do so will lead to eye mistracking.

  As described above, the conventional apparatus does not prevent mistracking. For this reason, in the conventional device, when an erroneous tracking occurs, the eye is detected again in order to recover the eye tracking. However, this detection is not always possible and the eye cannot be accurately detected. There is a possibility of erroneous tracking, and it cannot be said that the return to the eye tracking process is smoothly performed.

  The present invention has been made in order to solve such a conventional problem, and the object of the present invention is to satisfactorily return to eye tracking processing when eye tracking fails. Is to provide a simple eye tracking device.

  The eye tracking device according to the present invention tracks an eye from a face image of tracking an eye from a face image of a person to be imaged continuously captured by an imaging device. The eye tracking device includes a face image acquisition unit that obtains a signal obtained by capturing a face image of the person to be imaged as face image data, and a second face part other than the eyes based on the face image data from the face image acquisition unit. And coordinate detecting means for detecting coordinates on the face image. In addition, the eye tracking device detects in time series the positional relationship calculating means for calculating the feature amount of the coordinates of the eye and the second face part from both coordinates detected by the coordinate detecting means, and the positional relationship calculating means. And classifying means for classifying the plurality of feature quantities according to a predetermined condition. Further, the eye tracking device includes a classification determination unit and a positional relationship determination unit. Then, the classification determination means, when the face portion other than the eye of the person being imaged is tracked as being an eye by mistake, the feature amount calculated by the positional relationship calculation means when the tracking is erroneously tracked, and The configuration is such that when the eye is detected after being tracked by mistake, both of the feature amounts calculated by the positional relationship calculation means belong to which of the feature amount categories classified by the classification means. Further, the positional relationship determining means is configured to determine that the detected tracking target is an eye when both the classifications determined by the classification determining means are different.

  According to the present invention, a feature amount of coordinates between the eyes and the second face part other than the eyes is calculated from the face image data, and a plurality of feature amounts obtained by detecting the feature amounts in time series are determined as predetermined conditions (for example, features) It is classified according to the size of the quantity. For this reason, similar feature amounts are classified into the same category.

  Then, when the eye of the person to be imaged is mistakenly tracked, that is, when the eyebrow is mistakenly tracked as being an eye, it is determined to which classification the feature amount at that time belongs. In addition, it is determined to which category the feature amount when the eye is detected after being tracked by mistake belongs. For this reason, it is possible to determine the classification to which the wrong tracking is performed and the subsequent detection.

  Next, when the determined classifications are different, the detected tracking target is determined to be an eye. Here, when the eyebrow is mistakenly tracked as being an eye, and when the eye is tracked accurately, the feature amount is different. Therefore, when the category when a new eye is detected is the same as the category when the eyebrow is tracked incorrectly, it can be determined that the eyebrow is detected at the time of detection, and the eyebrow is tracked incorrectly. If it is different from the category, it can be determined that the eye has been accurately detected even at the time of detection.

  Therefore, when eye tracking fails, it is possible to satisfactorily return to eye tracking processing.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described with reference to the drawings. In the following embodiments, a case where the eye tracking device is mounted on a vehicle will be described as an example.

  FIG. 1 is a hardware configuration diagram of an eye tracking device 1 according to an embodiment of the present invention. As shown in the figure, the eye tracking device 1 of this embodiment tracks an eye from a face image of tracking an eye from a face image of a person to be imaged continuously captured by an imaging device. When the eye tracking device 1 fails to track the eye, when the eye is detected again and the tracking process is restored, the eye tracking device 1 uses the data at the time of the previous tracking to improve the return to the tracking process. Is what you do. In the following, a case where a vehicle driver is imaged as an imaged person will be described as an example.

  The eye tracking device 1 includes a camera 10 and an eye detection device 20. The camera 10 is installed slightly in front of the driver, captures the driver's face, and outputs the obtained video signal. The eye detection device 20 receives the video signal from the camera 10 and obtains the coordinates of the driver's eyes and the second face part. Here, the second facial part refers to a facial part other than the eyes, such as a nose or a mouth. In the present embodiment, the nose (particularly the nostril) will be described below as an example of the second facial part. The eye detection device 20 outputs an eye-nose coordinate signal when the coordinates of the driver's eyes and nose are obtained.

  FIG. 2 is a data flow diagram showing details of the eye detection apparatus 20 shown in FIG. As shown in the figure, the eye detection device 20 includes a face image acquisition unit (face image acquisition means) 30, a face part detection unit 40, a face part tracking unit 50, and an eye-nose coordinate signal output unit 60.

  The face image acquisition unit 30 acquires a video signal from the camera 10 that has captured the driver's face image as face image data. Specifically, the face image acquisition unit 30 captures data input by a video signal and stores it as a digital image in a storage area in the apparatus 20. This stored digital image is a face image.

  The face part detection unit 40 detects the coordinates of the eyes and the nose from the entire face image when the apparatus 1 is started up or after erroneous eye tracking. Here, the face part detection unit 40 of the present embodiment detects the coordinates of both eyes as the coordinates of the driver's eyes.

  The face part tracking unit 50 extracts a minimal region smaller than the entire image from the face image based on the eye and nose coordinates detected by the face part detection unit 40. Further, the face part tracking unit 50 identifies and tracks the coordinates of the eyes and nose from the extracted minimal region by a pattern matching method or a shape recognition method. As described above, the face part tracking unit 50 identifies the eye and nose from the minimal region, and therefore can accurately track the eye and nose in real time.

  The eye-nose coordinate signal output unit 60 outputs the eye and nose coordinates detected and tracked by the face part detection unit 40 and the face part tracking unit 50 as an eye-nose coordinate signal. Specifically, the eye-nose coordinate signal output unit 60 is connected to, for example, an eye opening / closing eye detection device or a side-view detection device, and is configured to output an eye-nose coordinate signal to these devices. For this reason, the eye-nose coordinate signal is used to detect a driver's drowsiness or looking aside by using an open / closed eye detection device, a look-ahead detection device, or the like.

  The face part detection unit 40 and the face part tracking unit 50 are configured to read processing state information stored in a storage area in the apparatus 20 and execute processing. Here, the processing state is as shown in FIG. FIG. 3 is an explanatory diagram of the processing state.

  The processing state represents the current state of the eye tracking device 1. Specifically, the processing state is “tracking” when the eye is accurately tracked, and “redetection” when the eye tracking fails. When the ocular nose is detected from the entire image, the detection is “detection”. When the ocular nose is detected from the entire image, but is not yet detected, the detection is “detecting”.

  More specifically, if the eye tracking has been mistakenly tracked as an eye as a failure to track the eye, the processing status will be “redetection (mistracking)”, For example, when the driver cannot visually track the eyes by visually recognizing the rear, the processing state becomes “redetection (normal)”.

  As shown in FIG. 3, this processing state is maintained as “tracking” as long as the eye is accurately tracked. Further, when the processing state is “tracking” and the eyebrow is tracked by mistake, the processing state shifts from “tracking” to “redetection (mistracking)”. Further, if the eye cannot be imaged when “tracking” is set, the state is changed from “tracking” to “redetection (normal)”.

  In addition, the processing state transitions to “being detected” after “redetection (false tracking)” and “redetection (normal)”. In the “detecting” state, when an ocular nose is detected from the entire image, transition is made to “detection”, and when it is not detected, “detecting” is maintained. In addition, when the processing state is “detection”, if the ocular nose detected from the entire image is accurately detected, the processing state transits from “detection” to “tracking”, and the ocular nose is accurately detected. If, for example, the eyebrow is detected again as an eye, the processing state returns from “detection” to “detecting”.

  Here, the face part detection unit 40 operates and processes when a signal indicating “detection”, “detecting”, “redetection (false tracking)”, and “redetection (normal)” is input as the processing state. It is the composition which performs. On the other hand, the face part tracking unit 50 is configured to operate and perform processing when a signal indicating “tracking” is input as the processing state. In the processing state “tracking”, the face part detection unit 40 is configured to execute a data collection process (an interocular distance frequency distribution generation process described later).

  With the above configuration, in the present apparatus 1, when the face part can be accurately tracked, the face part tracking unit 50 tracks the eyes and nose from the minimal area, and when the eye tracking fails, the face part detection unit 40 Thus, the eyes and nose are detected from the entire image. Then, when the eye tracking fails, the apparatus 1 determines whether or not the face part detection unit 40 has accurately detected the eye, and when the eye is correctly detected, the processing state is changed from “detection” to “ It is configured to shift to “tracking” and return to the eye tracking process. That is, the apparatus 1 is configured to confirm that the tracking target is an eye and return to the eye tracking process before returning to eye tracking. On the other hand, the apparatus 1 is configured to return the processing state from “detection” to “detecting” when the eye is not accurately detected, and detect the eye again without returning to the eye tracking process. .

  FIG. 4 is a data flow diagram showing details of the face part detection unit 40 shown in FIG. As shown in the figure, the face part detection unit 40 includes a nose coordinate detection unit (coordinate detection unit) 41, an eye coordinate detection unit (coordinate detection unit) 42, and an eye nose coordinate determination unit 43.

  The nose coordinate detection unit 41 detects the coordinates of the nose on the face image based on the face image data from the face image acquisition unit 30 when the processing state is “being detected”. The eye coordinate detection unit 42 detects eye coordinates on the face image based on the face image data from the face image acquisition unit 30 when the processing state is “being detected”. Here, when the driver wears glasses or the like, the eye coordinate detection unit 42 makes it difficult to detect the eyes due to the influence of the glasses or the like, but when the nose coordinates as the second face part are detected. The eye coordinates are detected by using the nose coordinates. Specifically, the eye coordinate detection unit 42 can guarantee the detection of an eye according to a condition such as a nose being under the perpendicular bisector of the eye.

  The eye-nose coordinate determination unit 43 determines the eye-nose coordinates from the nose coordinates and the eye coordinates detected by the nose coordinate detection unit 41 and the eye coordinate detection unit 42 when the processing state is “detection”. In addition, when the processing state is “detection”, the eye-nose coordinate determination unit 43 is configured to determine whether the nose coordinates and the eye coordinates detected by the nose coordinate detection unit 41 and the eye coordinate detection unit 42 are correctly detected. ing. At this time, when the processing state is “re-detection (false tracking)” or “re-detection (normal)”, the eye-nose coordinate determination unit 43 determines the eye-nose coordinates again from the past eye-nose coordinates. Then, the eye-nose coordinate determination unit 43 acquires the feature amount from the past eye-nose coordinate, and determines whether the nose coordinate and the eye coordinate are correctly detected based on the relationship with the feature amount.

  FIG. 5 is a data flow diagram showing details of the eye-nose coordinate determination unit 43 shown in FIG. As shown in the figure, the eye-nose coordinate determination unit 43 includes a positional relationship calculation unit (positional relationship calculation unit) 43a, a frequency distribution generation unit 43b, a class determination unit 43c, and a positional relationship determination unit (positional relationship determination unit) 43d. have.

  The positional relationship calculation unit 43a calculates the feature amount of the coordinates between the eyes and the nose from the coordinates detected by the nose coordinate detection unit 41 and the eye coordinate detection unit 42 when the processing state is “detection”. It is. Here, in this embodiment, the feature amount corresponds to, for example, the distance between the eye and the nose on the image. Therefore, it can be said that the positional relationship calculation unit 43a in the present embodiment calculates the interocular distance.

  The frequency distribution generation unit 43b collects the data of the interocular distance detected by the positional relationship calculation unit 43a in time series, and generates the frequency distribution from the collected data of the interocular distances. . Here, the frequency distribution is a collection of a plurality of interocular distance data, and is distribution data indicating the number of times the interocular distance is detected. Specifically, in the frequency distribution of the interocular distance, when the interocular distance pixel number is “95”, the frequency (number of detections) is “180”, and the interocular distance pixel number is “135”. The data structure has information such that the frequency (number of detections) is “52”.

  The class determination unit 43c classifies the frequency distribution generated by the frequency distribution generation unit 43b according to a predetermined condition. Specifically, the class determination unit 43c classifies a plurality of interocular distance data, which is the basis of the interocular distance frequency distribution, according to a predetermined condition. Here, the predetermined condition includes, for example, the size of the feature amount. That is, the class determination unit 43c classifies the collected data of the distance between the eyes and nose for each predetermined size. Note that the predetermined condition is not limited to this, and may be another one, for example, one based on an inclination described later.

  In addition, the class determination unit 43c determines in which category the interocular distance when the eye tracking fails and the interocular distance when the eye is detected again from the entire image after the failure belong to. Is. That is, the class determining unit 43c determines the distance between the nose when the processing state is “redetection (false tracking)” or “redetection (normal)” and the interocular distance when the processing state is “detection”. It is determined which category it belongs to. Furthermore, the class determination unit 43c is configured to output the determined classification information.

  Here, the class determination unit 43c specifically has the following configuration. FIG. 6 is a data flow diagram showing details of the class determination unit 43c shown in FIG. As shown in the figure, the class determination unit 43c includes a labeling unit (classification unit) 43c1 and a class determination unit (classification determination unit) 43c2.

  The labeling unit 43c1 classifies a plurality of feature amounts detected in time series by the positional relationship calculation unit 43a according to a predetermined condition. Specifically, the labeling unit 43c1 classifies the interocular distance frequency distribution according to the inclination when the vertical axis is the frequency and the horizontal axis is the number of pixels of the interocular distance. The labeling unit 43c1 divides the frequency distribution and labels each of the divided distributions. That is, the labeling unit 43c1 sets each of the divided labels as L (n) (n is an integer equal to or greater than 1), and the label data L (1) to L (n), which is a set of these, is sent to the class determination unit 43c2. Output.

  The class determination unit 43c2 determines which label (classification) the interocular distance when the eye tracking fails and the interocular distance when the eye is detected again from the entire image after the failure belong to. To do. That is, the class determining unit 43c2 determines the distance between the nose when the processing state is “redetection (false tracking)” or “redetection (normal)” and the interocular distance when the processing state is “detection”. Which label (category) it belongs to is determined. Then, the class determining unit 43c2 determines that the distance between the eyes and nose when, for example, the eye tracking fails belongs to, for example, the label L (3), and the distance between the eyes and nose obtained from the entire image after the tracking failure is the label L (2 ) And the like.

  Reference is again made to FIG. The positional relationship determination unit 43d determines whether or not the nose coordinates and the eye coordinates detected by the nose coordinate detection unit 41 and the eye coordinate detection unit 42 are correctly detected based on the classification determined by the class determination unit 43c2. It is. In addition, the positional relationship determination unit 43d sets the processing state to “detection” when the nose coordinates and the eye coordinates are not correctly detected, and detects the nose coordinates and the eye coordinates again. Further, when the nose coordinates and the eye coordinates are correctly detected, the positional relationship determination unit 43d sets the processing state to “tracking”, shifts to the tracking process, determines the eye nose coordinates, and sets the eye nose coordinates. This is output to the eye-nose coordinate signal output unit 60 shown in FIG.

Next, an outline of the operation of the eye tracking device 1 will be described with reference to FIGS. First, when the processing state is “tracking”, as shown in FIGS. 1 and 2, in the eye tracking device 1, the driver's face image is captured by the camera 10, and the video signal is the face of the eye detection device 20. The image is input to the image acquisition unit 30 (S11). Then, when the face image data is stored in the storage area, the face image acquisition unit 30 outputs the face image data (S12). Then, the face part detection unit 40 and the face part tracking unit 50 input face image data (S13, S14).
At this time, the face part detection unit 40 and the face part tracking unit 50 read the processing state information stored in the storage area in the apparatus 20 (S15, S16). Here, since the processing state is “tracking”, the face part tracking unit 50 executes the eye tracking process.

  At this time, the face part tracking unit 50 reads the data of the eye-nose coordinates obtained in the previous process (S17). Then, the face part tracking unit 50 extracts a minimal region from the face image based on the read eye-nose coordinates. Thereafter, the face part tracking unit 50 traces the eye from the minimal region by specifying the eye nose by pattern matching or shape recognition. Then, when the ocular nose is accurately identified, the face part tracking unit 50 outputs a signal indicating the processing state “tracking” (S18).

  Further, the face part tracking unit 50 outputs the data of the identified ocular nose coordinates (S19), and the ocular nose coordinate signal output unit 60 reads the data (S20). Then, the eye-nose coordinate signal output unit 60 outputs the eye-nose coordinate signal to the open / close eye detection device, the side-view detection device, or the like (S21).

  On the other hand, when the eye part and nose cannot be accurately identified, the face part tracking unit 50 outputs a signal indicating the processing state “redetection” (S18). Here, the face part tracking unit 50 detects the opening / closing of the eye by detecting the vertical width of the tracking target and, for example, when the eyes are closed for a certain time (approximately 3 seconds) or longer, the face part other than the eyes (such as eyebrows) ) Is output in error, and a signal indicating the processing state “redetection (mistracking)” is output (S18). In addition to this, the face part tracking unit 50 also has face parts other than the eyes (such as eyebrows) even when the eyes are not closed for a certain period of time (approximately 1 minute) or when an object corresponding to eyebrows cannot be detected on the eyes. Is detected and the processing state “redetection (mistracking)” signal is output (S18). And this apparatus 1 will perform the process when a process state is "redetection (false tracking)."

  Further, the face part tracking unit 50 tracks the eyes accurately, such as when the hand and handle overlap the face, or the driver looks back, but the eyes are not captured because the eyes themselves are no longer imaged. When tracking becomes impossible, a signal indicating the processing state “redetection (normal)” is output (S18). Then, the apparatus 1 executes processing when the processing state is “redetection (normal)”.

  Next, an outline of the operation when the processing state is “redetection (false tracking)” will be described. As shown in FIG. 2, when the signal of the processing state “re-detection (mistracking)” is input (S15), the face part detection unit 40 reads the previous ophthalmic nose coordinate data when tracking fails (S22). . That is, as shown in FIGS. 3 and 4, when the positional relationship calculation unit 43a in the eye-nose coordinate determination unit 43 inputs a signal of the processing state “redetection (mistracking)” (S31, S51), tracking failure The eye nose coordinate data is read (S32, S52).

  And the positional relationship calculation part 43a calculates | requires the interocular distance from the eye-nose coordinate data at the time of failure. Next, the positional relationship calculation unit 43a outputs data on the distance between the eyes and nose (S53). Thereafter, the frequency distribution generation unit 43b inputs the data of the interocular distance (S54). Then, the frequency distribution generation unit 43b generates an interocular distance frequency distribution. At this time, the frequency distribution generation unit 43b first obtains a class from the interocular distance data. FIG. 7 is an explanatory diagram illustrating an example of the interocular distance frequency distribution. In the frequency distribution shown in the figure, the class is set for every 20 pixels. Then, the frequency distribution generation unit 43b determines whether the interocular distance at the time of failure corresponds to any class. For example, the frequency distribution generation unit 43b determines that it corresponds to a class having 120 to 140 pixels. Next, the frequency distribution generation unit 43b outputs the obtained class information (S55).

  Further, when the frequency distribution generation unit 43b obtains the class, the frequency distribution generation unit 43b reads data of the interocular distance frequency distribution generated in the past (S56). Then, as shown in FIG. 6, the frequency distribution generation unit 43b adds the frequency to the previously obtained class (for example, a class having 120 to 140 pixels), and the class to which the oldest data belongs (for example, 80 to 100 pixels). Delete the information from (class). Then, the frequency distribution generation unit 43b generates a new interocular distance frequency distribution and outputs this data (S57).

  Next, the class determination unit 43c inputs a signal indicating the processing state “redetection (false tracking)” (S58), and further reads the data of the class and the interocular distance frequency distribution (S59, S60). That is, the labeling unit 43c1 shown in FIG. 6 inputs a signal of the processing state “re-detection (false tracking)” (S71), and reads the data of the class and the interocular distance frequency distribution (S72, S73).

  Then, the labeling unit 43c1 classifies the interocular distance frequency distribution according to a predetermined condition. Specifically, in the example illustrated in FIG. 7, the labeling unit 43c1 classifies each class by the class in which the slope between the classes changes from negative to positive, and sets each class as labels L (1) to L (3). . Thereafter, the labeling unit 43c1 outputs the label data L (1) to L (3) (S74).

  Further, when the signal of the processing state “redetection (false tracking)” is input (S75), the class determination unit 43c2 reads the class data and the label data L (S76, S77). Next, in the example shown in FIG. 7, the class determination unit 43c2 determines which label L (1) to L (3) the class obtained by the frequency distribution generation unit 43b belongs to. In this case, the class determination unit 43c2 determines that the class belongs to the label L (3), for example.

  Thereafter, the class determining unit 43c2 registers the label L (3) as the class c and registers the class c as the previous class pc. Next, the class determination unit 43c2 outputs the data of the previous class pc (S61, S78). Then, as shown in FIG. 5, the positional relationship determination unit 43d reads a signal indicating that the processing state is “redetection (false tracking)” (S68), and outputs a signal indicating that the processing state is “detecting”. (S63).

  Further, the positional relationship determination unit 43d reads the data of the previous class pc (S62) and stores it. By storing the data of the previous class pc that is the data at the time of tracking failure in this way, the positional relationship determination unit 43d accurately determines the previous class pc when the eye nose is detected from the entire image later. It can be used as a reference whether or not it has been detected.

  Note that the processing when the processing state is “redetection (normal)” is basically the same as “redetection (false tracking)”. Accordingly, processing is performed in the same manner as described above.

Next, an outline of the operation when the processing state is “detecting” will be described. When the processing state is “detecting”, as shown in FIGS. 1 and 2, in the eye tracking device 1, the driver's face image is captured by the camera 10, and the video signal is acquired by the eye detection device 20. Input to the unit 30 (S11). Then, when the face image data is stored in the storage area, the face image acquisition unit 30 outputs the face image data (S12). Then, the face part detection unit 40 and the face part tracking unit 50 input face image data (S13, S14).
At this time, the face part detection unit 40 and the face part tracking unit 50 read the processing state information stored in the storage area in the apparatus 20 (S15, S16). Since the processing state is “being detected”, the face part detection unit 40 executes an eye detection process.

  That is, the nose coordinate detection unit 41 and the eye coordinate detection unit 42 illustrated in FIG. 4 receive the processing state “detecting” signal (S33, S34), and detect the nose and eye coordinates. At this time, the nose coordinate detection unit 41 and the eye coordinate detection unit 42 input face image data (S35, S36).

  And the nose coordinate detection part 41 detects the coordinate value of a nose as shown in FIGS. FIG. 8 is an explanatory diagram showing a face image and nose coordinates obtained from the face image, (a) shows the face image, and (b) is extracted along the pixel rows in the vertical and horizontal directions of the image. Extraction points are shown, and (c) shows nose candidates.

  As shown in the figure, when a face image as shown in FIG. 8A is input, the nose coordinate detection unit 41 detects the pixel density along the vertical pixel row (vertical line) of the face image. , One element is determined for each local increase in density in the pixel column and is used as an extraction point. Further, the nose coordinate detection unit 41 determines the pixel density along the horizontal pixel row (horizontal line) of the face image after setting the extraction point in the vertical direction of the image, and locally detects the density of the pixel row. One element is determined for each increase and is used as an extraction point.

  9 and 10 are graphs showing the density value (light quantity) of each pixel for each pixel column, and FIG. 9 shows the density value of the pixel column Ya in the image vertical direction shown in FIG. Reference numeral 10 denotes the density value of the pixel column Xa in the horizontal direction of the image.

  First, as shown in FIG. 9, the density value locally decreases at the nostril and mouth. For this reason, the nose coordinate detection part 41 will determine the pixel of a nostril and a mouth part as an extraction point about Ya line. Moreover, as shown in FIG. 10, the density value locally decreases in both nostril portions. For this reason, the nose coordinate detection unit 41 determines pixels of both nostrils as extraction points for the Xa line.

  The extraction points determined in the vertical and horizontal directions are shown in FIG. In addition, when the nose coordinate detection unit 41 determines extraction points in the vertical and horizontal directions of the image, a point where the vertical extraction point and the horizontal extraction point overlap is set as a nostril candidate point. Here, two nostrils are adjacent in the horizontal direction of the normal image. For this reason, the nose coordinate detection unit 41 leaves candidate points that can be two pairs in the horizontal direction of the image as candidates for nostrils, and deletes other candidate points. Thereby, the nose coordinate detection part 41 will acquire the image shown in FIG.8 (c), and will detect a nose coordinate.

  In addition to this, as a method for detecting the nose coordinates, as disclosed in Japanese Patent Laid-Open No. 2002-251617, a method for obtaining the coordinates of the nose by detecting the nose muscles may be used.

  When the nose coordinates are obtained as described above, the nose coordinate detection unit 41 outputs nose coordinate data (S37). Moreover, the eye coordinate detection part 42 detects the coordinate value of an eye as shown in FIGS. 11 is an explanatory diagram of a face image, FIG. 12 is an explanatory diagram showing a partial area of the face image, and FIG. 13 is a graph showing the density value of the pixel column Yb in the image vertical direction shown in FIG. It is.

  As shown in the figure, when a face image as shown in FIG. 11 is input, the eye coordinate detection unit 42 detects the pixel density along a vertical pixel row (vertical line) of the face image, and the pixel One element is determined for each local increase in concentration in the column and is used as an extraction point. Then, the eye coordinate detection unit 42 determines extraction points for all lines in the pixel column in the vertical direction of the image.

  Here, as shown in FIG. 12, the density value of the pixel column Yb in the vertical direction of the image is locally reduced at the eye, the spectacle frame, and the mouth. For this reason, the eye coordinate detection part 42 will determine the pixel of a nose eye, a spectacles frame, and a mouth part as an extraction point about Yb line.

  Thereafter, the eye coordinate detection unit 42 obtains a pixel group in which the extraction points are assembled. Here, the eye has a shape extending in the lateral direction. For this reason, the eye pixel group shown in FIG. 13 is continuous in the horizontal direction of the image in shape. For this reason, the eye coordinate detection unit 42 detects an eye pixel group by selecting a pixel group continuous in the horizontal direction of the image from various pixel groups. Then, the eye coordinate detection unit 42 acquires an eye coordinate value from the detected eye pixel group.

  The eyeglass frame and mouth are also connected in the horizontal direction of the image in shape, and thus may be detected as eyes. In view of this, the eye coordinate detection unit 42 of the present embodiment, as described in Japanese Patent Laid-Open No. 11-96379, takes advantage of the preset eye length and the fact that the eyes are symmetrical. Identify and detect eye coordinates.

  Furthermore, the eye coordinate detection unit 42 of the present embodiment reads the data of the nose coordinates from the nose coordinate detection unit 41 (S38). Then, the eye coordinate detection unit 42 detects an eye using the nose coordinates detected by the nose coordinate detection unit 41. That is, the eye coordinate detection unit 42 identifies the eye when the nose is below the perpendicular bisector of the eye.

  As described above, the eye coordinate detection unit 42 detects the eye coordinates. Then, the eye coordinate detection unit 42 outputs eye coordinate data (S39). Here, the eye coordinates and the nose coordinates will be further described. FIG. 14 is an explanatory diagram of eye coordinates and nose coordinates. Note that FIG. 14 also shows coordinates when the eyebrows are erroneously detected as an eye in addition to the nose.

  As shown in the figure, the eye coordinates of the right eye are about 250 to 300 pixels for the X coordinate and about 200 pixels for the Y coordinate. Further, the eye coordinates of the left eye are about 400 to 450 pixels for the X coordinate and about 200 pixels for the Y coordinate. Furthermore, the coordinates of the right eyebrow are about 230 to 350 pixels for the X coordinate and about 150 to 200 pixels for the Y coordinate. In addition, the left eyebrow coordinates are about 380 to 550 pixels for the X coordinate and about 100 to 200 pixels for the Y coordinate.

  And nose coordinates exist under the perpendicular bisector of these eyes and eyebrows. For this reason, as described above, it can be said that there is a possibility of mistracking the eyebrows as being eyes. On the other hand, it can be said that it can be a standard for specifying the eye by having the nose under the perpendicular bisector of the eye.

  Then, the eye-nose coordinate determination unit 43 inputs the data of the nose coordinate and the eye coordinate (S40, S41). Next, the eye-nose coordinate determination unit 43 outputs a signal indicating the processing state “detection” (S42). Thereafter, the apparatus 1 executes processing when the processing state is “detection”.

  When nose coordinate and eye coordinate data cannot be input, that is, when either the nose coordinate detection unit 41 or the eye coordinate detection unit 42 cannot detect the nose coordinate or the eye coordinate, the eye nose coordinate determination unit 43 outputs a signal indicating that the processing state is “detecting” (S42). As a result, the apparatus 1 maintains the “detecting” processing state, and again executes the processing when the processing state is “detecting”.

  Next, an outline of the operation when the processing state is “detection” will be described. As shown in FIG. 5, when the processing state is “detection”, the positional relationship calculation unit 43a inputs the newly detected nose coordinate and eye coordinate data (S64). And the positional relationship calculation part 43a calculates | requires the distance between eyes nose from these nose coordinates and eye coordinates, and outputs the data (S53).

  After that, the frequency distribution generation unit 43b inputs the data of the interocular distance (S54), obtains the class of the inputted interocular distance as described above, and outputs the obtained class information (S55). Further, when the frequency distribution generation unit 43b obtains the class, it reads the data of the interocular distance frequency distribution generated in the past (S56), generates a new interocular distance frequency distribution, and outputs this data. (S57).

  Then, as shown in FIG. 6, the labeling unit 43c1 of the class determination unit 43c inputs the processing state “detection” signal (S71), and reads the data of the class and interocular distance frequency distribution (S72, S73). ).

  Next, the labeling unit 43c1 classifies the interocular distance frequency distribution according to a predetermined condition, and outputs label data L (1) to L (3) (S74). Thereafter, when the signal of the processing state “detection” is input (S75), the class determination unit 43c2 reads the class data and the label data L (S76, S77).

  Then, the class determination unit 43c2 determines which label the class obtained by the frequency distribution generation unit 43b belongs to, and sets the determined label as the class c. Next, the class determination unit 43c2 registers this class c as the current class cc, and outputs data of the current class cc (S65, S79). Thereafter, as shown in FIG. 5, the positional relationship determination unit 43d reads a signal indicating that the processing state is “detection” and data of the current class cc (S66, S68).

  Here, in the positional relationship determination unit 43d, the previous class pc is stored in the processing of the processing state “redetection (false tracking)” or “redetection (normal)”. Next, a process when the processing state transits to “detection” through “redetection (false tracking)” will be described. When the processing state “redetection (false tracking)” is performed, the positional relationship determination unit 43d registers, for example, the class c when the eyebrow is tracked as the previous class pc. Therefore, at the time of the processing state “detection”, the previous class pc that reflects the interocular distance when the eye is mistracked as an eyebrow, and the current class cc that reflects the newly detected interocular distance. It will be obtained.

  Then, the positional relationship determination unit 43d compares the previous class pc with the previous class pc. Here, when the processing state transitions from “redetection (false tracking)” to “detection” via “detecting”, when the current class cc is the same as the previous class pc, the positional relationship determination unit 43d A signal indicating that the processing state is “detecting” is output (S63). And this apparatus 1 will perform the process when a process state is "detecting."

  On the other hand, when the processing state transitions from “redetection (false tracking)” to “detection” through “detecting” and the current class cc is different from the previous class pc, the processing state is set to “tracking”. To do. That is, since “re-detection (mistracking)” has been performed, the fact that the current class cc is different from the previous class pc means that the eye has been detected from a distance different from the interocular distance when mistracked in the past. It can be said that it has not made the same mistake as at the time of mistracking. For this reason, it can be determined that the eye has been detected accurately, and the positional relationship determination unit 43d outputs a signal indicating the processing state “tracking” (S63, S23).

  Then, the positional relationship determination unit 43d (the eye-nose coordinate determination unit 43 in FIG. 4) outputs the data of the eye-nose coordinates (S43, S67), and the eye-nose coordinate signal output unit 60 shown in FIG. Is input (S20). Thereafter, the eye-nose coordinate signal output unit 60 outputs an eye-nose coordinate signal to an open / closed eye detection device, a side-view detection device or the like (S21).

  When the processing state transitions from “redetection (normal)” to “detection” through “detection”, when the current class cc is different from the previous class pc, the positional relationship determination unit 43d determines that the processing state “detection” A “medium” signal is output (S63). And this apparatus 1 will perform the process when a process state is "detecting."

  On the other hand, when the processing state transitions from “redetection (normal)” to “detection” via “detecting” and the current class cc is the same as the previous class pc, the processing state is set to “tracking”. To do. In other words, since “re-detection (normal)” has passed, the fact that the current class cc is the same as the previous class pc means that the eye was detected from the same distance as the interocular distance when accurately tracking in the past. Therefore, it can be determined that the detection of the eyes has been performed accurately. For this reason, the positional relationship determination unit 43d outputs a signal indicating the processing state “tracking” (S63, S23).

  As described above, the present apparatus 1 determines whether or not the eye is correctly detected by comparing the interocular distance at the time of tracking failure with the interocular distance newly detected from the face image. Can be satisfactorily returned to the tracking process.

  Next, processing contents of each component of the apparatus 1 will be described in detail based on a flowchart. FIG. 15 is a flowchart showing a detailed operation of the positional relationship calculation unit 43a shown in FIG. First, the positional relationship calculation unit 43a reads a processing state signal and determines to which one the current processing state belongs (ST10). Here, when the processing state is “detection” or “tracking”, the positional relationship calculation unit 43a reads data of eye coordinates and nose coordinates. Then, the positional relationship calculation unit 43a acquires the coordinate values (rx, ry) for the right eye and the coordinate values (lx, ly) for the left eye from the eye coordinates. Further, the positional relationship calculation unit 43a acquires the coordinate value (nx, ny) of the nose from the nose coordinates (ST11).

  When the processing state is “re-detection (mistracking)” or “re-detection (normal)”, the positional relationship calculation unit 43a reads the data of the eye-nose coordinates at the time of the previous tracking failure. Then, the positional relationship calculation unit 43a obtains the right eye coordinate value (rx, ry), the left eye coordinate value (lx, ly), and the nose coordinate value (nx, ny) from the eye nose coordinates ( ST12).

Next, after step ST11 or ST12, the positional relationship calculation unit 43a calculates the interocular distance based on the following equation 1 (ST12). Note that I indicates the interocular distance.

  Specifically, the interocular distance I is as shown in FIG. FIG. 16 is an explanatory diagram of the interocular distance I, where (a) shows the face image, and (b) shows the interocular distance I between each face part. For example, it is assumed that the face image of FIG. 16A is acquired by the camera 10, and the coordinate values of the right eye A, the left eye B, and the nose C are acquired by the nose coordinate detection unit 41 and the eye coordinate detection unit 42. At this time, the positional relationship calculation unit 43a obtains the center point D between the right eye A and the left eye B from the equation (1), obtains the distance between the center point D and the nose C, and calculates the interocular distance I and To do.

  And after calculating | requiring the interocular distance I, the positional relationship calculation part 43a will output the information of the calculated interocular distance I to the frequency distribution production | generation part 43b.

  By the way, when the processing state is “detecting”, the coordinates of the eyes and nose have not been detected yet, so the positional relationship calculation unit 43a sets the interocular distance I to “NULL” (ST14). Then, the positional relationship calculation unit 43a outputs information about the interocular distance I set to “NULL” to the frequency distribution generation unit 43b. And the process by the positional relationship calculation part 43a is complete | finished.

  Next, processing of the frequency distribution generation unit 43b will be described. FIG. 17 is a flowchart showing a detailed operation of the frequency distribution generation unit 43b shown in FIG. First, the frequency distribution generation unit 43b reads data on the interocular distance I and determines whether or not the interocular distance I is “NULL” (ST20). Here, when it is determined that the interocular distance I is “NULL” (ST20: YES), the frequency distribution generation unit 43b sets the class d to “NULL” (ST21). Then, the frequency distribution generation unit 43b outputs the class d information set to “NULL” to the class determination unit 43c, and the processing by the frequency distribution generation unit 43b ends.

  On the other hand, when determining that the interocular distance I is not “NULL” (ST20: NO), the frequency distribution generation unit 43b deletes the oldest class data D (i) from the frequency distribution H (ST22). That is, as described with reference to FIG. 7, the frequency distribution generation unit 43 b performs the same processing as when class data having 80 to 100 pixels is deleted.

  Thereafter, the frequency distribution generation unit 43b calculates a class d of the interocular distance I (ST23). Here, the class is obtained by a calculation formula d = int (I / W). W is the data width between classes. That is, W indicates a width of 20 pixels as shown in FIG.

Note that if the data width W is too large, the amount of information is lost. If the data width W is too small, a frequency distribution having irregular irregularities is obtained. That is, it is desirable that the data width W is appropriate and the number of classes is also appropriate. Here, the number of classes can be obtained in advance based on the Sturges formula.

Here, C is a class number and n is the number of data. Specifically, the class number C is as shown in Table 1 below.

  Thus, the frequency distribution generation unit 43b sets the class number C to an appropriate number, and sets the class data width W based on this. Thereby, the frequency distribution generation unit 43b generates an appropriate interocular distance frequency distribution.

  Next, the frequency distribution generation unit 43b adds the calculated class d to the class data D and updates the frequency distribution H (ST24). That is, as described with reference to FIG. 7, the frequency distribution generation unit 43 b performs the same process as when class data having 120 to 140 pixels is added.

  Thereby, an example of the obtained interocular distance frequency distribution is shown. FIG. 18 is an explanatory diagram illustrating an example of the interocular distance frequency distribution. In FIG. 18, the data width W is 5 pixels.

  As shown in the figure, the distance between the eyes and the nose is the highest in the vicinity of about 95 pixels. On the other hand, the distance between the eyebrows and the nose is the highest in the vicinity of about 140 pixels. That is, when the eye is accurately tracked, the interocular distance I is calculated to be about 95 pixels, whereas when the eyebrows are mistakenly tracked as the eye, When the distance I is calculated, the value is about 140 pixels. The frequency distribution generation unit 43b generates such an interocular distance frequency distribution and provides the information to the class determination unit 43c. In addition, all the data shown in the figure are data obtained by adding both data.

  Next, detailed processing by the labeling unit 43c1 of the class determination unit 43c will be described. FIG. 19 is a flowchart showing a detailed operation of the labeling unit 43c1 shown in FIG. First, the labeling unit 43c1 reads a processing state signal and determines whether the processing state is “redetection (mistracking)” or “redetection (normal)” (ST30).

  Here, when it is determined that the processing state is neither “re-detection (false tracking)” nor “re-detection (normal)” (ST30: NO), the process by the labeling unit 43c1 ends. On the other hand, when it is determined that the processing state is “redetection (false tracking)” or “redetection (normal)” (ST30: YES), the labeling unit 43c1 determines whether the class d is “NULL”. Judgment is made (ST31).

  When it is determined that the class d is “NULL” (ST31: YES), the process by the labeling unit 43c1 ends. On the other hand, when it is determined that the class d is not “NULL” (ST31: NO), the labeling unit 43c1 clears the label. That is, in FIG. 7, when the labels L (1) to L (3) have been set in the previous process, the data relating to the labels L (1) to L (3) will be erased. Also, “1” is assigned to the label number T (ST32).

  Thereafter, the labeling unit 43c1 obtains the frequency difference (slope) px between the class currently being processed and the next class adjacent to it (ST33). Further, the labeling unit 43c1 employs the difference ox between the previous classes when there is no difference in the frequency of the adjacent classes (when the inclination is 0).

  Then, the labeling unit 43c1 updates the label number T when the frequency of the class currently being processed is a minimum point (ST34). That is, the labeling unit 43c1 increments the label number T to “3” if the label L (2) has already been set. The labeling unit 43c1 determines that the frequency of the currently processed class is higher when the difference ox between the previous classes is smaller than “0” and the frequency difference px between the adjacent next classes is larger than “0”. Judge that it has a minimum point.

  Next, when it is determined in step ST34 that the point is a local minimum, the labeling unit 43c1 sets a label (ST35). At this time, if the label number T is “3”, the labeling unit 43c1 includes the class currently being processed so that there is no overlap with the labels L (1) and L (2), and the label Set L (3). The labeling unit 43c1 sets a label without including the currently processed class when the frequency of the currently processed class is “0”.

  Thereafter, the labeling unit 43c1 determines whether or not all classes have been processed (ST36). If it is determined that all classes have not been processed (ST36: NO), the labeling unit 43c1 returns the process to step ST33. On the other hand, if it is determined that all classes have been processed (ST36: YES), the label data L is output to the class determining unit 43c2, and the process by the labeling unit 43c1 is terminated.

  Specifically, the label data L is as shown in FIG. FIG. 20 is an explanatory diagram showing an image of label data. In the example shown in the figure, three labels are assigned to the interocular distance frequency distribution. Here, in the example shown in FIG. 20, there is a class with the frequency “0”, so that class is not included in any label. The labeling unit 43c1 outputs such label data L to the class determination unit 43c2.

  Next, the detailed operation of the class determination unit 43c2 will be described. FIG. 21 is a flowchart showing a detailed operation of the class determination unit 43c2 shown in FIG. As shown in the figure, first, the class determining unit 43c2 reads the data of the class d and determines whether or not the class d is “NULL” (ST40). Here, when it is determined that the class d is “NULL” (ST40: YES), the processing by the class determination unit 43c2 ends.

  On the other hand, when it is determined that the class d is not “NULL” (ST40: NO), the class determination unit 43c2 sets the label to which the class d belongs to class c (ST41). Then, the class determining unit 43c2 reads the processing state signal and determines to which of the current processing state it belongs (ST42).

  Here, when the processing state is “detection”, the class determination unit 43c2 registers the class c as the current class cc (ST43). When the processing state is “redetection (false tracking)” or “redetection (normal)”, the class determination unit 43c2 sets the class c reflecting the interocular distance at the time of the previous tracking failure to the previous class pc. (ST44).

  Thereafter, the class determination unit 43c2 outputs the data of the previous class pc or the current class cc to the positional relationship determination unit 43d. And the process by the class judgment part 43c2 is complete | finished. Furthermore, when the processing state is “detecting” or “tracking”, the class determination unit 43c2 ends the processing illustrated in FIG. 21 without registering the class c.

  FIG. 22 is a flowchart showing a detailed operation of the positional relationship determination unit 43d shown in FIG. First, the positional relationship determination unit 43d reads a processing state signal and determines which processing state it belongs to (ST50).

  Here, when the processing state is “being detected” or “tracking”, the processing by the positional relationship determination unit 43d ends. On the other hand, when the processing state is “redetection (false tracking)” or “redetection (normal)”, the positional relationship determination unit 43d updates the internal state (ST51). At this time, when the processing state is “redetection (false tracking)”, the positional relationship determination unit 43d sets the internal state to “redetection (mistracking)” and the processing state is “redetection (normal)”. In some cases, the internal state is “re-detection (normal)”.

  Thereafter, the positional relationship determination unit 43d sets the processing state to “being detected” (ST52). Thereafter, the positional relationship determination unit 43d sets the number of retries as an initial value (ST53). Next, the positional relationship determination unit 43d ends the process illustrated in FIG. 22, and the apparatus 1 performs the process of “detecting” the process state.

  Here, the number of retries means the upper limit number of times for returning the processing state from “detected” to “detecting”. That is, the present apparatus 1 detects an eye-nose from the entire image after tracking failure, and when this detection itself is highly likely to be erroneous (an undesired result is obtained by comparing the previous class pc and the current class cc). In this case, the processing state is returned from “detected” to “detecting”, but the number of times more than the number of retries is not returned.

  If the processing state is “detection”, the positional relationship determination unit 43d determines whether or not the previous class is “NULL” (ST54). Here, when the previous class is “NULL” (ST54: YES), that is, when neither “re-detection (false tracking)” nor “re-detection (normal)” has passed, such as when the device 1 is started, The positional relationship determination unit 43d acquires the eye-nose coordinates based on the eye coordinate and nose coordinate data (signal S69 in FIG. 5) (ST62). Then, the positional relationship determination unit 43d sets the processing state to “tracking” (ST63), and ends the processing illustrated in FIG.

  On the other hand, when the previous class is not “NULL” (ST54: NO), that is, any one of “re-detection (mistracking)” and “re-detection (normal)” has passed at least once. Reads the internal state and determines which internal state it belongs to (ST55).

  Here, when the internal state is “redetection (normal)”, the positional relationship determination unit 43d determines whether or not the current class cc and the previous class pc are the same (ST56). If it is determined that the current class cc and the previous class pc are the same (ST56: YES), the process proceeds to step ST62.

  That is, when the internal state is “redetection (normal)”, it can be said that the processing state has transitioned from “tracking” to “detection” via “redetection (normal)”. For this reason, it can be said that the previous class pc reflects the interocular distance I when the eyes are accurately detected. The fact that the current class cc is the same as the previous class pc indicates that It is highly probable that they have been detected. Therefore, the process proceeds to step ST62, and the positional relationship determination unit 43d acquires the eye-nose coordinates from the eye coordinate and nose coordinate data (ST62). Then, the positional relationship determination unit 43d sets the processing state to “tracking” (S63). Thereby, this apparatus 1 makes favorable return to a tracking process. Thereafter, the processing by the positional relationship determination unit 43d ends.

  If it is determined that the current class cc and the previous class pc are not the same (ST56: NO), the process proceeds to step ST58. Here, if the current class cc is different from the previous class pc, it can be said that the eyes are detected from a distance different from the interocular distance I when the eyes are accurately detected. For this reason, it can be said that there is a low possibility that the eyes have been accurately detected from the entire image, and the positional relationship determination unit 43d advances the process to step ST58 without acquiring the eye-nose coordinates.

  In step ST58, the positional relationship determination unit 43d decrements the number of retries (ST58). Then, the positional relationship determination unit 43d sets the processing state to “being detected” (ST59). Thereby, this apparatus 1 will detect an ocular nose again from a whole image.

  Next, the positional relationship determination unit 43d determines whether or not the number of retries is “0” (ST60). Here, when it is determined that the number of retries is not “0” (ST60: NO), the processing by the positional relationship determination unit 43d ends.

  On the other hand, when it is determined that the number of retries is “0” (ST60: YES), the positional relationship determination unit 43d sets the internal state to “NULL” (ST61). Thereby, in step ST55 mentioned above, it is forced to shift to step ST62. And it is trying to prevent the situation that it continues without being able to detect an eye correctly.

  By the way, when it is determined in step ST55 that the internal state is “redetection (false tracking)”, the positional relationship determination unit 43d determines whether or not the current class cc and the previous class pc are different (ST57). . If it is determined that the current class cc and the previous class pc are different (ST57: YES), the process proceeds to step ST62.

  That is, when the internal state is “redetection (false tracking)”, it can be said that the processing state has transitioned from “tracking” to “detection” via “redetection (tracking)”. For this reason, the previous class pc reflects the interocular distance I when the eyebrow is tracked by mistake, and the current class cc is different from the previous class pc. It can be said that there is a high probability of detection. Therefore, the process proceeds to step ST62, and the positional relationship determination unit 43d acquires the eye-nose coordinates from the eye coordinate and nose coordinate data (ST62). Then, the positional relationship determination unit 43d sets the processing state to “tracking” (S63). Thereby, this apparatus 1 makes favorable return to a tracking process. Thereafter, the processing by the positional relationship determination unit 43d ends.

  If it is determined that the current class cc and the previous class pc are not different (ST57: NO), that is, if it is determined that the current class cc and the previous class pc are the same, the process proceeds to step ST58. Here, if the current class cc is the same as the previous class pc, it can be said that the eye is detected from the same distance as the interocular distance I when the eyebrow is tracked by mistake. For this reason, it can be said that there is a low possibility that the eyes have been accurately detected from the entire image, and the positional relationship determination unit 43d advances the process to step ST58 without acquiring the eye-nose coordinates. And the positional relationship determination part 43d performs the process of step ST58-ST61 similarly to the above, and complete | finishes the process shown in FIG.

  In this manner, the eye tracking device 1 according to the present embodiment obtains the feature amount of the coordinates between the eyes and the second face part other than the eyes from the face image data, and detects them in time series. A plurality of feature amounts are classified according to a predetermined condition (for example, the size of the feature amount). For this reason, similar feature amounts are classified into the same category.

  When the driver's eyes are tracked incorrectly, that is, when the eyebrows are mistakenly tracked as being eyes, it is determined to which category the feature amount at that time belongs. In addition, it is determined to which category the feature amount when the eye is detected after being tracked by mistake belongs. For this reason, it is possible to determine the classification to which the wrong tracking is performed and the subsequent detection.

  Next, when the determined classifications are different, the detected tracking target is determined to be an eye. Here, when the eyebrow is mistakenly tracked as an eye, the feature amount is different from when the eye is tracked accurately. For this reason, at the time of eyebrow tracking, the classification of the feature quantity belonging to the time when the eye is accurately tracked is different. Therefore, when the newly detected category is the same as the category when the eyebrow was tracked incorrectly, it can be determined that the eyebrow was detected at the time of detection, and the category when the eyebrow was tracked incorrectly If it is different from the above, it can be determined that the eye is accurately detected even at the time of detection.

  Therefore, when eye tracking fails, it is possible to satisfactorily return to eye tracking processing.

  In addition, when the driver's eyes have been tracked but the eyes can no longer be tracked, that is, the driver has been tracking the eyes, for example, the driver can not see the back and image the eyes. If it is determined, it is determined to which category the feature quantity at that time belongs. In addition, it is determined to which category the feature amount when the eye is detected after tracking becomes impossible. For this reason, it is possible to determine the classification to which both the case where tracking becomes impossible and the case where detection is performed thereafter.

  Then, when both of the determined classifications are the same, the newly detected tracking target is determined to be an eye. Here, for example, when the driver is tracking the eye, for example, the driver can not capture the eye by visually recognizing the back. It can be said. For this reason, the feature amount when tracking becomes impossible is substantially the same as that when tracking the eye accurately. Therefore, if the category when the eye is newly detected is different from the category when the eye can no longer be tracked, it can be determined that the eyebrows have been detected at the time of detection, and the same as the category when the eye can no longer be tracked When it becomes, it can be determined that the eye has been accurately detected even at the time of detection.

  Therefore, when the eye tracking fails, the return to the eye tracking process can be performed more satisfactorily.

  In addition, since the coordinates of at least one of the nose and the mouth are detected as the second face part, the nose is more stable than eyes that are easily affected by changes in the light environment or the like (especially when wearing glasses). Since the coordinates of the facial parts such as the mouth and the mouth are detected, it is possible to reduce the influence of the light environment in the calculation of the feature amount.

  In addition, since the distance between the eye and the second facial part on the image is calculated as the feature amount, the eye is correctly corrected even when the driver's posture changes compared to the method using the absolute position of the eye detected in the image. Can be detected.

  Next, a second embodiment of the present invention will be described. The eye tracking device 2 according to the second embodiment is the same as that of the first embodiment, but the eye-nose coordinate determination unit 43 is newly provided with an initialization processing unit (subjected person change detection means) 43e. This is different from the first embodiment.

  FIG. 23 is a data flow diagram showing details of the eye-nose coordinate determination unit 43 shown in FIG. As shown in the figure, the eye-nose coordinate determination unit 43 newly includes an initialization processing unit 43e.

  The initialization processing unit 43e detects that the vehicle driver has changed. Moreover, the initialization process part 43e will initialize the distance frequency distribution between eyes and nose, if it detects that a driver | operator changed. That is, the initialization processing unit 43e initializes the interocular distance frequency distribution composed of a plurality of interocular distance data calculated by the positional relationship calculation unit 43a. Further, the initialization processing unit 43e is configured to initialize the previous class pc in addition to the interocular distance frequency distribution. From the above, it can be said that the initialization processing unit 43e prevents erroneous processing using the data of the previous driver, even though the driver has changed.

  Such an initialization processing unit 43e reads data of nose coordinates and eye coordinates as needed (S90). Here, as long as the data of the nose coordinates and the eye coordinates can be read, the initialization processing unit 43e does not initialize the interocular distance frequency distribution and the previous class pc.

  However, if the data of the nose coordinates and the eye coordinates cannot be read continuously for a certain period of time, the initialization processing unit 43e determines that the driver has left the driver's seat. Then, the initialization processing unit 43e determines that there is a possibility that the driver will change because another driver may be seated in the driver's seat next, and the interocular distance frequency distribution and the previous The class pc is initialized (S91, S92).

  As a result, even if another driver is actually seated in the driver's seat, the frequency distribution between the distances between the eyes and the nose is regenerated, and erroneous processing can be prevented.

  In this way, according to the eye tracking device 2 according to the second embodiment, as in the first embodiment, when eye tracking has failed, it is possible to satisfactorily return to eye tracking processing. . In addition, when eye tracking fails, it is possible to perform better return to eye tracking processing. Furthermore, the calculation of the feature amount can be made less susceptible to the influence of the light environment, and the eye can be detected correctly even if the driver's posture changes.

  Furthermore, according to the second embodiment, the possibility of the driver's replacement is determined and information on a plurality of feature amounts is initialized, that is, the interocular distance frequency distribution is initialized. In addition, it is possible to avoid processing based on the previous driver information. Therefore, the return to the tracking process can be appropriately performed for each driver.

  Next, a third embodiment of the present invention will be described. The eye tracking device 3 according to the third embodiment is the same as that of the second embodiment, but the method by which the initialization processing unit 43e determines the possibility of driver change is that of the second embodiment. Is different.

  FIG. 24 is a hardware configuration diagram of the eye tracking device 3 according to the third embodiment, FIG. 25 is a data flow diagram showing details of the eye detection device 20 shown in FIG. 24, and FIG. 26 is a data flow diagram showing details of the eye-nose coordinate determining unit 43 shown in FIG.

  First, as shown in FIG. 24, the eye detection device 20 is configured to newly input a vehicle signal. Here, examples of the vehicle signal include a vehicle speed signal, a shift position signal, a key switch signal, and a main switch signal.

  As shown in FIGS. 25 and 26, the initialization processing unit 43e of the eye-nose coordinate determination unit 43 is configured to input a vehicle signal. The initialization processing unit 43e initializes the interocular distance frequency distribution and the previous class pc according to the state of the vehicle signal.

  Such an initialization processing unit 43e inputs vehicle signals as needed (S44, S93). For example, when it is detected as a vehicle signal that the vehicle speed is “0” continuously for a certain period of time, or when it is detected that the shift position is at the parking position for a certain period of time, the driver from the vehicle It is determined that there is a possibility that the driver will change.

  In addition, the initialization processing unit 43e determines that the driver has left the vehicle when detecting that the main switch has been turned off or when detecting the opening / closing of the door from the door opening / closing switch signal. Detect that there is a possibility of alternation.

  If the initialization processing unit 43e detects that there is a possibility that the driver may change, the initialization processing unit 43e initializes the interocular distance frequency distribution and the previous class pc (S91, S92). As a result, similar to the second embodiment, even if another driver is seated in the driver's seat next time, the frequency distribution between the distances between the eyes and the nose is regenerated, so that erroneous processing can be prevented. .

  In this manner, according to the eye tracking device 3 according to the third embodiment, similarly to the second embodiment, when eye tracking fails, it is possible to satisfactorily return to the eye tracking process. . In addition, when eye tracking fails, it is possible to perform better return to eye tracking processing. Furthermore, the calculation of the feature amount can be made less susceptible to the influence of the light environment, and the eye can be detected correctly even if the driver's posture changes. In addition, the return to the tracking process can be appropriately performed for each driver.

  Furthermore, according to the third embodiment, since a driver's change is detected based on at least one of a vehicle speed signal, a shift position signal, a key switch signal, and a door opening / closing signal, for example, an existing vehicle By using this device, it is possible to detect the driver change at a low cost and with certainty.

  Next, a fourth embodiment of the present invention will be described. The eye tracking device 4 according to the fourth embodiment is the same as that of the third embodiment, but instead of the initialization processing unit 43e inputting a vehicle signal, the driver's change is stored in the image data stored in advance. The method of determining the possibility of is different from that of the third embodiment.

  That is, the initialization processing unit 43e is configured to determine whether or not the difference in the difference image between the face image data acquired by the face image acquisition unit 30 and the stored image data is equal to or less than a set value. . Here, as the stored image data, for example, the face image data acquired by the face image acquisition unit 30 in the previous process may be used.

  An outline is shown in FIG. FIG. 27 is an explanatory diagram showing face image data and difference images in the previous and current processing, where (a) shows an example when there is no driver, and (b) shows that the driver is seated. An example of when.

  First, as shown in FIG. 27A, when the driver leaves the driver's seat, the face image data in the previous and current processing is the same. For this reason, the difference cannot be obtained, and the difference in the difference image is equal to or less than the set value. On the other hand, as shown in FIG. 27B, when the driver is seated in the driver's seat, the face image data in the previous process and the current process are different. For this reason, a difference is obtained and the difference in the difference image is equal to or greater than the set value.

  As described above, the initialization processing unit 43e determines whether or not the driver is seated from the difference image, and detects that the driver is seated when detecting that the driver is seated. . Then, the initialization processing unit 43e initializes the interocular distance frequency distribution and the previous class pc. Accordingly, as in the third embodiment, even if another driver is seated in the driver's seat next time, the frequency distribution between the distances between the eyes and the nose is regenerated, so that erroneous processing can be prevented. .

  In this way, according to the eye tracking device 4 according to the fourth embodiment, as in the third embodiment, when the eye tracking fails, it is possible to satisfactorily return to the eye tracking process. . In addition, when eye tracking fails, it is possible to perform better return to eye tracking processing. Furthermore, the calculation of the feature amount can be made less susceptible to the influence of the light environment, and the eye can be detected correctly even if the driver's posture changes. In addition, the return to the tracking process can be appropriately performed for each driver.

  Furthermore, according to the fourth embodiment, when the difference in the difference image between the acquired face image data and the stored image data is equal to or less than the set value, the possibility of driver change is detected. The driver change can be detected without using a new device.

  As described above, the present invention has been described based on the embodiments. However, the present invention is not limited to the above-described embodiments, and the embodiments may be combined, and changes may be made without departing from the spirit of the present invention. May be added. For example, in the above-described embodiment, the case where the vehicle driver is imaged has been described as an example. However, the present invention is not limited to this, and a passenger seat or a passenger in the rear seat may be imaged as the imaged person. Further, the eye tracking devices 1 to 4 may be mounted on other vehicles without being mounted on the vehicle, and a test device that tracks the eye of the person to be imaged and records the eye movement of the person to be imaged. It may be of the kind.

1 is a hardware configuration diagram of an eye tracking device 1 according to an embodiment of the present invention. It is a data flow diagram which shows the detail of the eye detection apparatus 20 shown in FIG. It is explanatory drawing of a processing state. It is a data flow diagram which shows the detail of the face part detection part shown in FIG. It is a data flow diagram which shows the detail of the ocular nose coordinate determination part shown in FIG. 6 is a data flow diagram showing details of the class determination unit shown in FIG. 5. It is explanatory drawing which shows an example of distance distribution between eye-nose distance. It is explanatory drawing which shows the nose coordinate obtained from a face image and the face image, (a) shows a face image, (b) shows the extraction point extracted along the pixel row | line | column of an image vertical direction and a horizontal direction , (C) show nose candidates. It is a graph which shows the density value (light quantity) of each pixel for every pixel column, and has shown the density value of the pixel column Ya of the image vertical direction shown to Fig.8 (a). It is a graph which shows the density value (light quantity) of each pixel for every pixel row, and has shown the density value of the pixel row Xa of the image horizontal direction. It is explanatory drawing of a face image. It is explanatory drawing which shows the partial area | region of a face image. 12 is a graph showing density values of a pixel row Yb in the image vertical direction shown in FIG. 11. It is explanatory drawing of an eye coordinate and a nose coordinate. It is a flowchart which shows the detailed operation | movement of the positional relationship calculation part shown in FIG. It is explanatory drawing of the interocular distance I, (a) shows the face image, (b) shows each face part and the interocular distance I. It is a flowchart which shows detailed operation | movement of the frequency distribution production | generation part shown in FIG. It is explanatory drawing which shows an example of distance distribution between eye-nose distance. It is a flowchart which shows detailed operation | movement of the labeling part shown in FIG. It is explanatory drawing which shows the image of label data. It is a flowchart which shows the detailed operation | movement of the class determination part shown in FIG. It is a flowchart which shows the detailed operation | movement of the positional relationship determination part shown in FIG. It is a data flow diagram which shows the detail of the ocular nose coordinate determination part shown in FIG. It is a hardware block diagram of the eye tracking apparatus which concerns on 3rd Embodiment. It is a data flow diagram which shows the detail of the eye detection apparatus shown in FIG. It is a data flow diagram which shows the detail of the ocular nose coordinate determination part shown in FIG. It is explanatory drawing which shows the face image data in a last time process and this time process, and a difference image, Comprising: (a) shows the example when there is no driver, (b) shows the example when the driver is seated. Show.

Explanation of symbols

1-4 Eye tracking device 10 Camera 20 Eye detection device 30 Face image acquisition unit (face image acquisition means)
40 ... face part detection unit 41 ... nose coordinate detection unit (coordinate detection means)
42. Eye coordinate detection unit (coordinate detection means)
43 ... Eye-nose coordinate determination unit 43a ... Position relationship calculation unit (position relationship calculation means)
43b ... Frequency distribution generation unit 43c ... Class determination unit 43c1 ... Labeling unit (sorting means)
43c2 ... Class determination section (classification determination means)
43d ... Positional relationship determining unit (Positional relationship determining means)
43e... Initialization processing unit (photographed person change detection means)
50 ... Face part tracking unit 60 ... Eye-nose coordinate signal output unit

Claims (9)

In an eye tracking device that tracks eyes from a face image of a person to be imaged continuously captured by an imaging device,
Face image acquisition means for obtaining a signal obtained by capturing a face image of the person to be imaged as face image data;
Coordinate detection means for detecting coordinates on the face image for both the eyes and the second face part other than the eyes based on the face image data from the face image acquisition means;
A positional relationship calculating means for calculating a feature quantity of coordinates between the eye and the second face part from both coordinates detected by the coordinate detecting means;
Sorting means for sorting a plurality of feature quantities detected in time series by the positional relationship calculating means according to a predetermined condition;
When a face part other than the eye of the person to be imaged is tracked as an eye by mistake, the feature amount calculated by the positional relationship calculation means when the tracking is erroneously performed, and the eye after being tracked by mistake Classification determination means for determining which of the feature quantity categories classified by the classification means belong to both of the feature quantities calculated by the positional relationship calculation means when detecting
A positional relationship determination means for determining that the detected tracking target is an eye when both of the classifications determined by the classification determination means are different;
An eye tracking device comprising: The classification determination unit is configured to calculate a feature amount calculated by the positional relationship calculation unit when the eye cannot be tracked because the eye of the person to be imaged is no longer present in the face image data. And determining whether both of the feature amounts calculated by the positional relationship calculation means when the eye is detected after being unable to be tracked belong to one of the feature amount categories classified by the classification means,
2. The eye tracking according to claim 1, wherein the positional relationship determination unit determines that the detected tracking target is an eye when both of the classifications determined by the category determination unit are the same. apparatus.   The eye tracking device according to claim 1, wherein the coordinate detecting unit detects coordinates of at least one of a nose and a mouth as the second face part.   4. The positional relationship calculation means calculates a distance between the eye and the second face part on the image as a feature amount of coordinates between the eye and the second face part. The eye tracking device according to claim 1.   The apparatus further comprises a person-to-be-photographed person change detecting unit that detects a possibility of the person-to-be-photographed change and initializes information on a plurality of feature amounts detected in time series by the positional relationship calculating unit. The eye tracking device according to any one of claims 1 to 4. The face image acquisition means obtains a signal obtained by capturing a face image of a vehicle driver as face image data,
The image-captured person change detection means detects the possibility of the image-captured person changing based on at least one of a vehicle speed signal, a shift position signal, a key switch signal, and a door opening / closing signal. Item 5. The eye tracking device according to Item 5.   If the difference in the difference image between the face image data acquired by the face image acquisition unit and the stored image data is equal to or less than a set value, the imaging subject change detection unit may change the imaging subject The eye tracking device according to claim 5, wherein the eye tracking device is detected. In an eye tracking device that tracks eyes from a face image of a person to be imaged continuously captured by an imaging device,
Based on the face image data obtained by capturing the face image of the person to be imaged, coordinates on the face image are detected for both the eyes and the second face part other than the eyes, and the eyes and the second face part are detected from these coordinates. And calculate the feature quantity of the coordinates of and collect the detected feature quantity in time series, classify the collected feature quantities according to the predetermined condition,
If the face part other than the eye of the person being imaged was tracked as an eye by mistake, the feature amount when the eye was tracked incorrectly, and the feature amount when the eye was detected after being tracked incorrectly An eye tracking device, characterized by determining to which section both belong, and determining that the detected tracking target is an eye when the sections to which both belong are different. In an eye tracking device that tracks eyes from a face image of a person to be imaged continuously captured by an imaging device,
Based on the face image data obtained by capturing the face image of the person to be imaged, coordinates on the face image are detected for both the eyes and the second face part other than the eyes, and the eyes and the second face part are detected from these coordinates. And calculate the feature quantity of the coordinates of and collect the detected feature quantity in time series, classify the collected feature quantities according to the predetermined condition,
If the subject's eyes no longer exist in the face image data, and the eye cannot be tracked, the feature amount when the eye cannot be tracked and the feature when the eye is detected after the eye cannot be tracked An eye tracking device, characterized by determining to which category both quantities belong, and determining that the detected tracking target is an eye when the categories to which both belong are the same.