JP2020126573A - Image processor, eye detection method, and gazing direction detection method - Google Patents

Abstract

To provide a technique to achieve a reduction of a processing load in detecting a gazing region and improvement of detection accuracy.SOLUTION: In feature point detection processing in S1, an image processor detects a plurality of feature points indicating a position of eyes from an overall imaged image. In eye detection processing in S2, the image processor sets a determination region in the imaged image so as to include the plurality of feature points detected in the feature point detection processing, and determines whether or not the plurality of feature points correctly indicate the position of the eyes using information obtained from the determination region. In the feature point detection processing, the image processor detects the plurality of feature points by giving an initial value of a shape vector representing the position of the plurality of feature points and using a correction amount of the shape vector calculated using the information of the imaged image to repeat processing to update the shape vector.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to a technique for detecting, from a captured image, a direction in which an eye gazes in the captured image.

Patent Document 1 below describes a device that detects a gaze area indicating the direction in which the line of sight is facing from an image in which a face is captured. Specifically, the face position is detected from the image, the face feature point is detected from the detected face position, the pupil position and the face direction are detected based on the detected feature point, and the gaze direction is detected from the pupil position. Then, the gaze area is detected by the geometric calculation of the line-of-sight direction, the face direction, and the camera position.

JP 2012-38106 A

However, as a result of detailed study by the inventor, the conventional technique disclosed in Patent Document 1 is not efficient, and it is difficult to detect the gaze area at high speed and with high accuracy. It was

That is, in the related art, since the gaze area is detected using the results of the face position detection, the face feature point detection, the pupil position detection, the face orientation detection, etc., the detection error in each detection is accumulated. However, the detection accuracy of the gaze area that is finally detected is poor.

Further, in the conventional technique, not only are there many types of detection processing to be executed, but a huge amount of processing is required for the face detection to be executed first, and the processing time also increases. That is, in face detection, a detector learned to respond to a specific pattern in the window is used to find a matching pattern by sequentially scanning by shifting the position and size on the image by the sliding window method. Is being done. In this method, it is necessary to evaluate windows clipped at different sizes and positions many times. In addition, most of the windows to be evaluated each time overlap with the previous one, resulting in poor efficiency, and there is plenty of room for improvement in terms of speed and memory bandwidth. Further, in the sliding window method, if there is variation in the angle of the object to be detected, it is necessary to configure the detector for each range of angles to some extent, and in this respect too, it cannot be said that the efficiency is good.

One aspect of the present disclosure is to provide a technique that reduces the processing load in detecting the gaze area and improves the detection accuracy.

One aspect of the present disclosure is an image processing apparatus, which includes a feature point detection unit (20:S1) and a determination unit (20:S2). The feature point detection unit detects a plurality of feature points representing eye positions from the entire captured image. The determination unit sets the determination region in the captured image so as to include the plurality of feature points detected by the feature point detection unit, and uses the information obtained from the determination region to correctly determine the eye positions of the plurality of feature points. It is determined whether or not it represents. The feature point detection unit uses a vector representing the positions of a plurality of feature points as a shape vector, gives an initial value of the shape vector, and uses the correction amount of the shape vector calculated using the information of the captured image to calculate the shape vector. By repeating the process of updating the, the plurality of feature points that represent the positions of the true eyes in the captured image are detected.

With such a configuration, it is not necessary to perform face detection when detecting the position of the eyes from the captured image, and the shape vector expressing the positions of a plurality of feature points is repeatedly corrected instead of pattern matching. Since a recursive method is used, the processing load can be reduced. As a result, the processing time can be shortened and the memory efficiency can be improved.

Further, in the present disclosure, since the feature point detection unit includes the determination unit that determines whether or not the eye detection is successful, it is possible to suppress the erroneously detected detection result from being output to the subsequent stage, and to improve the reliability of the detection result. It is possible to improve the sex.

One aspect of the present disclosure is an image processing apparatus, which includes a feature point detection unit (20:S1), a determination unit (20:S2), and a direction detection unit (20:S4). The feature point detection unit detects a plurality of feature points representing eye positions from the entire captured image. The determination unit sets the determination region in the captured image so as to include the plurality of feature points detected by the feature point detection unit, and uses the information obtained from the determination region to accurately determine the eye positions of the plurality of feature points. It is determined whether or not it represents. The direction detection unit is set to include the positions of the plurality of feature points detected by the feature point detection unit when the determination unit determines that the plurality of feature points correctly represent the eye position. Using the information of the eye peripheral region, the gaze direction, which is the direction in which the eye gazes in the captured image, is detected. Further, the direction detection unit, using a vector representing the gaze direction as a gaze vector, gives an initial value of the gaze vector, corrects the gaze vector by the correction amount of the gaze vector calculated using the information of the eye peripheral region, and The direction indicated by the selected gaze vector is detected as the gaze direction.

According to such a configuration, since the gaze direction is detected by the recursive method using the information of the eye peripheral region, the conventional method of detecting based on the detection result of the head position, the face direction, the line-of-sight angle, etc. The processing can be speeded up in comparison with. Moreover, the accumulation of errors due to the use of a plurality of detectors in series is suppressed, and the detection accuracy can be improved.

One aspect of the present disclosure is an eye detection method, which includes a feature point detection step (S1) and a determination step (S2). In the characteristic point detection step and the determination step, the same processing as the characteristic point detection unit and the determination unit in the first image processing apparatus described above is performed.

According to the eye detection method of the present disclosure, it is possible to obtain the same effect as that of the first image processing device described above.
One aspect of the present disclosure is a gaze direction detection method, which includes a feature point detection step (S1), a determination step (S2), and a direction detection step (S4). In the feature point detection step, the determination step, and the direction detection step, the same processing as the feature point detection unit, the determination unit, and the direction detection step in the second image processing apparatus described above is performed.

According to the gaze direction detecting method of the present disclosure, it is possible to obtain the same effects as the effects of the second image processing device described above.

It is a block diagram which shows the structure of a gaze detection system. It is a flowchart of a gaze area detection process. It is a figure explaining a regression tree. It is a flowchart of the learning process which produces|generates the parameter for feature points. It is a flowchart of a feature point detection process. It is a figure explaining the shape parameter and correction amount used by a feature point detection process. It is a figure which illustrates a case where a plurality of feature points detected by the feature point detection processing have succeeded in eye detection, and a case where eye detection has failed. It is a flowchart of the learning process which produces|generates a weak discriminator. It is a flow chart of eye judgment processing. It is a figure explaining an eye periphery image. It is a figure showing an example of setting a plurality of gaze fields. It is a flowchart of the learning process which produces|generates the parameter for areas. It is a flowchart of a region detection process.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.
[1. Constitution]
The visual line detection system 1 shown in FIG. 1 is a system including a camera 10 and an image processing device 20. The line-of-sight detection system 1 is installed in a vehicle, for example, and detects the line-of-sight direction of a driver, which is information required for driving support control and the like.

As the camera 10, for example, a known CCD image sensor or CMOS image sensor can be used. The camera 10 is arranged, for example, such that the face of the driver sitting in the driver's seat of the vehicle is included in the imaging range. The camera 10 periodically captures images and outputs captured image data to the image processing device 20.

The image processing apparatus 20 includes a microcomputer having a CPU 21 and a semiconductor memory (hereinafter, memory 22) such as a RAM or a ROM, and executes at least the line-of-sight detection process.

[2. processing]
The line-of-sight detection process executed by the image processing apparatus 20 will be described with reference to the flowchart of FIG. The line-of-sight detection process is activated each time the captured image data is acquired from the camera 10.

In S1, the image processing device 20 executes a feature point detection process. The feature point detection process is a process of detecting, from the acquired captured image, a plurality of feature points necessary for specifying the eye position in the image.

In subsequent S2, the image processing device 20 executes the eye determination process. The eye determination process determines whether or not the plurality of feature points correctly represent the eye position from the feature point peripheral image detected from the captured image based on the plurality of feature points detected in S1, that is, eye detection. Is a process of determining whether or not

In subsequent S3, if the image processing apparatus 20 determines that the eye detection has been successful as a result of the eye determination processing in S2, the image processing apparatus 20 proceeds to S4 and determines that the eye detection has failed. If so, the line-of-sight detection process ends.

In S4, the image processing device 20 executes the area detection process and ends the line-of-sight detection process. The area detection process uses the peripheral image of the eye detected from the captured image based on the plurality of feature points detected in S1, and the detected eye gaze is one of a plurality of preset eye gaze areas. This is a process for detecting whether or not it is suitable for.

Hereinafter, details of the feature point detection processing of S1, the eye determination processing of S2, and the area detection processing of S4 will be described in order.
Note that S1 corresponds to the characteristic point detecting unit and the characteristic point detecting step, S2 corresponds to the determining unit and the determining step, and S4 corresponds to the direction detecting unit and the direction detecting step.

[2-1. Feature point detection processing]
[2-1-1. Overview]
A method of detecting a plurality of feature points from a captured image used in the feature point detection process will be described.

As the characteristic points, points that are effective for specifying the positions where the right eye and the left eye are imaged in the captured image are used. In this embodiment, a total of four points, the outer corners of the eyes and the inner corners of the eyes, are set as the characteristic points. The types and number of feature points are not limited to these.

Here, a column vector listing the coordinates of all feature points is called a shape vector S. The shape vector S is represented by the equation (1), where the coordinates of the p-th feature point are (x _p , y _p ). However, P is the number of feature points, p=1, 2,..., P. T in the equation (1) indicates that it is a transposed matrix.

In the feature point detection process, the initial value S ⁽⁰⁾ of the shape vector S is given, and the estimated position of the feature point is repeatedly corrected by the correction amount calculated using the information obtained from the captured image, so that And a shape vector S that correctly represents the position of the inner corner of the eye is recursively calculated. The shape vector S ^(t) calculated in the t-th regression step is expressed by the equation (2). However, T is the number of times the correction is repeated, and t=1, 2,..., T. Further, the shape vector S ^(T) finally obtained by repeating the regression step T times is expressed by the equation (3). However, r _t is a correction amount used in the t-th regression step, and is represented by a vector having the same dimension as the shape vector S.

In the equation (2), at the t-th regression step, the updated shape vector S ^(t) is obtained by adding the correction amount r _t to the shape vector S ^(t-1) obtained in the previous regression step. It means to be calculated. The formula (3) is obtained by adding all the correction amounts r _{1 to} r _T obtained in each regression step from the first time to the Tth time to the initial value of the shape vector S ⁽⁰⁾ to obtain the final shape vector S. It means becoming ^(T) .

The correction amount r _t is obtained by acting on the correction function f _{K using} the shape vector S ^(t-1) obtained in the previous regression step and the captured image I as input information, as shown in the equation (4). can get. The correction function f _K is a function to which an additive model of a regression function using gradient boosting is applied.
Such a regression function is, for example, “One Millisecond Face Alignment with an Ensemble of Regression Trees” Vahid Kazemi and Josephine Sullivan, The IEEE Conference on CVPR, 2014, 1867-1874 (hereinafter referred to as reference 1), and “Greedy. Function Approximation: A gradient boosting machine”Jerome H. Friedman, The Annals of Statistics Volume 29, Number 5 (2001), 1189-1232 (hereinafter referred to as Reference 2).

The correction function f _K is a function whose value is recursively determined using K regression trees prepared in advance, and is defined by the equation (5). However, f ₀ is an initial value of the correction function f _K , g _k is a regression function whose value is determined by a regression tree identified by k, and k=1, 2,... K. Further, γ is a learning rate and is set to 0<γ<1. By reducing the value of γ, it is possible to suppress over-learning and cope with the diversity of the positions of the feature points.

All K regression trees have the same structure. For example, as shown in FIG. 3, as the regression tree 41, a binary tree that sequentially branches nodes into two is used. A node that is a branch point of the branch of the regression tree 41 is called a normal node 42, and a node that is a leaf of the regression tree 41 is called a terminal node 43. If the node index by which the normal node 42 identifies the normal node 42 is e, the pixel pair (P _e1 , P _e2 ) and the threshold θ _e are associated with the e-th normal node. A regression amount (that is, r _{k1 to} r _{k8 in} FIG. 3) is _associated with each of the terminal nodes 43.

The pixel pair (P _e1 , P _e2 ) is a base point (for example, the position of any one of the feature points on the captured image I set from the shape vector S ^(t-1) obtained in the previous regression step, or The average position of a plurality of characteristic points) is defined by relative coordinates. In each normal node 42, depending on whether the difference value (for example, the brightness difference) between the pixel pairs (P _e1 , P _e2 ) is higher or lower than the threshold θ _e , which branch to the node of the next layer is selected. It is decided whether to do it. In FIG. 3, P _e1 and P _e2 represent the difference value of the pixel. The terminal node 43 is reached by repeating the same processing while tracing the selected branch. The amount of regression associated with the reached terminal node 43 becomes a part of the output value of the regression function g _k .

By the way, the relative coordinates indicating the position of each pixel belonging to the pixel pair (P _e1 , P _e2 ) are expressed as a vector on the standard image (hereinafter, standard vector). The standard image here is an average image obtained by many learning samples.

However, the face in the captured image is not always captured in the same posture as the face in the standard image, and if the standard vector is applied to the captured image I as it is, a shift occurs. Therefore, when the pixel pair (P _e1 , P _e2 ) is used, the similarity matrix (hereinafter referred to as a conversion matrix) Q that reduces the deviation is used for each captured image I to correct the pixel position represented by the standard vector. In addition, the position of the pixel on the captured image I may be determined. The transformation matrix Q is a matrix indicating what kind of rotation, enlargement, or reduction is applied to a point on the reference image to best approximate the corresponding point on the captured image I. It is not essential to use the transformation matrix Q, but the estimation accuracy of the shape vector S can be improved by using the transformation matrix Q.

As described above, in the feature point detection processing, the initial value S ⁽⁰⁾ of the shape vector S, the initial value f ₀ of the regression function f _K , and each parameter that defines the regression tree 41 need to be prepared in advance. There is. Hereinafter, these parameters will be collectively referred to as feature point parameters.

The feature point parameter is generated by the image processing device 20 executing the learning process. However, the learning process does not necessarily have to be executed by the image processing device 20, and may be executed by a device other than the image processing device 20.

[2-1-2. Learning]
The learning process for setting the characteristic point parameters (hereinafter referred to as the first learning process) will be described with reference to the flowchart of FIG. When executing the first learning process, a large number of learning images serving as learning samples are stored in the memory 22. Here, N learning images are prepared, and the coordinates of the feature points in each learning image are detected in advance.

First, in S11, the image processing apparatus 20 acquires the coordinates of the feature points of each learning image, correct values of the shape vector S for each learning image (hereinafter, teacher data) to generate the S ₁ to S _N.
In subsequent S12, the image processing apparatus 20 calculates the average value of the teacher data S _{1 to SN} generated in S11 as the initial value S ⁽⁰⁾ of the shape vector S.

In subsequent S13, the image processing apparatus 20 initializes a regression step t representing the number of times of repeating the processing described below to 1.
In subsequent S14, the image processing device 20 calculates the transformation matrix Q _it of each learning image according to the equation (6).

Expression (6) is obtained by using the coordinates S _ip ^(t-1) of the feature point p in the shape vector S ^(t-1) obtained at the present time of each learning image and the teaching data S _{1 to} S _{N of} each learning image. When the distance from the result of applying the matrix Q to the coordinates S _p of the feature point p in the average shape vector of is summed for each feature point p, the matrix Q that minimizes the total value is the transformation matrix Q _it Means to However, p=1,2,...P and i=1,2,...N.

In subsequent S15, the image processing apparatus 20 calculates a correction residual ΔS _i ^(t) which is a difference between the teacher data Si and the shape vector S ^(t-1) for each learning image according to the equation (7).

In subsequent S16, the image processing apparatus 20 calculates the initial value f ₀ of the correction function f _K used for calculating the correction amount r _t using the equation (8).

The expression (8) is obtained by summing the distances between the correction residual ΔS _i ^(t) in each learning image and an arbitrary vector V having the same dimension as the shape vector S for all the learning images. It means that the vector V that minimizes is the initial value f ₀ of the correction function f _K learned in the processing of S17 to S23.

In subsequent S17, the image processing device 20 initializes the regression tree index k used for identifying the regression tree to 1.
In subsequent S18, the image processing apparatus 20 calculates the remaining correction amount c _k for each learning image by using the equation (9).

The expression (9) means that the difference between the correction residual ΔS _i ^(t) calculated in S15 and the correction amount obtained by the correction function f _k−1 during learning is set as the remaining correction amount c _k .

In subsequent S19, the image processing device 20 selects a pixel pair used for generating the regression tree 41.
In subsequent S20, the image processing apparatus 20 uses the difference value of the pixel pair selected in S19 as an index for classifying the learning images, and performs regression such that a value close to the remaining correction amount c _k is obtained in all the learning images. The tree 41 is generated. That is, the regression function g _k realized by the regression tree 41 is generated. For the selection of such pixel pairs and the generation of the regression tree, for example, the method described in the paragraph 2.3.2 of the above-mentioned reference document 1 may be used.

In subsequent S21, the image processing apparatus 20 corrects the correction function f _k by the equation (10) using the regression function g _k generated in S20.

In subsequent S22, the image processing device 20 increments the regression tree index k by 1.

In subsequent S23, the image processing apparatus 20 determines whether k>K. When a negative determination is made in S23, the process is returned to S18. Thereby, generation of a new regression tree (that is, regression function g _k ) for the remaining correction amount c _k is repeated. On the other hand, when an affirmative determination is made in S23, that is, when K regression functions g _{1 to} g _K are generated, the process proceeds to S24.

In S24, the image processing apparatus 20 uses the correction amount r _t , which is the value of the correction function f _K obtained in the processes of S18 to S23, as shown in Expression (4), and according to Expression (2), Update the vector S.

In subsequent S25, the image processing device 20 increments the regression step t by 1.
In subsequent S26, the image processing apparatus 20 determines whether or not t>T. When a negative determination is made in S26, the process returns to S2. Thus, the generation of the correction function f _K used for calculating the correction amount r _{t associated with} the regression step t is repeated. On the other hand, when an affirmative determination is made in S26, that is, when the generation of the T correction functions f _K used in each of the regression steps t=1 to T is finished, the learning process of the feature point parameter is finished. ..

In the first learning process, one correction function f _K is generated for each T regression steps, and K regression functions g _k are used for one correction function f _K , so that a total of T×K. Regression functions g _k (that is, the regression tree 41) are generated. Further, hereinafter, the regression tree 41 identified by the regression step t and the regression tree index k is represented by a regression tree RT _tk .

[2-1-3. processing]
The feature point detection processing executed by the image processing apparatus 20 in S1 will be described with reference to the flowchart of FIG. In the feature point detection process, the feature point parameter generated by the first learning process is used.

First, in S31, the image processing apparatus 20 acquires the captured image I from the camera 10.
In subsequent S32, the image processing device 20 initializes the regression step t to 1.
In subsequent S33, the image processing apparatus 20, the transformation matrix _{Q t} for captured images I acquired in S31, is calculated using equation (11).

Expression (11) is obtained by using the coordinates S _p ^(t-1) of the feature point p in the shape vector S ^(t-1) currently obtained from the captured image I and the training data S _{1 to} S _{N of} each learning image. When the distance from the result of applying the matrix Q to the coordinates S _p of the feature point p in the average shape vector of is summed for each feature point p, the matrix Q that minimizes the total value is the transformation matrix Q _t. Means to

In subsequent S34, the image processing device 20 initializes the regression tree index k to 1.
In subsequent S35, the image processing apparatus 20 acquires the value of the regression function g _k using the regression tree RT _tk identified by the regression step t and the regression tree index k, and uses the equation (10) to correct the correction function f _k. Update _k . However, in the process using the regression tree RT _tk , the position of the pixel pair associated with each normal node 42 is corrected by the conversion matrix Q _t calculated in S33 and used.

In subsequent S36, the image processing device 20 increments the regression tree index k by 1.
In subsequent S37, the image processing device 20 determines whether or not k>K. That is, it is determined whether all the regression trees RT _{t1 to} RT _{tK associated} with the regression step t have been used in the calculation of the correction function f _k . If a negative determination is made in S37, the process returns to S35, and if an affirmative determination is made in S37, the process proceeds to S38.

In S38, the image processing apparatus 20 sets the value of the correction function f _K finally obtained by the processes of S35 to S37 as the correction amount r _t, and updates the shape vector S using the equation (2). That is, the shape vector S ^(t) is calculated.

In subsequent S39, the image processing device 20 increments the regression step t by 1.
In subsequent S40, the image processing apparatus 20 determines whether or not t>T. If a negative determination is made in S40, the process returns to S33, and if an affirmative determination is made in S40, the process ends.

That is, as shown in FIG. 6, in the feature point detection processing, the correction amount r _t is calculated for each regression step, and the updated shape vector S ^(t) is set with the initial value S ⁽⁰⁾ of the shape vector as a starting point. By repeating the generation, the finally calculated shape vector S ^(T) becomes the estimation result of the positions of the four feature points.

[2-2. Eye judgment processing]
[2-2-1. Overview]
The eye determination process determines whether or not the shape vector S representing the positions of the outer corners of the eyes and the inner corners of the eyes has been correctly estimated by the feature point detection process, that is, whether the eye has been successfully detected.

This suppresses the output of an incorrect gaze area in the subsequent processing when the eye detection fails. As an example in which the eye detection fails, as shown in FIG. 7, when a person is not reflected in the first place, a wrong point may be detected although a person is reflected. In addition to this, there is a case where some or all of the eyes are blocked due to some reason and cannot be seen.

In the eye determination process, the center coordinates S _L and S _R of each eye are calculated for each of the right eye and the left eye from the shape vector S obtained by the feature point detection process, and the center coordinates S _L and S _R are set as the centers. A rectangular image area having a width Dw and a height Dh is set as the determination area I _G. The determination area I _G may be normalized so as to absorb the variation in the face size on the captured image I. Specifically, the determination area I _G may be enlarged or reduced so that the width of the inner corner of the eye, the width of the inner corner of the eye, and the width between the eyes calculated from the shape vector S have the same size.

Furthermore, in the eye determination process, it is determined whether or not the eye detection is successful using a large number of weak classifiers that receive information obtained from the determination area I _G as an input.
Let M be the number of weak classifiers, m=1, 2,..., M, and the m-th weak classifier h _m (I _G ) is expressed by equation (12). P _m1 and P _m2 are the brightness of the pixel pair selected from the determination area I _G , and the position of the pixel pair is relative to the position indicated by either of the two center coordinates S _L and S _R , or the average position of both. Defined in relative position. θ _m is a threshold value, and α _mH and α _mL are weights.

That is, the weak discriminator h _m (I _G ) outputs the weight α _mH or α _mL as the determination result depending on whether the difference value P _m1 −P _m2 of the pixel pair is larger than the threshold θ _m .

Then, according to the output of the strong discriminator H(I _G ), which is obtained by combining M weak discriminators h _m (I _G ), when H(I _G )>0, success, H(I _G ) ≤ 0, it is determined to be a failure. It should be noted that λ is a threshold for the output of the strong discriminator H(I _G ), and is arbitrarily set in consideration of the determination accuracy and the like.

As described above, in the eye determination process, it is necessary to prepare M weak classifiers h _m (I _G ) in advance.

The weak classifier h _m (I _G ) is generated by the image processing device 20 performing the learning process. For example, Real Adaboost, which is ensemble learning, may be used for learning. However, the learning process does not necessarily have to be executed by the image processing device 20, and may be executed by a device other than the image processing device 20.

[2-2-2. Learning]
Learning processing for generating the weak discriminator h _m (I _G ) (hereinafter, second learning processing) will be described with reference to the flowchart in FIG. 8. When executing the second learning process, a large number of learning images serving as learning samples are stored in the memory 22. The learning image includes, in addition to the correct answer image in which the eyes are reflected, an incorrect answer image in which the eyes are hidden or the person is not reflected in the first place. In addition, a result of performing the feature detection process of S1 on these learning images and a result of previously determining whether or not the detection result is successful in eye detection are also prepared.

In S51, the image processing device 20 initializes the weight of each learning image. The initial value of the weight may be set uniformly for all learning images, for example.
In subsequent S52, the image processing apparatus 20 initializes a discriminator index m for identifying a weak discriminator to 1. At the same time, the maximum number of weak classifiers generated by this processing is set to M.

In subsequent S53, the image processing device 20 generates a weak discriminator h _m (I _G ). Specifically, for various pixel pairs, difference values of pixel pairs are calculated for all learning images. Then, a pixel pair P _m1 and P _m2 and a threshold value θ _m that can best separate an image that succeeds in eye detection (hereinafter, a success image) and a failure image (hereinafter, a failure image) are selected. Further, the weights α _mH and α _mL change their values according to the accuracy of separation. For setting the weights α _mH and α _mL and updating the weights of each learning image, for example, the method shown in Schapire, RE and Singer, Y.: Improved Boosting Algorithms Using Confidencerated Predictions, Machine Learning, etc. can be used. Good.

In subsequent S54, the image processing apparatus 20 until the by the m generated weak classifier _{h 1 (I G) ~h m} (I G) strongly bound the identifier H _{(I G),} all of this It is determined whether the learning image has been correctly separated into a success image and a failure image. If an affirmative decision is made in S54, the process proceeds to S58, and if a negative decision is made in S54, the process proceeds to S55.

In S58, the image processing device 20 sets the value of the discriminator index m to the number M of generated weak discriminators, and ends the process.
In subsequent S55, the image processing device 20 increments the discriminator index m by 1.

In subsequent S56, the image processing device 20 determines whether or not m>M is exceeded. If a positive determination is made in S56, the process ends. In this case, M set in S51 is the number of generated classifiers. On the other hand, if a negative determination is made in S56, the process proceeds to S57.

In S57, the image processing device 20 updates the weight of the learning image so that the weight of the learning image that has failed to be separated in S54 increases, and returns the process to S53. That is, the weights of the learning images are adjusted so that a weak classifier that can accurately separate the images that have failed to be generated is generated.

[2-2-3. processing]
The eye determination processing executed by the image processing apparatus 20 in S2 will be described with reference to the flowchart of FIG. In the eye determination process, the M weak classifiers h _m (I _G ) generated by the second learning process are used.

In S61, the image processing device 20 initializes the discriminator index m to 1, and also initializes the integrated value J of the determination result of the weak discriminator h _m (I _G ) to 0.
In subsequent S62, the image processing device 20 causes the weak discriminator h _m (I _G ) to act on the pixel pair extracted from the determination region I _G set based on the shape vector S, in (14). Integrate as shown in the formula. This calculation is equivalent to calculating the first term on the right side of Expression (13).

In subsequent S63, the image processing device 20 increments the discriminator index m by 1.

In subsequent S64, the image processing device 20 determines whether or not m>M. If an affirmative decision is made in S64, it is considered that the decision by the M weak discriminators has been completed, and the process proceeds to S65. On the other hand, when a negative determination is made in S64, it is determined that there is a weak discriminator for which the determination has not been performed, and the process returns to S62.

In S65, the image processing apparatus 20, the judgment value H by (15) as shown in equation strong classifier H (I _G) by subtracting the weight threshold λ from the integrated value J of the determination result by the weak classifier calculate. Note that the equations (14) and (15) realize the equation (13) shown above.

In subsequent S66, the image processing device 20 determines whether or not the determination value H is greater than 0.

When an affirmative decision is made in S67, the image processing apparatus 20 advances the process to S67, decides that the eye detection has succeeded, and ends the process.
When a negative determination is made in S67, the image processing device 20 advances the processing to S68, determines that the eye detection has failed, and ends the processing. The result of the determination as to whether or not the eye detection has succeeded is notified to the subsequent process using, for example, a flag.

[2-3. Area determination processing]
[2-3-1. Overview]
A method of detecting a gaze area, which is an area gazed by the eye, from the image information of the area around the eye, which is used in the area determination process, will be described.

The eye peripheral area I _E is set based on the shape vector S (that is, a plurality of feature points) determined to have succeeded in eye detection. Specifically, as shown in FIG. 10, as the eye peripheral area _IE , a rectangular area of a predetermined size including an area from the outer corner of the eye to the inner corner of the eye is used for each of the right eye and the left eye.

Note that a region similar to the determination region I _G used for eye determination may be used as the eye peripheral region.
As the gaze area, for example, as shown in FIG. 11, five gaze areas of a front area a ₁ , an upper area a ₂ , a lower area a ₃ , a right area a ₄ , and a left area a ₅ are set. Further, the closed eyes state may be set as an independent category. The area shape and the number of areas of the gaze area are not limited to this.

Here, a vector having the same number of elements as the number D of gaze areas is called a score vector A. The score vector A is represented by Expression (16), and each element is associated with any of the gaze areas a _{1 to} a ₅ . This gaze area score vector A takes a one-hot format in which the value of the element corresponding to the corresponding gaze area is large and the value of the element corresponding to the other gaze area is small, and finally the largest value is set. The gaze area corresponding to the element is set as the detection result. The score vector A corresponds to the gaze vector.

In the region determination process, as shown in Expression (17), an initial value A ⁽⁰⁾ of the score vector A is given, and the score vector A is calculated by the correction amount R calculated using the information obtained from the peripheral image of the eye. The correction is performed to calculate the score vector A that represents the area where the line of sight is gazing.

The correction amount R is obtained by acting on the correction function F _{K using} the initial value A ⁽⁰⁾ of the score vector and the eye peripheral region I _E as input information, as shown in Expression (18). The correction function F _K is a function to which an additive model of a regression function is applied, like the one used in the feature amount detection processing.

The correction function F _K is a function whose value is recursively determined using K regression trees prepared in advance, and is defined by the equation (19). F ₀ is an initial value of the correction function F _K , G _k is a regression function whose value is determined by a regression tree identified by k, and k=1, 2,... K. Further, γ is a learning rate and is set to 0<γ<1. By reducing the value of γ, overlearning is suppressed. The set values of K and γ do not have to be the same as those used for eye detection.

The structure of the regression tree is the same as that of the regression tree used in the feature point detection process, and thus the description thereof is omitted.

In the area detection processing, the initial value A ⁽⁰⁾ of the score vector is corrected by the correction amount R calculated using the correction function F _K , and the gaze corresponding to the element having the maximum score in the corrected score vector A The area is output as the result of detecting the gaze area. If the maximum score is equal to or lower than a preset threshold value, the output reliability may be low and the output may be indefinite.

In this way area detecting process is prepared as an initial value A of the score vector A ^(0), the initial value F ₀ of the correction function F _K, the parameters and pre defining the regression tree used for generating the correction function F _K Must have been Hereinafter, these parameters will be collectively referred to as area parameters.

The region parameter is generated by the image processing device 20 executing the learning process. However, the learning process does not necessarily have to be executed by the image processing device 20, and may be executed by a device other than the image processing device 20.

[2-3-2. Learning]
The learning process for setting the region parameters (hereinafter, the third learning process) will be described with reference to the flowchart in FIG. When the third learning process is executed, a large number of learning images serving as learning samples are stored in the memory 22. Here, it is assumed that N learning images are used and the result of manually detecting the eye gaze area in each learning image is prepared in advance.

First, in S71, the image processing device 20 acquires the detection result of the gaze area in each learning image, and generates correct values (hereinafter, teacher data) A _{1 to} A _N of the score vector A for each learning image.

In subsequent S72, the image processing apparatus 20 calculates the average value of the teacher data A _{1 to} A _N generated in S71 as the initial value A ⁽⁰⁾ of the score vector.
In subsequent S73, the image processing apparatus 20 calculates a correction residual ΔA _i which is a difference between the teacher data A _i and the initial value A ⁽⁰⁾ of the score vector for each learning image according to the equation (20).

In subsequent S74, the image processing apparatus 20 calculates the initial value f ₀ of the correction function F _K used to calculate the correction amount R using the equation (21).

Equation (21) minimizes the total value when the distance between the modified residual ΔAi in each learning image and an arbitrary vector V having the same dimension as the score vector A is summed for all learning images. This means that the vector V is set to the initial value F ₀ of the correction function F _K.

In subsequent S75, the image processing device 20 initializes the regression tree index k used for identifying the regression tree to 1.
In subsequent S76, the image processing apparatus 20 calculates the remaining correction amount c _k for each learning image by using the expression (22).

In subsequent S77, the image processing device 20 selects a pixel pair used for generating a regression tree. The position of a pixel is represented by a relative position from a base point determined by a shape vector representing an eye position (that is, a plurality of feature points). Moreover, a point on the captured image corresponding to at least one of the outer corner of the eye and the inner corner of the eye is used as the base point.

In subsequent S78, the image processing apparatus 20 uses the difference value of the pixel pair selected in S77 as an index when classifying the learning images, and performs regression such that a value close to the remaining correction amount c _k is obtained in all the learning images. Generate a tree. That is, the regression function G _k realized by the regression tree is generated.

In subsequent S79, the image processing apparatus 20 updates the correction function F _k by the equation (23) using the regression function G _k generated in S78.

In subsequent S80, the image processing device 20 increments the regression tree index k by 1.

In subsequent S81, the image processing apparatus 20 determines whether k>K. If a negative determination is made in S80, the process returns to S76 to create a new regression tree, and if an affirmative determination is made in S80, the process ends.

The processing of S73 to S81 is the same as the processing of S15 to S23 described above.
In the third learning process, K regression functions G _k (that is, a regression tree) are generated for one correction function F _K.

[2-3-3. processing]
The area detection processing executed by the image processing apparatus 20 in S4 will be described with reference to the flowchart of FIG. In the area detection processing, the area parameters generated by the learning processing described above are used.

First, in S91, the image processing apparatus 20 acquires an image of the eye peripheral area I _E set based on the shape vector S ^(T) estimated in S1.
In subsequent S92, the image processing device 20 initializes the regression tree index k to 1.

In subsequent S93, the image processing device 20 acquires the value of the regression function G _k using the regression tree RT _k identified by the regression tree index k, and updates the correction function F _k using the equation (23).
In subsequent S94, the image processing device 20 increments the regression tree index k by 1.

In subsequent S95, the image processing apparatus 20 determines whether k>K. That is, it is determined whether all the regression trees RT _{1 to} RT _K have been used in the calculation of the correction function F _k . If the negative determination is made in S95, the process returns to S93, and if the positive determination is made in S95, the process proceeds to S96. By the processes of S93 to S95, the correction function F _k shown in the equation (19) is finally calculated, and as shown in the equation (18), the value obtained by the correction function f _k is the initial value of the score vector. It is the correction amount R for the value A ⁽⁰⁾ .

In S96, the image processing device 20 generates the score vector A modified using the equation (17).
In subsequent S97, the image processing apparatus 20 extracts the element having the maximum score SC from the corrected score vector A.

In subsequent S98, the image processing apparatus 20 determines whether SC>TH. TH is a threshold for determining the reliability of the determination result. If the affirmative determination is made in S98, the process proceeds to S99, and if the negative determination is made in S98, the process proceeds to S100.

In S99, the image processing apparatus 20 outputs the gaze area associated with the element having the maximum score SC as the detection result, and ends the processing.
In step S100, the image processing apparatus 20 determines that the detection result of the gaze area has low reliability, invalidates the detection result, and ends the processing.

[3. effect]
According to the embodiment described in detail above, the following effects are achieved.
(3a) In the present embodiment, when detecting the position of the eyes from the captured image, a recursive method of repeatedly correcting the shape vector S expressing the positions of a plurality of feature points is used. Thereby, a plurality of feature points are directly and simultaneously detected from the entire captured image I. Therefore, unlike the related art, it is not necessary to perform face detection and pattern matching is not used, so that the feature points can be efficiently detected. As a result, the processing time required to detect the eye position can be shortened and the memory efficiency in the processing can be improved.

(3b) In the present embodiment, the eye determination process determines whether or not the plurality of feature points detected by the feature point detection process correctly represent the eye position. Therefore, it is possible to prevent the erroneously detected detection result from being output to the region determination process, and thus to prevent the gaze region from being erroneously detected.

(3c) In the present embodiment, since the gaze area is estimated using the texture information of the eye peripheral area I _E , the conventional method of estimating the gaze area using the detection results of the head position, the face direction, the gaze angle, and the like By comparison, the processing time required to detect the gaze area can be shortened. Moreover, since the accumulation of errors due to the use of a plurality of detectors in series is also suppressed, it is possible to improve the detection accuracy of the gaze area and thus the detection accuracy of the gaze direction.

(3d) In the present embodiment, a regression tree is used to calculate the correction amount r of the shape vector S in the feature amount detection process and the correction amount R of the score vector A in the region detection process, and a parameter used for conditional branching of the regression tree. Is used as the difference value of the pixel pair. The difference value of the pixel pair is also used as the input parameter of the weak classifier used in the eye determination process. In other words, since the recursive processing uses simple parameters, the processing can be lightened. Moreover, since gradient boosting is used in both the feature point detection processing and the area detection processing, high speed operation can be realized.

(3e) In the present embodiment, as the position of the pixel pair, the position corrected by applying the conversion matrix Q to the standard vector is used. As a result, variations in the shape of the feature points due to rotation, enlargement, reduction, etc., which are different for each captured image, are absorbed, so that the detection accuracy of the process using the pixel pair can be improved.

[4. Other Embodiments]
Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and various modifications can be implemented.

(4a) In the above embodiment, the luminance difference, which is the difference value of the pixel pair, is used as the input information to the regression tree or the weak discriminator, but the present disclosure is not limited to this. For example, an absolute value of brightness or an average value of brightness within a certain range may be used.

(4b) In the above embodiment, the method of acquiring the regression function g _k using the regression tree is illustrated, but the regression tree may not be used as long as the method uses the regression function.
(4c) In the above embodiment, the configuration of correcting the position of the pixel pair that is an input to the regression tree by using the conversion matrix Q has been illustrated, but the position of the pixel pair does not need to be corrected by the conversion matrix Q. , May be used as it is.

(4d) In the above embodiment, an example in which a score vector in which each element of the vector is associated with a gaze area is used as the gaze vector has been shown, but the present disclosure is not limited to this. For example, a vector in which each element represents the coordinates in the three-dimensional space, that is, a vector in which the direction indicated by the vector indicates the gaze direction may be used as the gaze vector.

(4e) The image processing apparatus 20 and the method thereof according to the present disclosure are provided by configuring a processor and a memory programmed to execute one or a plurality of functions embodied by a computer program. It may be realized by a dedicated computer. Alternatively, the image processing apparatus 20 and the method thereof described in the present disclosure may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the image processing apparatus 20 and the method thereof according to the present disclosure include a processor and a memory programmed to execute one or a plurality of functions, and a processor configured by one or more hardware logic circuits. It may be realized by one or more dedicated computers configured by combination. Further, the computer program may be stored in a computer-readable non-transition tangible recording medium as an instruction executed by a computer. The method for realizing the function of each unit included in the image processing apparatus 20 does not necessarily include software, and all the functions may be realized by using one or a plurality of hardware.

(4f) A plurality of functions of one constituent element in the above embodiment may be realized by a plurality of constituent elements, or one function of one constituent element may be realized by a plurality of constituent elements. .. Further, a plurality of functions of a plurality of constituent elements may be realized by one constituent element, or one function realized by a plurality of constituent elements may be realized by one constituent element. Moreover, you may omit a part of structure of the said embodiment. Further, at least a part of the configuration of the above-described embodiment may be added or replaced with respect to the configuration of the other above-described embodiment.

(4g) In addition to the image processing device 20 described above, a system having the image processing device 20 as a constituent element, a program for causing a computer to function as the image processing device 20, and a non-transitional physical substance such as a semiconductor memory storing the program. The present disclosure can be implemented in various forms such as a recording medium, an eye detection method, and a gaze area detection method.

1... Line-of-sight detection system, 10... Camera, 20... Image processing device, 21... CPU, 22... Memory, 41... Regression tree, 42... Normal node, 43... End node.

Claims (23)

A feature point detection unit (20:S1) configured to detect a plurality of feature points representing eye positions from the entire captured image;
The determination region in the captured image is set so as to include the plurality of feature points detected by the feature point detection unit, and information obtained from the determination region is used to determine the eye positions of the plurality of feature points. A determination unit (20:S2) configured to determine whether or not the representation is correct,
Equipped with
The feature point detection unit uses a vector representing the positions of the plurality of feature points as a shape vector, gives an initial value of the shape vector, and calculates a correction amount of the shape vector calculated using information of the captured image. By using, by repeating the process of updating the shape vector, to detect the plurality of feature points representing the position of the true eye in the captured image,
Image processing device. The image processing apparatus according to claim 1, wherein
The feature point detection unit calculates a correction amount of the shape vector by using a regression function representing a relationship between the feature amount calculated from the captured image and the correction amount of the shape vector,
Image processing device. The image processing apparatus according to claim 2, wherein
The image processing device in which the regression function used in the feature point detection unit is a regression tree learned by gradient boosting and is realized as a set of weak hypotheses having a tree structure. The image processing apparatus according to claim 2 or 3, wherein
The feature point detection unit uses, as the feature amount, a difference value between two pixels selected from the captured image,
Image processing device. The image processing apparatus according to claim 4, wherein
The image processing apparatus, wherein the position of the pixel used to calculate the feature amount is represented by a relative position from a base point determined by the shape vector. The image processing apparatus according to claim 5, wherein
The feature point detection unit uses a transformation matrix that reduces a positional deviation between an average position of the plurality of feature points in the plurality of captured images and a position of the plurality of feature points in each of the captured images, and the conversion matrix An image processing apparatus for selecting the two pixels by using the relative position converted by. The image processing apparatus according to any one of claims 1 to 6,
The image processing apparatus, wherein the feature point detection unit uses, as the feature point, a point on the captured image corresponding to at least one of the outer corner of the eye and the inner corner of the eye. The image processing device according to any one of claims 1 to 7,
When the determination unit determines that the plurality of feature points correctly represent the eye position, the eye set including the positions of the plurality of feature points detected by the feature point detection unit. A direction detection unit (20:S4) for detecting a gaze direction, which is a direction in which the eye gazes in the captured image, using information on the peripheral region,
Image processing device. The image processing apparatus according to claim 8, wherein
The direction detection unit, by using a vector representing the gaze direction as a gaze vector, gives an initial value of the gaze vector, and corrects the gaze vector by a correction amount of the gaze vector calculated using information of the eye peripheral region. The image processing apparatus detects the direction indicated by the corrected gaze vector as the gaze direction. A feature point detection unit (20:S1) configured to detect a plurality of feature points representing eye positions from the entire captured image;
The determination region in the captured image is set so as to include the plurality of feature points detected by the feature point detection unit, and information obtained from the determination region is used to determine the eye positions of the plurality of feature points. A determination unit (20:S2) configured to determine whether or not the representation is correct,
When the determination unit determines that the plurality of feature points correctly represent the eye position, the eye set including the positions of the plurality of feature points detected by the feature point detection unit. A direction detection unit (20: S4) for detecting a gaze direction, which is a direction in which the eye gazes in the captured image, using information on the peripheral region,
The direction detection unit, by using a vector representing the gaze direction as a gaze vector, gives an initial value of the gaze vector, and corrects the gaze vector by a correction amount of the gaze vector calculated using information of the eye peripheral region. The image processing apparatus detects the direction indicated by the corrected gaze vector as the gaze direction. The image processing device according to claim 9 or 10, wherein
The direction detecting unit calculates a correction amount of the gaze vector by using a regression function representing a relationship between a feature amount calculated from the captured image and a correction amount for the score vector,
Image processing device. The image processing apparatus according to claim 11, wherein
The image processing device in which the regression function used in the direction detection unit is a regression tree learned by gradient boosting and is realized as a set of weak hypotheses having a tree structure. The image processing device according to any one of claims 9 to 12,
The direction detection unit uses a difference value of two pixels selected from the captured image as the feature amount,
Image processing device. The image processing apparatus according to claim 13, wherein
The image processing device in which the position of the pixel used in the calculation of the feature amount by the direction detection unit is represented by a relative position from a base point determined by the plurality of feature points detected by the feature point detection unit. .. The image processing apparatus according to claim 14, wherein
The said direction detection part is an image processing apparatus which uses the point on the said captured image corresponding to at least one of the outer corner of the eye and the inner corner of the eye as said base point. The image processing device according to any one of claims 9 to 15,
The direction detection unit is a vector having a plurality of elements that are associated with any of a plurality of preset gaze areas as the gaze vector, and corresponds to the gaze area located in the gaze direction. The gaze associated with the element having the largest value in the modified gaze vector using a one-hot type vector in which the assigned element has a large value and the other elements have a small value An image processing apparatus in which the direction in which a region is located is the gaze direction. The image processing apparatus according to claim 16, wherein:
The said direction detection part is an image processing apparatus which makes the said gaze direction the direction where the said gaze area matched with the element which has the largest value in the said corrected gaze vector is located. The image processing apparatus according to any one of claims 1 to 17,
The determination unit obtains a determination result using a strong discriminator that is a combination of a plurality of weak discriminators.
Image processing device. The image processing apparatus according to claim 18,
The determination unit uses a difference value between two pixels selected from the determination region as a parameter input to the weak discriminator,
Image processing device. The image processing device according to claim 19, wherein
The determination unit uses the determination region normalized so that the size of eyes estimated from the plurality of feature points detected by the feature point detection unit is constant,
Image processing device. The image processing apparatus according to any one of claims 1 to 20,
An image processing apparatus in which a face image of a driver is used as the captured image. An eye detection method for detecting an eye position from a captured image by image processing,
A feature point detecting step (S1) of detecting a plurality of feature points representing eye positions from the entire captured image;
The determination area in the captured image is set so as to include the plurality of feature points detected in the feature point detection step, and the information obtained from the determination area is used to determine the eye positions of the plurality of feature points. A determination step (S2) for determining whether or not it is represented correctly,
Equipped with
In the feature point detecting step, a vector expressing the positions of the plurality of feature points is used as a shape vector, an initial value of the shape vector is given, and a correction amount of the shape vector calculated using information of the captured image is calculated. By using, by repeating the process of updating the shape vector, to detect the plurality of feature points representing the position of the true eye in the captured image,
Eye detection method. A gaze direction detecting method for detecting a gaze direction, which is a direction in which an eye gazes from a captured image by image processing,
A feature point detecting step (S1) of detecting a plurality of feature points representing eye positions from the entire captured image;
The determination region in the captured image is set to include the plurality of feature points detected in the feature point detection step, and the information obtained from the determination region is used to determine the eye positions of the plurality of feature points. A determination step (S2) of determining whether or not the detection is correctly performed,
When it is determined in the determination step that the plurality of feature points correctly detect the eye positions, the positions of the plurality of feature points detected in the feature point detection step are set to be included. A direction detection step (S4) of detecting a gaze direction of the eye in the captured image using information about the eye peripheral region;
Equipped with
In the direction detecting step, a vector representing the gaze direction is used as a gaze vector, an initial value of the gaze vector is given, and the gaze vector is corrected by a correction amount of the gaze vector calculated using information of the eye peripheral region. Then, the gaze direction detecting method for detecting the direction indicated by the corrected gaze vector as the gaze direction.