JP7259648B2 - Face orientation estimation device and method - Google Patents

Description

The present disclosure relates to a face orientation estimation device and method for stably and accurately estimating the positions of facial organ points and the face orientation angle of a person's face in image data.

2. Description of the Related Art Various techniques are known for detecting a human face from image data and estimating the positions of facial organ points and the facial orientation angle of the detected face. For example, in Patent Document 1, a three-dimensional face model that defines a plurality of three-dimensional positions corresponding to a plurality of feature points including facial feature points on a human face is fitted to a face in an image to determine facial feature points. A three-dimensional face model fitting algorithm that estimates the position of points and angles of face orientation in an image, and a detection device that utilizes the same, are disclosed.

However, according to the conventional 3D face model fitting algorithm, the position estimation of facial feature points may not be stable over time. This may affect the accuracy of eye open/close detection, which is one of the processes based on position estimation of facial feature points.

Furthermore, according to the conventional technology, when the estimated organ point positions of the entire face are drawn to the side of the face such as the ears due to an action such as scratching the face with a hand, or when the estimated organ point positions of the eyes are pulled by the frame such as glasses. An event can also occur where the estimated position stabilizes at the wrong position if the is pulled. As a result, the three-dimensional face model is stabilized without being tightly fitted to the face image, and the face orientation estimation result continues to deviate.

The eye openness identification device disclosed in Patent Document 2 performs a tracking search for each facial organ point using a method such as template matching for a predetermined range centered on the position of each facial organ point in the previous frame. Further, the facial part search device disclosed in Patent Document 3 regards the position of each facial feature point in the previous frame as a rough position, and searches for detailed positions by tracking each facial feature point from that position. . However, even with the eye openness degree identifying device disclosed in Patent Document 2 and the facial part searching device disclosed in Patent Document 3, an action such as scratching the face with a hand causes the estimated organ point positions of the entire face to shift to the ear or the like. When the eye is pulled to the side of the face, or when the eye organ point estimation position is drawn in by the frame of glasses or the like, the estimated position may be stabilized at an incorrect position.

Japanese Patent Application Laid-Open No. 2007-249280 JP 2010-198313 A Japanese Patent Application Laid-Open No. 2003-281539

The present disclosure is an algorithm for accurately estimating the positions of facial organ points and the angle of the face orientation with respect to a human face in image data, which is temporally stable and does not stabilize at an erroneous estimated position. and a face orientation estimation apparatus and method using the same algorithm.

The face orientation estimation device of the present disclosure includes:
a face detection unit that detects human face image data from image data; and
The face orientation estimation device includes a face orientation estimation unit that estimates positions of facial organ points and angles of the face orientation with respect to detected face image data of a person.
When the face images detected by the face detection unit are continuous in a plurality of frames, the face orientation estimation unit performs the following for each frame:
(1) an estimated reference value calculation process for obtaining an estimated reference value from a global search fitting process;
(2) based on the three-dimensional face model that is the fitting result of the previous frame, a tracking process that performs a local search fitting process;
The (2) tracking process includes a correction process,
The correction process is
Calculate the difference between the face orientation angle of the three-dimensional face model that is the result of the fitting of the previous frame and the face orientation angle of the three-dimensional face model of the estimated reference value calculated by the estimated reference value calculation process (1). , if the difference is equal to or greater than a predetermined threshold, the 3D face model with the latest estimated reference value is used as the basis of the local search fitting process instead of the 3D face model that is the fitting result of the previous frame. Dimensional face model and
The facial orientation estimating unit outputs the position of the facial organ point and the facial orientation angle for each frame based on the result of the local search fitting process.

The apparatus and method for estimating face orientation according to the present disclosure are capable of stabilizing a person's face in image data over time without stabilizing an erroneous estimated position. Orientation angles can be estimated.

2 is a block diagram showing the functional configuration of the face orientation estimation device according to Embodiment 1; FIG. FIG. 10 is a diagram for explaining an application example of the face orientation estimation device according to the present disclosure; 4 is a flowchart showing the overall operation of the face orientation estimation device according to Embodiment 1; 5 is a flow chart showing a process of estimating positions of facial feature points and angles of face orientation by a face orientation estimation unit in the face orientation estimation device according to Embodiment 1; 5(a-1), (a-2), (a-3), and (a-4) show position estimation for eyes because position estimation of facial feature points in face image data is not stable over time. FIG. 10 is a diagram schematically showing how an actual target changes from eye (a-1)→eyebrow (a-2)→eye (a-3)→eyebrow (a-4). FIGS. 5(b-1)(b-2)(b-3)(b-4) are actual objects of position estimation for eyes like FIGS. 5(a-1) to (a-4). This is a diagram schematically showing how the eye open/closed detection process produces an erroneous output of open→closed→open→closed, even though the eyes are actually kept open. is. FIG. 4 is a diagram illustrating an overview of an algorithm for estimating the positions of facial feature points and the angle of the face orientation, according to the present disclosure;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments according to the present invention will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configurations may be omitted. This is to avoid unnecessary verbosity in the following description and to facilitate understanding by those skilled in the art.

It is noted that the inventors provide the accompanying drawings and the following description in order for those skilled in the art to fully understand the present disclosure, which are intended to limit the claimed subject matter. isn't it.

[Background to this disclosure]
2. Description of the Related Art Various techniques are known for detecting a human face from image data and estimating the positions of facial organ points and the facial orientation angle of the detected face. Japanese Patent Application Laid-Open No. 2002-200000 discloses that a three-dimensional face model that defines a plurality of three-dimensional positions corresponding to a plurality of feature points including facial feature points on a human face is fitted to a face in an image to determine facial feature points. A three-dimensional face model fitting algorithm that estimates the position and angle of the face orientation in an image, and a detection device that utilizes the same, are disclosed.

However, if the position estimation of facial feature points is not stable over time, it may affect the accuracy of eye open/close detection, which is one of the processes based on facial feature point positions. 5(a-1), (a-2), (a-3), and (a-4) show position estimation for eyes because position estimation of facial feature points in face image data is not stable over time. FIG. 10 is a diagram schematically showing how an actual target changes from eye (a-1)→eyebrow (a-2)→eye (a-3)→eyebrow (a-4). On the other hand, FIGS. 5(b-1), (b-2), (b-3), and (b-4) show the positions for the eyes as in FIGS. 5(a-1) to (a-4). Because the actual target of estimation fluctuates, the eye open/closed detection process incorrectly outputs open → closed → open → closed even though the eyes are actually open. It is a figure shown typically. In this way, position estimation of facial feature points that is not stable over time can affect the accuracy of eye open/close detection.

Therefore, by utilizing the fact that movements of facial organ point positions for each frame are generally minute in moving images with a frame rate of 15 frames/second or 30 frames/second, the positions of facial organ points are detected by tracking processing. Estimates can be considered. In other words, it is conceivable to search for facial feature points (ie, fitting process) around the facial feature point position estimation result (ie, three-dimensional face model fitting result) in the previous frame as a starting point.

However, by combining the tracking process, the facial organ point position estimation result and the eye opening/closing detection in the subsequent process are temporally stable. When the eye is drawn to the side of the face, or when the estimated position of the organ point of the eye is pulled by the frame of eyeglasses or the like, an event may occur in which the estimated position is stabilized at an incorrect position. As a result, the three-dimensional face model is stabilized without being tightly fitted to the face image, and the face orientation estimation result continues to deviate.

The present disclosure is a technique devised by the inventor to solve such problems. FIG. 6 is a diagram illustrating an outline of an algorithm for estimating the position of facial features and the angle of the face orientation according to the present disclosure. As shown in FIG. 6, three-dimensional face model fitting tracking processing (local search fitting processing) ((A1)-(A2), (B1)-(B2)) and global search fitting processing ((C1)-( C2)) are performed in parallel.

An estimated reference value is calculated from the global search fitting process ((C1)-(C2)) for each frame.

In the tracking process, the 3D face model, which is the result of the previous frame fitting, is arranged on the face image. Subsequently, the face orientation angle and the face orientation estimation reference value (estimated face orientation angle at the reference value) and the difference is calculated ((A1), (B1)). For example, the yaw angle is used to calculate the difference Diff. Here, if the difference is smaller than the predetermined threshold (A1), local search fitting processing is performed based on the arranged three-dimensional face model that is the fitting result of the previous frame (A2). If the difference is equal to or greater than a predetermined threshold (B1), the 3D face model of the estimated reference value is rearranged on the face image instead of the 3D face model that is the fitting result of the previous frame (B1′). , based on the relocated estimated reference value 3D face model, a local search fitting process is performed (B2).

By doing so, it is possible to implement an algorithm that is stable over time and that accurately estimates the positions of facial organ points and the facial orientation without being stabilized at an erroneous estimated position.

Since the technology according to the present disclosure is based on the tracking processing of the 3D face model fitting, the facial organ point position estimation result and the eye opening/closing detection in subsequent processing are temporally stable. Furthermore, the technology according to the present disclosure constantly compares the tracking processing result with the estimated reference value calculated from the global search fitting processing result, and corrects the tracking processing result when there is a deviation of a predetermined threshold or more. (That is, replace with the 3D face model of the estimated reference value). Therefore, for example, when the estimated organ point positions of the entire face are drawn into the side of the face such as the ears due to an action such as scratching the face with a hand, or when the estimated organ point positions of the eyes are pulled by a frame such as glasses. In this case, it is possible to suppress the occurrence of an event in which the estimated position is stabilized at an incorrect position, and as a result, the 3D face model stabilizes without being firmly fitted to the face image, and the face orientation estimation result continues to deviate. .

Therefore, the technique according to the present disclosure can accurately estimate the positions of facial organ points and the facial orientation in a facial image, stably over time, without being stabilized at an erroneous estimated position. Furthermore, since the global search fitting process is to suppress the occurrence of an event in which the estimated position stabilizes at an incorrect position, it is sufficient to perform approximate fitting, and this also maintains real-time performance. can do.

[3D face model fitting used in the present disclosure]
Algorithms for 3D face model fitting utilized in the present disclosure will be described. There are various algorithms for 3D face model fitting. The three-dimensional face model fitting algorithm disclosed in Patent Document 1 is one example used in the technical field of on-vehicle monitoring sensors that require real-time performance and high accuracy. The 3D face model fitting algorithm used in the present disclosure may be the 3D face model fitting algorithm disclosed in Patent Document 1, or may be another 3D face model fitting algorithm.

The 3D face model fitting algorithm utilized in the present disclosure is roughly as follows. Obtained based on the difference between the correct model in which each node of the model is placed at the correct position of the facial feature point using the training image and the error model in which any node is placed at the wrong position, and the error model Correlation information about the node feature values that are obtained is acquired in advance. When detecting facial feature points from an input image, create a three-dimensional model that defines the three-dimensional positions of a plurality of nodes, project each node onto the input image, acquire the node feature amount from the projected points, Based on this node feature amount and learned correlation information, an error estimator indicating the deviation between the current position of each node and the position of the corresponding feature point is obtained. Furthermore, based on this error estimate and the current position of each node, the three-dimensional position of each facial feature point in the input image is estimated, and each node is moved accordingly.

In the "rough fitting" used in the present disclosure, in the learning image used in the learning stage for obtaining the correlation, the difference between the correct model and the error model is relatively large, and the correlation is formed. be done.

On the other hand, in the "detailed fitting" used in the present disclosure, the learning image used in the learning stage for obtaining the correlation has a relatively small difference between the correct model and the error model, thereby forming the correlation. be done.

Algorithms for 3D face model fitting utilized in this disclosure may be other than those described above.

[Application example]
An example to which the face orientation estimation device according to the present disclosure can be applied will be described with reference to FIG. FIG. 2 is a diagram for explaining an application example of the face orientation estimation device 14 according to the present disclosure.

FIG. 2 is a block diagram showing the internal configuration of the vehicle control unit 4 and the driver monitoring sensor 12, both of which are mounted on the automobile. The vehicle control unit 4 includes an ECU (electronic control unit) 6 and an actuator 8 . A plurality of ECUs 6 may be provided, and a plurality of actuators 8 may be provided.

The driver monitoring sensor 12 is a device that monitors the driver's facial expression in real time, and includes a camera 16 that is an imaging device and an image processor 14 that is a face direction estimation device. The image processing unit 14, which is a face orientation estimation device, has a CPU 18 equivalent to a hardware processor, a ROM (Read Only Memory) 20 equivalent to a memory, and a RAM (Random Access Memory) 22 equivalent to a memory. These components are connected to each other via appropriate buses so that data can be sent and received.
Furthermore, the CPU 18 of the driver monitoring sensor 12 and the ECU 6 of the vehicle control unit 4 are connected via a CAN (Control Area Network) 10 .

The CPU 18 controls the execution of programs stored in the ROM 20 or RAM 22 and performs data calculation and processing. The CPU 18 is an arithmetic unit that executes various programs (eg, a program for a 3D face model fitting algorithm). The CPU 18 receives various input data from the camera 16 and the ECU 6 of the vehicle control unit 4 and outputs the calculation results of the input data to the ECU 6 of the vehicle control unit 4 via the CAN 10 and stores them in the RAM 22 .

The ROM 20 is a memory unit from which data can only be read, and is composed of, for example, a semiconductor memory element. The ROM 20 stores programs such as applications executed by the CPU 18, data, and the like.

The RAM 22 is a data rewritable storage unit, and is composed of, for example, a semiconductor memory element. The RAM 22 stores input images from the camera 16, for example.

In the vehicle control unit 4 and the driver monitoring sensor 12 as described above, the face orientation estimation device is implemented by the image processing unit 14 .

[Configuration example]
An embodiment as a configuration example of the face orientation estimation device 14 will be described below.

1. Embodiment 1
1.1. Configuration The configuration of face orientation estimation device 14 according to Embodiment 1 will be described with reference to FIG. FIG. 1 is a block diagram showing the functional configuration of face orientation estimation device 14 according to the present embodiment.

The face orientation estimation device 14 is configured by a face detection section 23 , a face orientation estimation section 24 , an eye open/close detection section 40 and a line of sight estimation section 42 . The face detection unit 23 detects human face image data from image data captured by the camera 16 or the like. The facial orientation estimating unit 24 estimates the positions of facial organ points and facial orientation angles with respect to the detected face image data of a person. The eye open/close detection unit 40 detects the open/close of the eyes based on the detected face image data of the person and the estimated positions of the facial organ points and the face orientation angle data. The line-of-sight estimation unit 42 estimates the line-of-sight direction based on the detected face image data of a person, the estimated positions of facial organ points and face orientation angle data, and the detected eye opening/closing data. do. Note that the face orientation estimation device 14 does not have to include the line-of-sight estimation unit 42 . Furthermore, the face orientation estimation device 14 does not have to include the eye open/close detection section 40 .

The face orientation estimation unit 24 includes a first three-dimensional face model placement unit 26, a global search three-dimensional face model fitting unit 28, an estimated reference value calculation unit 30, a local search three-dimensional face model fitting unit 32, a second three-dimensional A face model placement unit 34, a comparison determination unit 36, and a three-dimensional face model fitting result correction unit 38 are included.

A first three-dimensional face model placement unit 26 initially lays out a three-dimensional face model for face image data.
A global search 3D face model fitting unit 28 fits a 3D face model to the approximate position of the face image data based on the initially arranged 3D face model. That is, the global search three-dimensional face model fitting unit 28 performs global search fitting processing (see FIG. 4, step S34). The global search fitting process is a process of searching the entire face and roughly fitting it. The fitting performed in the global search fitting process by the global search three-dimensional face model fitting unit 28 is the above-described "rough fitting".

The estimated reference value calculation unit 30 calculates the three-dimensional face model of the estimated reference value based on the result obtained by the global search fitting process (see FIG. 4, step S34) for the most recent several frames, for example, the most recent five frames. That is, for example, the position of the facial organ point and the angle of the face direction are calculated as the estimation reference value.

The local search 3D face model fitting unit 32 receives the 3D face model that has undergone the global search fitting process (see step S34 in FIG. 4), the 3D face model that is the result of fitting the previous frame, or the latest estimated reference value. Based on the three-dimensional face model, local search fitting processing (see FIG. 4, steps S40 and S52) is performed on the face image data. The local search fitting process is a process of searching for local areas in the face and fitting them in detail. The local search fitting process performed by the local search three-dimensional face model fitting unit 32 will be described later.

The second three-dimensional face model placement unit 34 places a three-dimensional face model, which is the result of fitting the previous frame, on the face image data.
The comparison/determination unit 36 calculates the face direction angle and the estimated reference value in the 3D face model placed by the second 3D face model placement unit 34 (that is, the 3D face model that is the fitting result of the previous frame). A difference Diff between the estimated reference value calculated by the unit 30 and the face direction angle in the three-dimensional face model is calculated, and it is determined whether or not the difference Diff is smaller than a predetermined threshold. Here, for example, the yaw angle is used to calculate the difference Diff.

If the comparison determination unit 36 determines that the difference Diff is equal to or greater than the predetermined threshold value, the 3D face model fitting result correction unit 38 applies the latest estimated face model instead of the 3D face model that is the fitting result of the previous frame. The three-dimensional face model of the reference value is rearranged on the face image data.

The facial orientation estimating unit 24 calculates the positions of facial organ points and facial orientation angles in the frame based on the local search fitting processing (see step S40 and step S52 in FIG. 4) by the local search three-dimensional face model fitting unit 32. Output.

1.2. Operation The operation of the face orientation estimation device 14 configured as described above will be described below.

1.2.1. Overall Operation FIG. 3 is a flowchart showing the overall operation of the face orientation estimation device 14 according to the first embodiment. After the face direction estimation device 14 starts operating (step S02), the tracking flag "TrackingFlag" is set to "FALSE" (step S04), and "t" is initialized (step S06). The tracking flag "TrackingFlag" is set to "TRUE" at the end of processing for the first frame if a face can be detected in successive frames, and is set to "FALSE" if a frame in which face detection is not possible is reached. be done.

Next, the face detection unit 23 performs face image data detection processing on the image data of the tth frame (step S08). If the face can be detected (step S10, YES), the face orientation estimator 24 estimates the position of the facial organ point and the angle of the face orientation (step S14), and increments "t" (step S16). If the face cannot be detected (step S10, NO), the tracking flag "TrackingFlag" is set to "FALSE" (step S12), and "t" is incremented (step S16).

If "t" is not the end value (step S18, NO), the face direction estimation process is performed from the face detection process to the next frame (step S08-).

If "t" reaches the end value (step S18, YES), the operation of the face direction estimation device 14 is ended (step S20).

1.2.2. Face Orientation Estimation Processing FIG. 4 is a flow chart showing the details of the face orientation estimation processing (FIG. 3, step S14) of the face orientation estimation unit 24 in the face orientation estimation device 14 according to the first embodiment.

After the start of face direction estimation processing (step S30), first, estimation reference value calculation processing (steps S32 to S36) is executed. First, the first three-dimensional face model placement unit 26 initially lays out a three-dimensional face model for face image data (step S32). Next, the global search 3D face model fitting unit 28 fits a 3D face model to the approximate position of the face image data based on the initially arranged 3D face model (step S34). The fitting performed in the global search fitting process by the global search three-dimensional face model fitting unit 28 is the above-described "rough fitting".

Next, the estimated reference value calculation unit 30 calculates the three-dimensional face model of the estimated reference value from the result obtained by the global search fitting process (step S34) for the most recent predetermined frames, for example, the most recent five frames. , that is, for example, the position of the facial organ point and the angle of the face orientation as the estimated reference value are calculated (updated) (step S36). Data for the latest five frames are used, for example, as follows. Based on a predetermined value, for example, the angle of the face, out of the five frames, the data of the frame with the maximum predetermined value and the data of the frame with the minimum predetermined value are excluded, and the data of the three frames. is calculated as the three-dimensional face model of the estimated reference value.

Note that when the most recent frame is only one frame, it is used as the estimated reference value. If the most recent frames are only two frames, the average value of those two frames is used as the estimated reference value. If the most recent frames are only 3 or 4 frames, the data of the frame with the maximum predetermined value and the data of the frame with the minimum predetermined value are excluded, and the average value of the remaining frames is used as the estimated reference value. do.

The estimated reference value is obtained for the following reasons. First, it is intended to suppress the occurrence of influence due to a large error due to fitting failure or the like. Also, it is to utilize the fact that in a moving image such as 15 frames/second or 30 frames/second, the movement between 5 frames can be regarded as very small.

After the estimated reference value calculation process (steps S32 to S36), it is determined whether or not the tracking flag "TrackingFlag" is "TRUE", that is, whether or not a face has been detected in the previous frame (step S38). . If the tracking flag "TrackingFlag" is not "TRUE" (step S38, NO), still image processing (processing of the first frame of face detection) (steps S40 to S42) is executed. If the tracking flag "TrackingFlag" is "TRUE" (step S38, YES), tracking processing (steps S44 to S52) is executed.

In the still image processing (processing of the first frame of face detection) (steps S40 to S42), first, the local search 3D face model fitting unit 32 performs global search fitting processing (step S34) on the 3D face model. Based on this, local search fitting processing (to be described later) is performed on the face image data (step S40). Next, the tracking flag "TrackingFlag" is set to "TRUE" (step S42). After the still image processing (process of the first frame of face detection), the face direction estimation unit 24 outputs the positions of the facial organ points and the angle of the face direction in the frame based on the result of the local search fitting process (step S40) ( step S54).

In the tracking process (steps S44 to S52), first, the second 3D face model placement unit 34 places the 3D face model, which is the result of fitting the previous frame, on the face image data (step S44). . Next, the comparison determination unit 36 determines the angle of the face orientation in the 3D face model placed by the second 3D face model placement unit 34 (that is, the 3D face model that is the fitting result of the previous frame), and the estimated A difference Diff between the estimated reference value calculated by the reference value calculation unit 30 and the face direction angle in the three-dimensional face model is calculated (step S46). Here, for example, the yaw angle is used to calculate the difference Diff.

If the difference Diff is equal to or greater than the predetermined threshold (step S48: NO), the 3D face model fitting result correction unit 38 uses the latest estimated reference value of 3 instead of the 3D face model that is the fitting result of the previous frame. The dimensional face model is rearranged on the face image data (step S50). In response to this, the local search 3D face model fitting unit 32 performs local search fitting processing (to be described later) on the face image data based on the 3D face model with the latest estimated reference value ( step S52). If the difference Diff is less than the predetermined threshold (step S48, YES), the 3D face model fitting result correction unit 38 does not perform any processing. Local search fitting processing (to be described later) is performed on the face image data based on the three-dimensional face model that is the fitting result of the frame (step S52).

Thus, steps S46, S48, and S50 constitute a correction process.

After the tracking process (steps S44 to S52), the face orientation estimating unit 24 outputs the position of the facial organ point and the face orientation angle in the frame (step S54) from the result of the local search fitting process (step S52).

After outputting the positions of the facial feature points and the angle of the face orientation (step S54), the face orientation estimation processing ends (step S56).

1.2.3. Local Search Fitting Processing Local search fitting processing (steps S40 and S52) in the face orientation estimation processing shown in FIG. 4 will be described. Various local search fitting processes can be assumed. Below is an example.

Set the underlying 3D face model. In this example, as described above, the three-dimensional models that are the basis are the following three.
(1) A three-dimensional face model that has undergone the global search fitting process (that is, the above-described "rough fitting process") (Fig. 4, step S34) (2) A three-dimensional face model that is the fitting result of the previous frame (3) The latest 3D face model with estimated reference value of

Next, the above-described "detailed fitting" is performed from a plurality of directions including the position of the basic 3D face model while shifting the position of the basic 3D face model horizontally or vertically. The shift amount here is, for example, about half the mouth width. Furthermore, the results of fitting from multiple directions including the position of the underlying 3D face model are integrated. In this integration, the fitting score calculated immediately after the "detailed fitting" is equal to or greater than a predetermined threshold (that is, a predetermined first threshold) is weighted by the fitting score and integrated.

While shifting the position of the (based or) 3D face model that has undergone the integration process to the left and right or up and down, etc., from multiple directions including the position of the (based or) 3D face model that has undergone the integration process, details Performing the fitting and integrating the results of the fitting from multiple directions including the position of the 3D face model (underlying or) undergoing the integration process may be performed by iterating until all of the following conditions are met, for example: (repeat).
(Condition 1) The integrated fitting score is greater than a predetermined second threshold.
(Condition 2) The amount of change in face orientation angle is smaller than a predetermined third threshold. Here, the "variation amount of the face orientation angle" is a value of variation from the face orientation angle calculated during the processing flow in the previous iteration.
Note that an upper limit is provided for the number of iterations for ending (terminating) processing.

Through the above, the result of local search fitting is obtained. The result obtained in this way is a stable and accurate estimation of the face direction.

Further, the above-described "detailed fitting" may be used as it is as the local search fitting process (steps S40 and S52).

1.3. Summary As described above, the face orientation estimation device 14 according to the present embodiment includes the face detection unit 23 for detecting human face image data from image data, and a face orientation estimating unit 24 for estimating the position of and the angle of the face orientation. When face images detected by the face detection unit 23 are continuous in a plurality of frames, the face direction estimation unit 24 performs the following for each frame:
(1) Estimated reference value calculation processing for obtaining an estimated reference value from global search fitting processing, and (2) Tracking processing for performing local search fitting processing based on the three-dimensional face model that is the result of fitting of the previous frame. (2) Tracking processing includes correction processing, and correction processing is calculated by the face orientation angle in the 3D face model, which is the result of fitting the previous frame, and (1) estimated reference value calculation processing. A difference between the estimated reference value and the face orientation angle of the 3D face model is calculated, and if the difference is equal to or greater than a predetermined threshold, the 3D face model that is the fitting result of the previous frame is replaced with the latest face model. The three-dimensional face model of the estimated reference value is used as the basis of the local search fitting process, and the face orientation estimation unit 24 calculates the position of the facial organ point and the face orientation for each frame from the result of the local search fitting process. output the angle of

As described above, the face direction estimation apparatus according to the present embodiment stabilizes the face of a person in the image data over time, prevents stabilization at an erroneous estimated position, and accurately identifies facial organ points. Position and face orientation angle can be estimated.

(Other embodiments)
As described above, Embodiment 1 has been described as an example of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can be applied to embodiments in which modifications, replacements, additions, omissions, etc. are made as appropriate.

The face orientation estimation device according to Embodiment 1 is assumed to be applied to a driver monitoring sensor mounted on an automobile, but the application is not limited to the driver monitoring sensor. For example, it can be applied to a monitoring system that monitors the facial expressions of workers in a factory, a detection system that detects a specific person by installing a camera in a station, a square, or the like, and detects the facial expression of that person.

Also, the accompanying drawings and detailed description have been provided to explain the embodiments. Therefore, among the components described in the attached drawings and detailed description, there are not only components essential for solving the problem, but also components not essential for solving the problem in order to exemplify the above technology. can also be included. Therefore, it should not be determined that those non-essential components are essential just because they are described in the accompanying drawings and detailed description.

In addition, the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, replacements, additions, omissions, etc. can be made within the scope of the claims or equivalents thereof.

4... vehicle control unit, 6... ECU, 8... actuator, 10... CAN, 12... driver monitoring sensor, 14... face orientation estimation device (image processing unit), 16. Camera 18 CPU 20 ROM 22 RAM 23 Face detection unit 24 Face orientation estimation unit 26 First three-dimensional face model Placement unit 28 Global search three-dimensional face model fitting unit 30 Estimation reference value calculation unit 32 Local search three-dimensional face model fitting unit 34 Second three-dimensional face model Placement unit 36 Comparison determination unit 38 Three-dimensional face model fitting result correction unit 40 Eye open/close detection unit 42 Gaze estimation unit.

Claims (4)

a face detection unit that detects human face image data from image data; and
A face direction estimating device comprising a face direction estimating unit for estimating positions of facial organ points and angles of face direction with respect to detected face image data of a person,
When the face images detected by the face detection unit are continuous in a plurality of frames, the face orientation estimation unit performs the following for each frame:
(1) an estimated reference value calculation process for obtaining an estimated reference value from the global search fitting process;
(2) based on the three-dimensional face model that is the fitting result of the previous frame, a tracking process that performs a local search fitting process;
The (2) tracking process includes a correction process,
The correction process is
Calculate the difference between the face orientation angle of the three-dimensional face model that is the result of the fitting of the previous frame and the face orientation angle of the three-dimensional face model of the estimated reference value calculated by the estimated reference value calculation process (1). , if the difference is equal to or greater than a predetermined threshold, the 3D face model with the latest estimated reference value is used as the basis of the local search fitting process instead of the 3D face model that is the fitting result of the previous frame. Dimensional face model and
The face orientation estimation unit outputs the position of the facial organ point and the angle of the face orientation for each frame from the result of the local search fitting process.
Face orientation estimation device. The estimated reference value is calculated from the results obtained in the global search fitting process in the last predetermined few frames,
The face direction estimation device according to claim 1. A computer-implemented face orientation estimation method comprising:
a step of detecting human face image data from image data;
estimating positions of facial organ points and facial orientation angles with respect to the detected human face image data;
and outputting the position of the facial organ point and the angle of the face orientation in the facial image data,
In the estimating step, when the face images detected in the detecting step are continuous in a plurality of frames, for each frame,
(1) an estimated reference value calculation process for obtaining an estimated reference value from the global search fitting process;
(2) based on the three-dimensional face model that is the fitting result of the previous frame, it includes a tracking process that performs a local search fitting process;
The (2) tracking process includes a correction process,
The correction process is
Calculate the difference between the face orientation angle of the three-dimensional face model that is the result of the fitting of the previous frame and the face orientation angle of the three-dimensional face model of the estimated reference value calculated by the estimated reference value calculation process (1). , if the difference is equal to or greater than a predetermined threshold, the 3D face model with the latest estimated reference value is used as the basis of the local search fitting process instead of the 3D face model that is the fitting result of the previous frame. Dimensional face model and
In the outputting step, the position of the facial organ point and the face orientation angle are output for each frame from the result of the local search fitting process.
Face orientation estimation method. The estimated reference value is calculated from the results obtained in the global search fitting process in the last predetermined few frames,
The face direction estimation method according to claim 3.

Images