JP2021033938A - Device and method for estimating facial direction - Google Patents

Abstract

To provide a device and a method for estimating a facial direction of a person in image data stably and precisely.SOLUTION: The facial direction estimating device includes: a face detector for detecting facial image data of a person from image data; a facial direction estimation unit for estimating the angle of the direction of the face in the facial image data of a detected person. The facial direction estimation unit includes: a three-dimensional face model initial setting unit: a three-dimensional face model moving (displacing) unit; a three-dimensional face model minute fitting unit; a three-dimensional face model fitting score calculation unit; and a three-dimensional face model integration unit.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a face orientation estimation device and a method for estimating the face orientation of a person in image data in a stable and accurate manner.

Various techniques are known that detect a human face from image data and estimate the orientation of the detected face. For example, Patent Document 1 estimates the orientation of a face in an image by fitting a three-dimensional face model that determines a plurality of three-dimensional positions corresponding to a plurality of feature points on a human face to the face in the image. A three-dimensional face model fitting algorithm and a detection device using the algorithm are disclosed.

However, when trying to detect the orientation of a face facing in an oblique direction by the conventional three-dimensional face model fitting algorithm as described above, the estimation accuracy may not be stable. In the case of a face facing diagonally close to the profile, that is, a face having a large yaw angle from the front, and particularly in the case of a face on which ornaments such as eyeglasses are hung, the back side of the face. Since the facial feature points are hard to see (that is, they are not clear on the image), the 3D face model is hard to fit, resulting in a local solution, which may affect the facial orientation estimation accuracy. Because there is.

The face feature point detection device disclosed in Patent Document 2 performs fitting processing with a deformed three-dimensional face model having an unnatural shape by predetermining rules for the positional relationship of face feature points, and falls into a local solution. It suppresses the result of detecting facial feature points with low accuracy. However, when the facial feature point detecting device has a large yaw angle from the front such as a profile, especially when a decorative item such as eyeglasses is worn, when the fitting process is performed by a natural facial feature point model. In addition, it is possible to fall into a local solution. As a result, in the face feature point detection, there is a possibility that the face feature point detection with low accuracy, that is, the face orientation angle estimation with low accuracy is performed.

Further, the feature point position detecting device disclosed in Patent Document 3 allows more desired facial feature points by inputting initial information of a small number of major facial feature point positions such as eyes, nose, and mouth from the outside. Estimate the approximate position of. Since the feature point position detection device starts the face feature point search from the input position, it is possible to prevent erroneous fitting of a face shape such as a posture or facial expression different from the correct answer, that is, to fall into a local solution. Will be done. However, in the feature point position detecting device, a small number of facial feature points must be input from the outside in advance, and therefore, the feature point position detecting device is not suitable for high-speed and real-time processing.

JP-A-2007-249280 Japanese Unexamined Patent Publication No. 2011-128966 Japanese Patent No. 6387831

The present disclosure provides an algorithm for estimating the face orientation of a person in image data in a stable and accurate manner, and a face orientation estimation device and method using the algorithm.

The face orientation estimation device of the present disclosure is
It is a face orientation estimation device including a face detection unit that detects human face image data from image data and a face orientation estimation unit that estimates the angle of the face orientation with respect to the detected human face image data. The face orientation estimation unit includes a three-dimensional face model initial setting unit, a three-dimensional face model moving (shifting) unit, a three-dimensional face model detailed fitting unit, a three-dimensional face model fitting score calculation unit, and three-dimensional. Including face model integration part
The three-dimensional face model initial setting unit sets the three-dimensional face model on the face image data.
In addition
(1) The three-dimensional face model moving (shifting) unit shifts the position of a given three-dimensional face model by the amount of shifting one or more times to generate a plurality of three-dimensional face models for detailed fitting.
(2) The three-dimensional face model detailed fitting unit performs a process of performing detailed fitting of a plurality of three-dimensional face models for detailed fitting generated by shifting the position with respect to the face image data.
(3) The three-dimensional face model fitting score calculation unit calculates the fitting score of the three-dimensional face model immediately after the detailed fitting process by the three-dimensional face model detailed fitting unit.
(4) The three-dimensional face model integration unit integrates a plurality of three-dimensional face model fitting results to calculate the angle of the face orientation.
The face orientation estimation unit repeatedly executes the processes (1) to (4) and outputs the angle of the face orientation.

The face orientation estimation device and method according to the present disclosure can estimate the face orientation of a person in the image data in a stable and accurate manner.

It is a block diagram which shows the functional structure of the face orientation estimation apparatus which concerns on Embodiment 1. FIG. It is a figure for demonstrating the application example of the face orientation estimation apparatus which concerns on this disclosure. It is a flowchart which shows the whole operation of the face orientation estimation apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows the face orientation estimation processing of the face orientation estimation part in the face orientation estimation apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows the integration process of M 3D face model fitting results of the 3D face model integration part in the face orientation estimation apparatus which concerns on Embodiment 1. FIG. In the flowchart shown in FIG. 5a, after the start, the variables are initialized, and the M fitting results are weighted by the fitting score Sm and integrated (added). It is a flowchart which shows the integration process of M 3D face model fitting results of the 3D face model integration part in the face orientation estimation apparatus which concerns on Embodiment 1. FIG. In the flowchart shown in FIG. 5b, the integrated value is calculated by normalization, the face-facing angle is calculated, and the process ends. FIGS. 6 (a-1) and 6 (a-2) show a three-dimensional face model composed of only points (nodes) of two types for a face image having a yaw angle of about −40 °. , It is a figure which shows the combined state. 6 (b-1) to 6 (b-7) are views showing the shape of the three-dimensional face model when the yaw angle is changed. It is a figure explaining the outline of the face orientation estimation algorithm which concerns on this disclosure.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters and duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessary redundancy of the following description and to facilitate the understanding of those skilled in the art.

It should be noted that the inventors intend to limit the subject matter described in the claims by those skilled in the art by providing the accompanying drawings and the following description in order to fully understand the present disclosure. is not it.

[Background to this disclosure]
Various techniques have been developed so far for detecting a human face from image data and estimating the orientation of the detected face. Patent Document 1 estimates the orientation of a face in an image by fitting a three-dimensional face model that determines a plurality of three-dimensional positions corresponding to a plurality of feature points on a human face to the face in the image. A three-dimensional face model fitting algorithm and a detection device using the algorithm are disclosed.

However, when trying to detect the orientation of a face facing in an oblique direction by a conventional three-dimensional face model fitting algorithm, the estimation accuracy is not stable. A three-dimensional face because the facial feature points on the back side of the face are not clear on the image when the yaw angle from the front is large and it is close to the profile, especially when ornaments such as glasses are hung. This is because the model is difficult to fit and may result in a local solution.

6 (b-1) to 6 (b-7) are views showing the shape of the three-dimensional face model when the yaw angle is changed. The front-facing model has a yaw angle of 0 ° (FIG. 6 (b-7)). As we move from FIG. 6 (b-6) to FIG. 6 (b-1), the 3D face model points deeply to the right, and the yaw angle shifts from "about -10 °" to "about -60 °". There is.

In FIGS. 6 (a-1) and 6 (a-2), a three-dimensional face model composed of only points (nodes) of two types is combined with a face image having a yaw angle of about −40 °. It is a figure which shows the state. In FIG. 6 (a-2), a three-dimensional face model having a yaw angle of about −40 ° is matched, and in FIG. 6 (a-1), a three-dimensional face model having a yaw angle of about −50 ° is matched. Has been done.
As shown in FIG. 6 (a-1), for a face image having a yaw angle of about −40 °, an erroneous estimation may occur if the yaw angle is about −50 °. This is due to the following factors.
(Factor 1) Many facial feature points in a face image with a yaw angle of about -40 ° fit even for a three-dimensional face model with a yaw angle of about -50 °.
(Factor 2) Since the facial feature points on the back side (right side here) of the face are not clear (that is, it is hard to see), the three-dimensional face model with a yaw angle of about -40 ° is also three-dimensional with a yaw angle of about -50 °. It is difficult for the facial feature points on the back side of the face to fit to any of the face models.
That is, the three-dimensional face model fitting results in a local solution, which causes erroneous estimation and affects the accuracy of face orientation estimation.

The present disclosure is a technique devised by the inventor in order to solve such a problem. FIG. 7 is a diagram illustrating an outline of the face orientation estimation algorithm according to the present disclosure. First, the three-dimensional face model is placed at the initial position of the face image (FIG. 7A), and the three-dimensional face model is fitted at the approximate position. That is, rough fitting is performed on the face image (FIG. 7 (b)). Rough fitting does not have to be performed. Next, the position of the three-dimensional face model is shifted to the left, right, up and down, and detailed fitting is performed from a plurality of directions (FIGS. 7 (c-1) and 7 (c-2)). The amount of shift here is, for example, about half the width of the mouth. In addition, the results of fitting from multiple directions are integrated. At the time of this integration, the result that the fitting score calculated immediately after the detailed fitting is equal to or higher than a predetermined threshold value (that is, a predetermined third threshold value) is weighted by the fitting score and integrated.

The position of the three-dimensional face model that has undergone the initial placement process that may include the rough fitting process or the integrated process is shifted to the left, right, up and down, etc., and detailed fitting is performed from multiple directions, and fitting from multiple directions. Integrating the results is repeated (iterated), for example, until all of the following conditions are met.
(Condition 1) The integrated fitting score is larger than the predetermined first threshold value.
(Condition 2) The amount of variation in the face-facing angle is smaller than the predetermined second threshold value.
However, there may be an upper limit on the number of iterations to end (discontinue) the process.

Through the above, the final result of the three-dimensional face model fitting is obtained (FIG. 7 (d)). The final result obtained in this way is to estimate the face orientation stably and accurately.

[Three-dimensional face model fitting used in this disclosure]
The three-dimensional face model fitting algorithm used in the present disclosure will be described. There are various algorithms for 3D face model fitting. The three-dimensional face model fitting algorithm disclosed in Patent Document 1 is an example used in the technical field of an in-vehicle monitoring sensor that requires real-time performance and high accuracy. The three-dimensional face model fitting algorithm used in the present disclosure may be the three-dimensional face model fitting algorithm disclosed in Patent Document 1, or may be another three-dimensional face model fitting algorithm.

The three-dimensional face model fitting algorithm used in the present disclosure is as follows. Obtained based on the difference between the correct model in which each node of the model is placed in the correct position of the facial feature point and the error model in which any node is placed in the wrong position, and the error model using the training image. Information on the correlation with respect to the created node feature amount is acquired in advance. When detecting facial feature points from the input image, a 3D model that defines the 3D positions of multiple nodes is created, each node is projected onto the input image, and the node feature amount is acquired from the projection points. Based on this node feature amount and the learned correlation information, an error estimator indicating the deviation between the current position of each node and the position of the corresponding feature point is acquired. Further, based on this error estimator and the current position of each node, the three-dimensional position of each face feature point in the input image is estimated, and each node is moved accordingly.

In the "rough fitting" used in the present disclosure, in the learning image used in the learning stage for acquiring the correlation, an image in which the difference between the correct answer model and the error model is relatively large is used, and a correlation is formed by this. Will be done. On the other hand, in the "detailed fitting" used in the present disclosure, in the learning image used in the learning stage for acquiring the correlation, an image in which the difference between the correct answer model and the error model is relatively small is used, and a correlation is formed by this. Will be done.

The three-dimensional face model fitting algorithm used in the present disclosure may be other than the above.

[Application example]
An example to which the face orientation estimation device according to the present disclosure can be applied will be described with reference to FIG. FIG. 2 is a diagram for explaining an application example of the face orientation estimation device 14 according to the present disclosure.

FIG. 2 is a block diagram showing an internal configuration of a vehicle control unit 4 and a driver monitoring sensor 12 mounted on an automobile. The vehicle control unit 4 includes an ECU (electronic control unit) 6 and an actuator 8. The ECU 6 may have a plurality of ones, and the actuator 8 may also have a plurality of ones.

The driver monitoring sensor 12 is a device that monitors the driver's facial expression in real time, and includes a camera 16 that is an image pickup device and an image processing unit 14 that is a face orientation estimation device. The image processing unit 14 which is a face orientation estimation device has a CPU 18 corresponding to a hardware processor, a ROM (Read Only Memory) 20 corresponding to a memory, and a RAM (Random Access Memory) 22 corresponding to the memory. Each of these configurations is connected to each other via an appropriate bus so that data can be transmitted and received.
Further, the CPU 18 of the driver monitoring sensor 12 and the ECU 6 of the vehicle control unit 4 are connected via a CAN (Control Area Network) 10.

The CPU 18 controls, calculates, and processes data related to the execution of a program stored in the ROM 20 or the RAM 22. The CPU 18 is an arithmetic unit that executes various programs (for example, a program for a three-dimensional face model fitting algorithm). The CPU 18 receives various input data from the camera 16 and the ECU 6 of the vehicle control unit 4, outputs the calculation result of the input data to the ECU 6 of the vehicle control unit 4 via the CAN 10, and stores the input data in the ROM 20 and the RAM 22. To do.

The ROM 20 is a storage unit capable of only reading data, and is composed of, for example, a semiconductor storage element. The ROM 20 stores, for example, an application program or data executed by the CPU 18.

The RAM 22 is a storage unit capable of rewriting data, and is composed of, for example, a semiconductor storage element. The RAM 22 stores an input image or the like from the camera 16.

In the vehicle control unit 4 and the driver monitoring sensor 12 as described above, the face orientation estimation device is realized by the image processing unit 14.

[Configuration example]
Hereinafter, embodiments of the face orientation estimation device 14 as a configuration example will be described.

1. Embodiment 1
1.1. Configuration The configuration of the face orientation estimation device 14 according to the first embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a functional configuration of the face orientation estimation device 14 according to the present embodiment.

The face orientation estimation device 14 is composed of a face detection unit 23, a face orientation estimation unit 24, an eye opening / closing detection unit 38, and a line-of-sight estimation unit 40. The face detection unit 23 detects human face image data from the image data captured by the camera 16 or the like. The face orientation estimation unit 24 estimates the orientation (angle) of the face with respect to the detected face image data of the person. The eye opening / closing detection unit 38 detects the opening / closing of the eyes based on the detected face image data of the person and the estimated face orientation data. The line-of-sight estimation unit 40 estimates the direction of the line of sight based on the detected face image data of the person, the estimated face orientation data, and the detected eye opening / closing data. The face orientation estimation device 14 does not have to include the eye opening / closing detection unit 38 and the line-of-sight estimation unit 40.

Further, the face orientation estimation unit 24 includes a three-dimensional face model initial setting unit 26, a three-dimensional face model moving (shifting) unit 30, a three-dimensional face model detailed fitting unit 32, a three-dimensional face model fitting score calculation unit 34, and a three-dimensional face model fitting score calculation unit 34. The three-dimensional face model integration unit 36 is included. Further, the three-dimensional face model initial setting unit 26 includes a three-dimensional face model rough fitting unit 28. The three-dimensional face model initial setting unit 26 may not include the three-dimensional face model rough fitting unit 28.

The three-dimensional face model initial setting unit 26 arranges the three-dimensional face model at an initial position with respect to the face image data. At this time, the three-dimensional face model initial setting unit 26 first places the three-dimensional face model at approximately the position with respect to the face image data by the three-dimensional face model rough fitting unit 28 based on the arranged three-dimensional face model. Fitting, rough fitting is performed, and the 3D face model after rough fitting is used as the initial 3D face model. Rough fitting by the three-dimensional face model rough fitting unit 28 may not be performed.

The three-dimensional face model moving (shifting) unit 30 shifts the position of the three-dimensional face model left and right, up and down, etc. one or more times for fitting from a plurality of directions. The amount of shift here is, for example, about half the width of the mouth. This is due to the following reasons.
(Reason 1) In order to ensure that the features at both ends of each facial organ point (for example, the feature on the left side when fitting from the left) are at least fitted in the fitting process in multiple directions.
(Reason 2) "Mouth width" is the maximum length of organ points and should be used as a reference.
The amount of shift may be another value.
By shifting, for example, M (M ≧ 1) three-dimensional face models for detailed fitting are generated.

The three-dimensional face model detailed fitting unit 32 performs detailed fitting based on one of the M (mth (1 ≦ m ≦ M)) three-dimensional face models.
The 3D face model fitting score calculation unit 34 calculates the fitting score of the 3D face model immediately after the detailed fitting by the 3D face model detailed fitting unit 32.
Further, the three-dimensional face model integration unit 36 integrates the results of detailed fitting from a plurality of directions to calculate the angle of the face orientation.

1.2. Operation The operation of the face orientation estimation device 14 configured as described above will be described below.

1.2.1. Overall operation FIG. 3 is a flowchart showing the overall operation of the face orientation estimation device 14 according to the first embodiment. After the operation of the face orientation estimation device 14 is started (step S02), the initialization process of "t" is performed (step S04).

Next, the face detection unit 23 performs face image data detection processing on the image data at the t-frame (step S06). If the face can be detected (step S08 / YES), the face orientation estimation unit 24 estimates the face orientation (step S10) and increments “t” (step S12). If face detection cannot be performed (step S08 / NO), "t" is incremented as it is (step S12).

If "t" is not the end value (step S14 / NO), the face orientation estimation process is performed from the face detection process for the next frame (step S06 to).

When “t” becomes the end value (YES in step S14), the operation of the face orientation estimation device 14 ends (step S16).

1.2.2. Face orientation estimation process FIG. 4 is a flowchart showing the contents of the face orientation estimation process (FIG. 3, step S10) of the face orientation estimation unit 24 in the face orientation estimation device 14 according to the first embodiment.

After the face orientation estimation process is started (step S20), the three-dimensional face model initial setting unit 26 initially arranges (initializes) the three-dimensional face model on the face image data (step S22). At this time, the three-dimensional face model initial setting unit 26 first performs rough fitting on the face image data by the three-dimensional face model rough fitting unit 28 based on the arranged three-dimensional face model, and 3 after the rough fitting. The three-dimensional face model may be the initial arrangement of the three-dimensional face model. Rough fitting by the three-dimensional face model rough fitting unit 28 does not have to be performed. Further, the rough fitting is not performed in the three-dimensional face model initial setting unit 26 which does not include the three-dimensional face model rough fitting unit 28. "Rough fitting" is described in the above-mentioned [3D face model fitting used in the present disclosure].

Next, the number of iterations is initialized (i = 1) (step S26). Subsequently, the three-dimensional face model moving (shifting) unit 30 shifts the position of the three-dimensional face model left and right, up and down, etc. one or more times (step S28). As described above, the amount of shift is, for example, about half the width of the mouth. By shifting, M (M ≧ 1) three-dimensional face models for detailed fitting are generated.

When the number of iterations (i) is "1", the three-dimensional face model that is the basis of the position shift processing (step S32) of the three-dimensional face model has undergone the initial placement processing (step S22). When the number of iterations (i) is plural, the three-dimensional face model that is the basis of the position shift processing (step S32) of the three-dimensional face model is "M three-dimensional face models" in step S40 described later. This is the process of "integrating the fitting result" immediately before and at least once.

Next, m, which is a variable that is incremented from 1 to M, is initialized (m = 1) (step S30). Subsequently, the 3D face model detailed fitting unit 32 finely fits the 3D face model to the face image data based on the mth 3D face model among the M pieces (step S32). "Detailed fitting" is described in the above-mentioned [3D face model fitting used in the present disclosure].

Next, the 3D face model fitting score calculation unit 34 calculates the fitting score of the 3D face model immediately after the detailed fitting process (step S32) by the 3D face model detailed fitting unit 32 (step S34). The fitting score is calculated as follows, for example. The correlation value between the feature amount stored in the ROM 20 and acquired based on the correct answer model and the feature amount acquired based on the m-th three-dimensional face model that has undergone the detailed fitting process in step S32 is obtained. Use as a fitting score. The fitting score is not limited to this value.

Next, the variable m is incremented (step S36). Here, if m is M or less (step S38 / NO), detailed fitting processing (step S32) and calculation of the fitting score (step S34) are performed on the incremented m-th three-dimensional face model.

If m is larger than M (step S38 / YES), the three-dimensional face model integration unit 36 integrates M three-dimensional face model fitting results (step S40). This step S40 will be described later with reference to FIGS. 5a and 5b. By integrating the three-dimensional face model fitting results of M pieces by the three-dimensional face model integrating unit 36, the angle of the face orientation in the face image is calculated.

Subsequently, if the integrated fitting score is larger than the predetermined first threshold value and the face orientation variation is smaller than the predetermined second threshold value (step S42 · YES), the face orientation estimation unit 24 determines the final face orientation angle. Is output (step S48), and the face orientation estimation process is completed (step S50). Here, the "integrated fitting score" is a fitting score related to the three-dimensional face model via the integrated processing (step S40) of M three-dimensional face model fitting results. That is, the "integrated fitting score" is, for example, equal to or higher than a predetermined third threshold value in the integration process (step S40) of M three-dimensional face model fitting results, which will be described later, and is therefore used for weighting. It is an average value for the fitting score to be obtained. Further, the “face orientation variation” is a variation value from the face orientation angle calculated during the processing flow in which the number of iterations is “i-1”. When i = 1, that is, when i-1 = 0, the fluctuation value from the face orientation angle of the three-dimensional face model that has undergone the rough fitting process in step S24 is defined as “face orientation variation”.

If the integrated fitting score is greater than the predetermined first threshold value and the facial variation is smaller than the predetermined second threshold value (step S42 / NO), the number of iterations is incremented (i ++), and then the next step is performed. Enter the number of iterations of (step S44).

If the number of iterations (i) is equal to or less than the predetermined upper limit number of iterations "N" (step S46 / NO), the three-dimensional face model position shift processing (step S28) to M three-dimensional face model fittings. The flow of the result integration process (step S40) is performed (repeated). As the number of iterations (i) increases, the integration process (step S40) of M three-dimensional face model fitting results by the three-dimensional face model integration unit 36 is performed more often. That is, the three-dimensional face model that is the basis of the position shift processing (step S28) of the three-dimensional face model also goes through a lot of integration processing (step S40) of M three-dimensional face model fitting results.

If the number of iterations (i) is larger than the predetermined upper limit number of iterations "N" (step S46 / YES), the face orientation estimation unit 24 outputs the angle of the face orientation at that time (step S48). The face orientation estimation process is completed (step S50).

1.2.3. Integration processing of M 3D face model fitting results FIGS. 5a and 5b show M 3D face model fitting results of the 3D face model integration unit 36 in the face orientation estimation device 14 according to the first embodiment. It is a flowchart which shows the integrated process of. In the flowcharts shown in FIGS. 5a and 5b, the face orientation is calculated by weighting and integrating the M detailed fitting results with the fitting score.

The variables and constants used in the flowcharts shown in FIGS. 5a and 5b are defined as follows.
-"ParaN": The number of predetermined fitting model parameters (constant).
-"ModelPara [k]": The kth fitting model parameter among the "paraN" fitting model parameters.
The "fitting model parameter" represents, for example, how many times the three-dimensional face model is rotated in the left-right direction.
-"ValidModelScoreSum": The total value of the fitting scores of the three-dimensional face model (hereinafter referred to as the valid model) satisfying "fitting score Sm ≥ predetermined third threshold value".
-"ValidModelParaSum [k]": For the kth fitting model parameter "ModelPara [k]", weight the valid model satisfying "fitting score Sm ≥ predetermined third threshold value" with the fitting score Sm for fitting. The total (integrated) model parameter values.
-"Model [m] .ModelPara [k]": The kth fitting model parameter in the mth 3D face model.

In the flowcharts shown in FIGS. 5a and 5b, after roughly starting (FIG. 5a, step S60), the variables are initialized (FIG. 5a, steps S62 to S70), and M detailed fitting results are weighted by the fitting score Sm. Accumulation (FIGS. 5a, steps S72 to S88), and after integration, the integrated value is calculated by normalization to calculate the face-facing angle (FIG. 5b, steps S90 to 98), and the process ends (FIG. 5b, steps S100). .. Each step of the flowchart shown in FIGS. 5a and 5b will be described below.

In the flowchart shown in FIG. 5a, after the start (step S60), the "validModelScoreSum" (total value of the fitting scores of the valid models) is initialized (step S62), the variable k is incremented from 0 to "paraN", and all the "validModelScoreSum" "validModelParaSum [k]" is initialized (step S64, step S66, step S68, and step S70).

When the initialization of "validModelParaSum [k]" is completed (step S70 / NO), "1" is first set for m to be incremented from 1 to M (m = 1) (step S72).

Subsequently, if the fitting score Sm is smaller than the predetermined third threshold value (step S74 / YES) for the m-th three-dimensional face model among the M pieces, the fitting result integration process (steps S76 to S84) is performed. Without doing so, m is incremented (step S86), and as long as m is M or less (step S88 / NO), step S74 or less is repeated.

If the fitting score Sm is equal to or higher than the predetermined third threshold value (step S74 / NO), the variable k is incremented from 0 to “paraN”, and the k-th in the m-th three-dimensional face model among the M pieces. Each of the fitting model parameters "Model [m] .ModelPara [k]" is weighted by the fitting score Sm, and the fitting model parameter values are added. That is, the following processing is performed (step S76, step S78, step S80, and step S82).

Further, the fitting process of the fitting score Sm of the effective model is performed. That is, the following processing is performed (step S84).

Subsequently, m is incremented (step S86), and as long as m is M or less (step S88 / NO), step S74 or less is repeated. If m becomes larger than M (step SS88 / YES), the process proceeds to the processing steps of the calculation of the integrated value by normalization and the calculation of the face orientation angle in steps S90 to FIG. 5b shown in FIG. 5b.

In the flowchart shown in FIG. 5b, the variable k is incremented from 0 to “paraN”, and the k-th fitting model parameter “ModelPara [k]” is normalized to calculate the integrated value. That is, the integrated value "ModelPara [k]" is obtained by dividing "validModelParaSum [k]" by "validModelScoreSum" as in the following processing (step S90, step S92, step S94, and Step S96).

The angle of the face orientation in the face image data is calculated from the integrated value of the fitting model parameters, which is the integrated fitting result (step S98), and the integration process of the M three-dimensional face model fitting results is completed (step). S100).

1.3. Summary As described above, the face orientation estimation device according to the present embodiment includes the face detection unit 23 that detects the human face image data from the image data, and the face orientation with respect to the detected human face image data. The face orientation estimation device 14 includes a face orientation estimation unit 24 for estimating an angle. The face orientation estimation unit 24 includes a 3D face model initial setting unit 26, a 3D face model moving (shifting) unit 30, a 3D face model detailed fitting unit 32, and a 3D face model fitting score calculation unit 34. And the three-dimensional face model integration unit 36. The three-dimensional face model initial setting unit 26 sets the three-dimensional face model on the face image data. Further, (1) the three-dimensional face model moving (shifting) unit 30 shifts the position of a given three-dimensional face model one or more times by the amount of shifting to generate a plurality of three-dimensional face models for detailed fitting. , (2) The three-dimensional face model detailed fitting unit 32 performs a process of detailed fitting the three-dimensional face model for a plurality of detailed fittings generated by shifting the position with respect to the face image data, and (3). The 3D face model fitting score calculation unit 34 calculates the fitting score of the 3D face model immediately after the detailed fitting process by the 3D face model detailed fitting unit 32, and (4) the 3D face model integration unit 36 determines. The angle of the orientation of the face is calculated by integrating a plurality of three-dimensional face model fitting results. The face orientation estimation unit 14 repeatedly executes the processes (1) to (4) above, and outputs the angle of the face orientation.

As described above, the face orientation estimation device according to the present embodiment can estimate the angle of the face orientation with respect to the human face in the image data in a stable and accurate manner.

(Other embodiments)
As described above, the first embodiment has been described as an example of the technology disclosed in the present application. However, the technique in the present disclosure is not limited to this, and can be applied to embodiments in which changes, replacements, additions, omissions, etc. are made as appropriate.

The face orientation estimation device according to the first embodiment is assumed to be applied to a driver monitoring sensor mounted on an automobile, but the application example is not limited to the driver monitoring sensor. For example, it can be applied to a monitoring system that monitors the facial expression of a worker in a factory, a detection system that detects a specific person after installing a camera in a station, a plaza, or the like, and detects the facial expression of that person.

Also provided are accompanying drawings and detailed description to illustrate embodiments. Therefore, among the components described in the attached drawings and the detailed description, not only the components essential for solving the problem but also the components not essential for solving the problem in order to exemplify the above technology. Can also be included. Therefore, the fact that these non-essential components are described in the accompanying drawings or detailed description should not immediately determine that those non-essential components are essential.

Further, since the above-described embodiment is for exemplifying the technique in the present disclosure, various changes, replacements, additions, omissions, etc. can be made within the scope of claims or the equivalent scope thereof.

4 ... Vehicle control unit, 6 ... ECU, 8 ... Actuator, 10 ... CAN, 12 ... Driver monitoring sensor, 14 ... Face orientation estimation device (image processing unit), 16.・ ・ Camera, 18 ・ ・ ・ CPU, 20 ・ ・ ・ ROM, 22 ・ ・ ・ RAM, 23 ・ ・ ・ Face detection part, 24 ・ ・ ・ Face orientation estimation part, 26 ・ ・ ・ 3D face model initial setting part , 28 ... 3D face model rough fitting part, 30 ... 3D face model moving (shifting) part, 32 ... 3D face model detailed fitting part, 34 ... 3D face model fitting score calculation Unit, 36 ... 3D face model integration unit, 38 ... eye opening / closing detection unit, 40 ... line-of-sight estimation unit.

Claims (8)

A face detection unit that detects human face image data from image data, and
It is a face orientation estimation device including a face orientation estimation unit that estimates the angle of the face orientation with respect to the detected human face image data.
The face orientation estimation unit
3D face model initial setting part and
3D face model movement (shift) part,
3D face model details fitting part and
3D face model fitting score calculation unit and
Including 3D face model integration part
The three-dimensional face model initial setting unit initially arranges the three-dimensional face model on the face image data.
In addition
(1) The three-dimensional face model moving (shifting) unit shifts the position of a given three-dimensional face model by the amount of shifting one or more times to generate a plurality of three-dimensional face models for detailed fitting.
(2) The three-dimensional face model detailed fitting unit performs a process of finely fitting a three-dimensional face model for a plurality of detailed fittings generated by shifting the position with respect to the face image data.
(3) The three-dimensional face model fitting score calculation unit calculates the fitting score of the three-dimensional face model immediately after the detailed fitting process by the three-dimensional face model detailed fitting unit.
(4) The three-dimensional face model integration unit integrates a plurality of three-dimensional face model fitting results to calculate the angle of the face orientation.
The face orientation estimation unit is a face orientation estimation device that repeatedly executes the processes (1) to (4) and outputs the angle of the face orientation. In the above (4), the fitting score related to the three-dimensional face model that has passed through the integration processing of the plurality of three-dimensional face model fitting results in the above (4) is larger than the predetermined first threshold value and is the number of repetitions in the previous time. Until the variation value from the angle of face orientation becomes smaller than the predetermined second threshold value.
Or
Until the number of iterations exceeds the maximum number of iterations
Executed,
After repeating the above processes (1) to (4), the angle of the face orientation is output.
The face orientation estimation device according to claim 1. In the process of (4) above, the three-dimensional face model integration unit integrates a plurality of three-dimensional face model fitting results and calculates the angle of the face orientation.
For a plurality of detailed fitting results, if each fitting score is equal to or higher than a predetermined third threshold value, the fitting score is weighted and integrated.
After integration, the integrated value is calculated by normalization to calculate the face-facing angle.
The face orientation estimation device according to claim 1. The fitting score related to the three-dimensional face model that has passed through the integrated processing of the plurality of three-dimensional face model fitting results in (4) above is the average value of the fitting scores that are equal to or higher than the predetermined third threshold value among the plurality of fitting scores. Is,
The face orientation estimation device according to claim 3. A computer-run face-to-face estimation method
Steps to detect human face image data from image data,
The steps to initially place the 3D face model on the face image data, and
A face orientation estimation method including a step of repeatedly executing the following processes (processes 1) to (process 4) and outputting the angle of the face orientation.
(Process 1) A process of shifting the position of a given three-dimensional face model one or more times by a shift amount to generate a plurality of three-dimensional face models for detailed fitting.
(Process 2) A process of detailed fitting a three-dimensional face model for a plurality of detailed fittings generated by shifting the position with respect to the face image data.
(Process 3) A process of calculating the fitting score of the three-dimensional face model immediately after the detailed fitting process in the above (process 2).
(Process 4) A process of integrating a plurality of three-dimensional face model fitting results and calculating the angle of face orientation. The fitting score related to the three-dimensional face model that has passed through the integration process of the plurality of three-dimensional face model fitting results in the above (process 4) is larger than the predetermined first threshold value, and the above-mentioned (process) is performed in the previous number of repetitions. Until the fluctuation value from the angle of the face orientation in 4) becomes smaller than the predetermined second threshold value.
Or
Until the number of iterations exceeds the maximum number of iterations
Executed,
After repeating the above processes (process 1) to (process 4), the angle of the face orientation is output.
The face orientation estimation method according to claim 5. In the process of calculating the angle of the face orientation by integrating the plurality of three-dimensional face model fitting results in the above (process 4),
For a plurality of detailed fitting results, if each fitting score is equal to or higher than a predetermined third threshold value, the fitting score is weighted and integrated.
After integration, the integrated value is calculated by normalization to calculate the face-facing angle.
The face orientation estimation method according to claim 5. The fitting score related to the three-dimensional face model that has passed through the integrated processing of the plurality of three-dimensional face model fitting results in the above (process 4) is the average of the fitting scores that are equal to or higher than the predetermined third threshold value among the plurality of fitting scores. Value,
The face orientation estimation method according to claim 7.

Images