JP4942197B2 - Template creation apparatus, facial expression recognition apparatus and method, program, and recording medium - Google Patents

Description

  The present invention relates to a template creation device, a facial expression recognition device, a method, a program, and a recording medium according to a human facial expression recognition technique based on image processing.

  Human facial expression recognition technology is one of the important elemental technologies of human-machine interaction, and it is expected to realize a method that can recognize facial expressions without limiting the face orientation. On the other hand, most of the human facial expression recognition methods that have been developed so far are limited to the front face and cannot cope with changes in the orientation of the face. In addition, although some methods can cope with changes in face orientation, a detailed three-dimensional face shape model is required for each target person, and there is a problem that it takes time for a prior registration procedure.

  For example, Non-Patent Document 1 proposes a method for estimating the head posture and facial expression in real time. However, since the three-dimensional coordinates of the feature points such as the eye to be tracked and the end points of the mouth are calculated by stereo vision, there is a problem that it cannot be applied to an existing video image taken with a single eye.

Non-Patent Document 2 proposes another method for estimating the head posture and facial expression in real time, but the average value for a plurality of three-dimensional coordinates of feature points such as the eye and mouth end points to be tracked, It is necessary to measure in advance using stereo vision or a range scanner. Furthermore, there is a problem that face shapes and facial expression changes between individuals with large fluctuations cannot be accurately approximated.
SB Gokturk, C. Tomasi, B. Girod and J.-Y.Bouguet: "Model-based face tracking for view-independent facial expression recognition", Proc. Of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 287-293 (2002). F. Dornaika and F. Davoine: "Simultaneous facial action tracking and expression recognition using a particle filter", Proc. Of the Tenth IEEE International Conference on Computer Vision, 2, pp. 1733-1738 (2005).

  When an expression is to be recognized without limiting the face direction, it is necessary to define the image feature amount in a form that is not affected by changes in the face direction. When the three-dimensional shape of the face is given in advance, the influence of the face direction can be removed by normalizing the image using the face shape. However, preparing an accurate face shape model for each person is expensive and problematic in practice. Therefore, it is desired to develop a facial expression recognition method using only a rough geometric model.

In order to solve the above-mentioned problem, the invention described in claim 1 includes a shape model creation unit that creates a shape model that approximates the shape of a face based on an input face image, and a shape model created by the shape model creation unit. Then, by extracting the attention point based on the edge of each face part, the attention point extraction unit for extracting the attention point group composed of a plurality of attention points and the attention point extraction unit for the plurality of facial expressions are extracted. And a luminance distribution model creating unit that creates a template representing a luminance change model of the target point group.

  In order to solve the above-mentioned problem, the invention described in claim 1 includes a shape model creation unit that creates a shape model that approximates the shape of a face based on an input face image, and a shape model created by the shape model creation unit. A point-of-interest extraction unit that extracts a point-of-interest group consisting of a plurality of points of interest that satisfy a predetermined condition in FIG. 5 and a template that represents a brightness change model of the point-of-interest group extracted by the point of interest extraction unit for a plurality of facial expressions And a luminance distribution model creation unit for creating a template.

  The invention according to claim 2 is characterized in that the attention point extraction unit extracts the attention point on the shape model created based on the front image of the face input by the shape model creation unit. .

  According to a third aspect of the present invention, the brightness distribution model creation unit creates a template representing a brightness change model of a point of interest group extracted on a shape model created based on the front image of the face. It is characterized by being.

  According to a fourth aspect of the present invention, a shape model creation unit that creates a shape model that approximates the shape of a face based on an input face image, and a plurality of conditions that satisfy a predetermined condition on the shape model created by the shape model creation unit A point-of-interest extraction unit that extracts a point-of-interest group consisting of a plurality of points of interest, and a luminance distribution model creation that creates a template that represents a model of the luminance change of the point-of-interest group extracted by the point-of-interest extraction unit A template creation unit having a part, a prediction part for predicting a state distribution of a head posture and a facial expression based on an input face image, and an update for evaluating the predicted state distribution using a template created by the template creation part A facial expression and head posture estimation unit, and an estimated value calculation unit that calculates an estimated value of the head posture and facial expression from the evaluated state distribution

The invention according to claim 5 is a shape model creation process for creating a shape model that approximates the shape of a face based on an input face image, and an edge of each face part on the shape model created in the shape model creation process. The attention point extraction process that extracts the attention point group consisting of a plurality of attention points by extracting the attention points based on the target, and the luminance change of the attention point group extracted in the attention point extraction process for a plurality of facial expressions And a luminance distribution model creation process for creating a template representing the model of the template.

  According to a sixth aspect of the present invention, a shape model creation process for creating a shape model that approximates the shape of a face based on an input face image, and a plurality of conditions that satisfy a predetermined condition on the shape model created in the shape model creation process A point of interest extraction process that extracts a group of points of interest consisting of a plurality of points of interest, and a luminance distribution model that creates a template that represents the brightness change model of the points of interest extracted in the point of interest extraction process for multiple facial expressions Process, prediction process for predicting the state distribution of head posture and facial expression based on the input face image, update process for evaluating the predicted state distribution using the template created in the template creation process, and the evaluated state It is an expression recognition method characterized by having an estimated value calculation process of calculating an estimated value of a head posture and an expression from a distribution.

  The invention described in claim 7 is a program executed by the apparatus according to any one of claims 1 to 4 using a computer.

  The invention according to claim 8 is a computer-readable recording in which the apparatus according to any one of claims 1 to 4 is configured using a computer, and records a program executed in the computer. It is a medium.

  According to the present invention, a shape model that approximates the shape of a face is created based on an input face image, and a point of interest group that includes a plurality of points of interest that satisfy a predetermined condition on the created shape model is extracted. In addition, a template that represents the brightness change model of the point of interest group extracted for multiple facial expressions was created. The facial expression can be accurately recognized even when a large posture change is involved. In particular, there is a merit that a precise face shape is not required unlike other methods, and a model for each person can be easily registered.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an overall configuration of a facial expression recognition device 1 according to the present embodiment. In FIG. 1, in the input unit 2, a moving image of the face of a subject photographed using a CCD (charge coupled device) camera or the like as photographing means is given to the facial expression recognition device 1. That is, the input unit 2 samples a captured moving image at a predetermined sampling period to convert it into data, and supplies a data string including data at each time to the facial expression recognition device 1. Then, an estimated value 3 of the head posture and facial expression estimated in real time by the facial expression recognition device 1 is output from the output unit 4.

  As shown in FIG. 1, the facial expression recognition device 1 includes a template creation unit 12 that creates a template 11 in advance, and a head posture and facial expression estimation unit 13 that simultaneously estimates the head posture and facial expression of a person in a moving image. Become. These functions are realized by software processing.

  The template creation unit 12 extracts a point of interest group composed of a plurality of points of interest that satisfy a predetermined condition on the shape model created by the shape model creation unit 121 and the shape model created by the shape model creation unit 121. And a luminance distribution model creation unit 123 that prepares a template representing a distribution model of the luminance change of each attention point in each facial expression of the user. The attention point extracted by the attention point extraction unit 122 is used for estimation of the head posture and facial expression.

  The head posture and facial expression estimation unit 13 predicts the head posture and facial expression state distribution based on the input face image based on the particle filter framework to determine the head posture and facial expression state at the current time. By predicting the prediction unit 131 and evaluating the predicted state distribution using the template 11 created by the template creation unit 12 (evaluating the likelihood of the head posture and facial expression state and each state of the template) An update unit 132 that performs evaluation on the predicted state, and an estimated value calculation unit 133 that calculates estimated values of the head posture and facial expression based on the predicted state and its evaluation.

  FIG. 2 is a flowchart showing a more specific processing procedure of the template creation unit 12 of the facial expression recognition apparatus 1 shown in FIG. In FIG. 2, input data 124 and 127 are image data input from the input unit 2 in FIG. 1, and output data 125, 126, and 128 are data constituting the template 11 in FIG. Hereinafter, the template creation unit 12 will be described with reference to FIG.

  The shape model creation unit 121 creates a shape model 125 that can display parametric variables that approximate the face shape based on the face image 124 of the front and no expression. In the present embodiment, the creation is easy with only one parameter, and a cylinder closer to the actual face shape than the plane is used as the shape model. The radius of the cylinder is calculated based on a single facial image captured with the user facing no expression and the face facing the camera. For example, PA Viola and MJ Jones: “Rapid object detection using a boosted cascade of simple features , "Proc. Of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 511-518 (2001)). The value multiplied by a constant.

  In this embodiment, a cylinder is used as the shape model. However, in the framework of the present invention, a shape model capable of displaying any parameter such as a plane or an ellipsoid can be used. In addition, instead of creating a shape model on the spot by the shape model creation unit 121, any shape such as 3D data of an actual user's face prepared in advance or an average face shape of a plurality of people is used as a shape model. It is also possible to use it.

  The attention point extraction unit 122 extracts a plurality of attention points that satisfy predetermined conditions used for estimating the head posture and expression in the frontal and expressionless face image 124 used in the shape model 125, and calculates the image coordinates thereof. Remember.

  In the luminance distribution model creation unit 123, a luminance distribution model 128 is created for a plurality of face images 127 of the front and each expression. The facial expression luminance distribution model 128 is a model of how the luminance of each point of interest (61a, 62a, 61b, 62b, etc.) changes with each facial expression as shown in FIG. In the present embodiment, by preparing this facial expression luminance distribution model 128, the facial expression at that time is estimated from the luminance pattern of the point of interest in the input image by utilizing the fact that the luminance distribution varies depending on each facial expression.

  Here, assuming the independence of the luminance of each point of interest, each luminance is expressed by a normal distribution, and further, the distribution changes according to the expression. The reason why the luminance is expressed by a normal distribution is based on the idea that the luminance has a spread due to complex factors such as a shape model error and imaging system noise. However, the expression of luminance is not limited to the normal distribution, but can be performed using another probability distribution having a single peak (for example, a Cauchy distribution).

  FIG. 3 is a model of the same type as that of the shape model 125 in which the brightness of the attention points 61a and 62a on the shape model 125 created based on the front and expressionless facial image 124, and represents a front and angry expression. 7 is a conceptual diagram for explaining how the attention points 61b and 62b having the same coordinates on the shape model 7 change. FIG. In this case, the shape model 125 includes a face part 51a corresponding to the right eyebrow, a face part 52a corresponding to the left eyebrow, a face part 53a corresponding to the right eye, a face part 54a corresponding to the left eye, and a face part 55a corresponding to the mouth. It is assumed that each area is associated with a position as shown in the figure. The attention points 61a and 62a are set outside (attention point 61a) and inside (attention point 62a) of the face part 52a of the left eyebrow. In this case, as shown in the lower part of FIG. 3, the luminance of the attention point and the probability thereof are such that the luminance of the attention point 62a located inside the left eyebrow is lower than the luminance of the attention point 61a located outside the left eyebrow. The distribution is a value.

  On the other hand, when the facial expression changes from expressionless to angry, the attention point 61b of the same coordinate as the attention point 61a on the front and angry facial expression shape model 7 and the attention point 62b of the same coordinate as the attention point 62a are both Since it is located above the face part 52b corresponding to the left eyebrow, the luminance at the attention point 61b and the attention point 62b has a close distribution with portions overlapping each other.

  In addition to the face part 52b corresponding to the left eyebrow, the face model 51b corresponding to the right eyebrow, the face part 53b corresponding to the right eye, and the face part 54b corresponding to the left eye are included in the shape model 7 of the front and angry expression. It is also assumed that the face part 55b corresponding to the mouth is associated with a position as shown in the figure.

  The luminance distribution model 128 created by the luminance distribution model creation unit 123 in FIG. 2 is a state in which the user has been photographed by the shape model creation unit 121 and the attention point extraction unit 122 described above (a state in which the face faces the camera). The facial images are acquired from a plurality of facial images 127 in which each facial expression is displayed in order with the head still. This is because the position of each point of interest is fixed when the head is stationary, so that the luminance of each point of interest in each facial expression can be easily obtained from the luminance of those pixels. by. Here, for example, the luminance of each attention point in each facial expression obtained by the above-described method is used as the average of the respective normal distributions, and the calibration process is performed so that the standard deviation is 1/6 of the average.

FIG. 4 is a flowchart showing a more specific processing procedure of the attention point extraction unit 122 shown in FIG. Hereinafter, the attention point extraction unit 122 will be described with reference to FIG. First, a Laplacian-Gaussian filter is applied to the face image (step S11), and a zero-crossing pixel in the image is detected as an edge (step S12). Next, in the face area of the edge image, a set of point pairs as shown in FIG. 5 is extracted as a set of attention points (steps S13 to S15). That is,
First, all point pairs that straddle the edge of each facial part region and that have the center on the edge are set as target point pair candidates (step S13). Next, the point pair candidates are rearranged in descending order of the luminance difference between the points in the point pair (step S14). Then, in order from the point pair candidate having the largest luminance difference, selection is made if it is a prescribed distance away from the center of the already selected point pair (step S15). In this way, a set of point pairs (such as 10-1) as shown in FIG. 5 is extracted as a point of interest set (point of interest group).

  FIG. 5 is a diagram showing the face region 8 in the front / expressionless input image 124 detected as an edge in step S12. The face area 8 in FIG. 5 includes a face part area 81 corresponding to the right eyebrow, a face part area 82 corresponding to the left eyebrow, a face part area 83 corresponding to the right eye, and a face part area 84 corresponding to the left eye. There are a face part region 85 corresponding to the nose and a face part region 86 corresponding to the mouth. Further, a plurality of extracted attention points 9, 9,... Are shown on the face area 8. These attention points 9, 9,... Are point pairs (for example, attention point 9) consisting of a pair of attention points 9, 9 associated with each other so as to straddle the ends (edges) of the regions 81 to 85 of each face part. -1 and a point of interest 9-2) 10-1)).

  In the present embodiment, such a set of attention points 9, 9,... Is a certain number of points in order from the point pair candidates satisfying the following conditions in descending order of the luminance difference between the two points. It is automatically extracted by selecting a pair. The conditions are the following three.

  (C1) Crosses the edge. (C2) The center between the points of interest that make up the point pair (the center of the point pair) is on the edge of the face part. (C3) The center of the point pair is more than the specified distance from the centers of all the already selected point pairs.

  FIG. 5 shows an example of the arrangement of attention points 9, 9,... Extracted by the above method.

  In the present embodiment, the edge detection method is a pixel that crosses zero in the Laplacian-Gaussian filter image, but other edge detection methods such as the use of a Sobel filter may be used. Further, it is not always necessary to configure the attention point set as a set of point pairs centered on the edge, and it is divided by the edge of the face part region 85 corresponding to the left eyebrow in FIG. 5 as shown in FIG. 6B. What is necessary is just to define so that the attention points 9, 9,.

  6 shows an example (FIG. 6A) in which an attention point set created by selecting a plurality of attention points is composed of point pairs 10, 10,... Satisfying the above conditions, and an edge of a face part. FIG. 6B shows an example (FIG. 6B) configured from a plurality of attention points 9, 9,...

  Next, specific processing of the head posture and facial expression estimation unit 13 in FIG. 1 will be described. In the present embodiment, it is assumed that the head posture and facial expression at each time are conditionally independent when a face image at that time is given, and both are random variables that follow a first-order Markov process. If the relationship between the head posture h, the facial expression e, and the face image z at this time is expressed by a dynamic Bayesian network, it is as shown in FIG.

  In the present embodiment, the simultaneous posterior probability distribution of the head posture and facial expression at time t is estimated using all the face images obtained up to time t. This can be written as:

Here, z _t, face image in h _t and e _t each time t, head posture and facial _{expression, P (h t, e t} | z 1: t) all face image up to the time 1~t Simultaneous posterior probability distribution of head posture and facial expression at time t with z _{1: t} given, α = 1 / P (z _t ) is a probability normalization constant, P (h ₀ ) and P (E ₀ ) is a prior distribution of head posture and facial expression, respectively. Note that the right side of equation (1) is obtained by using a Bayes rule and conditional independence on the left side and performing marginalization.

  The head posture is the head posture angle corresponding to the position on the input image of the center of the face shape model, the stroke (pitch), the head swing (yaw), the tilt (roll) based on the face direction facing the camera, In addition, the expression is expressed by a number assigned to each expression with six continuous variables of scale.

In this embodiment, the head posture state transition model P (h _t | h _t1 ) is a random walk model in which each variable is independent. For the facial expression state transition model P (e _t | e _t1 ), the transition between all facial expressions is assumed to have the same probability. Note that these state transition models may be appropriate models depending on the scene where the facial expression recognition device 1 is applied.

  Since the variables related to head posture are continuous variables, and the posterior distribution becomes complicated due to shielding, etc., equation (1) cannot be solved exactly. Therefore, in the present embodiment, a head filter and a facial expression are simultaneously estimated using a particle filter which is one of methods for estimating such a probability density distribution. In the following, a method for simultaneously estimating the head posture and facial expression in the particle filter framework will be described.

  In the particle filter framework, the probability distribution of the head posture and the expression state is approximately expressed by a sample set called particles. Each particle holds one state to be estimated and its weight. In the present embodiment, the state held by each particle corresponds to the state of head posture (continuous variable) and facial expression (discrete variable). These states correspond to the state of the template 11.

  The state estimation of the template 11 using the particle filter is performed as shown in FIG. 8 by the expression / head posture / expression estimation unit 13 of FIG. That is, in the head posture and facial expression estimation unit 13, the prediction unit 131 predicts the posterior probability distribution of the state of the template 11 at that time, and the update unit 132 updates the posterior probability distribution of the state based on the observation information. . Then, the estimated value of the posterior probability distribution obtained by the estimated value calculation unit 133 is calculated. These processes will be described below with reference to FIG. Note that the observation information here is, for example, a posterior probability distribution of the state of the template 11 at that time predicted by the prediction unit 131.

  The prediction unit 131 predicts a particle group that approximates the head posture and facial expression at the current time (step S21). Specifically, sampling is first performed according to the weight of each particle from a particle group that approximates the state at the previous time t1. Next, for each sampled particle, the state of the held head posture and expression is changed according to the state transition model, so that the state of the particle at the current time t is predicted.

  Subsequently, the updating unit 132 calculates the likelihood of the image in the face state for each particle whose state is predicted by the prediction unit 131, and weights each particle in proportion to the likelihood (step S22). .

  In the estimated value calculation unit 133, the estimated value and estimated facial expression of the head posture at each time are calculated as facial expressions that maximize the expected value and the posterior probability, respectively (step S23). This is calculated by the following equation.

Here, h _t ^ and h _t ^(l) are the estimated values of the head pose, the head pose held by the l-th particle, e _t ^ is the estimated facial expression, and ω _t ^(l) is l (English letter L) represents the weight held by the particle.

  Hereinafter, specific processing in the updating unit 132 will be described with reference to FIG. First, it is determined whether or not each point of interest is self-shielded in the head posture state of each particle (step S31). When a convex body such as a cylinder is used as the shape model as in the present embodiment, this determination is performed by projecting each point of interest of the shape model, that is, a point where the normal vector faces the camera direction. A point where the camera direction component of the normal vector at the position is positive can be regarded as an unshielded point. The following processing in the update unit is only targeted for attention points that are not self-shielded.

  Subsequently, the position in the input image in the head posture state of each particle is calculated (step S32).

  Subsequently, the luminance at the position of the target point in the obtained input image is set as the luminance of the target point (step S33).

  Subsequently, the likelihood of the model of the image is calculated using the luminance of each target point obtained by the above method (step S34). In the present embodiment, it is assumed that the luminance of each attention point is independent, and the likelihood of each face image is calculated as the product of the likelihood of the luminance distribution model at each attention point with respect to the luminance distribution model. Here, it is assumed that the luminance of each attention point follows an independent normal distribution. Note that these normal distributions change according to facial expressions as described in the description of the template creation unit 12. At that time, the average and standard deviation of the normal distribution in each facial expression are values obtained by the above-described calibration processing by the template creation unit 12.

  Then, the likelihood of the face image in the head posture and facial expression is calculated as the product of the luminance at each point with respect to the luminance distribution model. Furthermore, the brightness of each attention point is assumed to be an outlier with a constant probability regardless of the brightness, and the one that has a uniform probability that is an outlier and a value with a large likelihood of the normal distribution is the value of that attention point. Likelihood.

  The calculation of the likelihood of the face image in the above head posture and facial expression is calculated by the following equation.

Here, P (z _t | h _t , e _t ) is the likelihood of the image z _t in the head posture h _t and the expression e _t at time t, and P (z _{i, t} | h _t , e _t ) Is the likelihood of the luminance z _{i, t} of the point of interest i in the head posture and expression at time t, P is the set of points of interest that are not self-occluded among the set of all points of interest, μ _i ( e _t ) and σ _i (e _t ) are the mean and standard deviation of the luminance distribution model in the facial expression e _t of the point of interest i. Further, P _outlier is a probability that the luminance at the attention point is an _outlier , and is assumed to be a constant value regardless of the luminance value. Specifically, the value of P _Ni when z _{i, t} = μ _i (e _t ) ± 3σ _i (e _t ) in equation (6) is P _outlier .

  In addition to the definition of the likelihood of the image, for each attention point, there is the inverse of the average of the sum of all the attention points of the robust function value with the Mahalanobis distance as an argument for the model of luminance at the calculated position. Conceivable.

  The likelihood calculation formula in this case is calculated by the following formula.

Here, N _P is the number of target points that are not self-shielding, [rho (·) is robust function, c is representative of the constants (e.g. c = 1).

  Finally, the weight of each particle is calculated as being proportional to the likelihood of the face image with respect to the head posture and facial expression of each particle calculated by the method described above, and the sum of these weights is 1. (Step S35).

  In the following, the image coordinate calculation process (step S32) of each target point in FIG. 9 will be described with reference to FIGS. First, the image coordinates of each attention point are calculated by the following method. The shape model coordinate system 201, which is the three-dimensional coordinate of the shape model 125 (see FIG. 2), has a center and a vertical axis that coincide with the face center on the image plane of the face image 124 used for point of interest extraction and the vertical axis of the image. To be defined. Then, by projecting each attention point such as the attention point 9a on the face image 124 to the shape model coordinate system 201, the shape model coordinates of each attention point such as the attention point 9b corresponding to the attention point 9a are obtained (step S41).

  Next, each attention including the attention point 9c corresponding to the attention point 9b on the shape model 125a deformed by performing two-dimensional translation, three-dimensional rotation and scaling according to the head posture state of each particle of the shape model 125 The image coordinates of the point are obtained (step S42).

  Then, each attention point such as the attention point 9c on the processed shape model 125a is projected onto the input image 124a on the same plane as the input face image 124 (step S43). Here, an image of each attention point such as the attention point 9d corresponding to the attention point 9c (that is, the attention point corresponding to the attention points 9b and 9a) is calculated. With the above processing, the image coordinates of each target point in step S32 in FIG. 9 are calculated.

  In this embodiment, a weak center projection is assumed in these projections, but any projection can be used.

  According to the present embodiment, the following remarkable effects can be expected.

  (1) It is possible to create a face shape model and a face brightness change model with respect to facial expressions that vary greatly between individuals and depending on the lighting environment by simple operations on the spot.

  (2) The user's facial expression that changes the head posture over a wide range can be recognized in real time and with high accuracy.

  The embodiment of the present invention is not limited to the above. For example, the blocks shown in FIG. 1 are integrated, divided, or further distributed and arranged via a communication line or the like. Is possible. The embodiment of the present invention can be configured by a computer and a program executed by the computer. Further, the entire program or a program constituting a part of the program, for example, a part related to the template creation unit 12 in FIG. 1, can be distributed through a communication line or a computer-readable recording medium. is there.

  Moreover, the facial expression recognition apparatus according to the present invention can be considered as having the following characteristics or can be modified by the above embodiment or by modifying a part thereof.

  (1) A facial expression recognition apparatus that estimates the simultaneous posterior probability distribution of the head posture state and the facial expression state at the current time using all face images obtained from the frame where the estimation is started to the current time. (2) A template for changing the luminance of the point of interest in the facial image accompanying the facial expression change is prepared in advance, and the template is changed according to the facial expression of the user at each time in the moving image. 2. The facial expression recognition device according to claim 1, wherein the facial expression and head posture of the person in the face image are estimated simultaneously from the luminance of the attention point. (3) In addition to changes in facial expression, a model of luminance distribution change due to a change in head posture that occurs in a general lighting environment is prepared for each facial expression, and this is changed according to the user's head posture and facial expression. A facial expression recognition device that simultaneously estimates the facial expression and head posture of a person in the face image from the luminance of the point of interest at each time by changing.

  (4) A facial expression recognition apparatus that prepares a luminance distribution model only for a swing motion that causes a large variation in illumination and a large amount of illumination. (5) A facial expression recognition device that prepares a template as a distribution of luminance caused by complex factors such as imaging system noise and an error with respect to an actual face shape of a template shape model as a distribution instead of a luminance value. (6) A facial expression recognition device using a shape that can be easily created as a template shape model, such as a planar model, a cylindrical model, or an ellipsoid model, that can be easily created. (7) A facial expression recognition apparatus using an actual three-dimensional shape measured by a laser scanner or stereo vision as a template shape model.

  (8) A facial expression recognition apparatus that uses only points defined discretely in a face area as a template. (9) A facial expression recognition apparatus that uses a set of discrete attention points arranged in the vicinity of an edge in a face region. (10) A facial expression recognition device that defines a set of point pairs straddling edges as a discrete point set.

  (11) The likelihood of the image at that time in a certain head posture and facial expression at each time is expressed as a likelihood of the model (luminance distribution model) expressed by the normal distribution of the luminance at the position of each calculated point of interest. A facial expression recognition device defined as a product. (12) A facial expression that defines the likelihood of a face image in the head posture and facial expression as the reciprocal of the sum of all the points of interest of the robust function value using the Mahalanobis distance as an argument for the model of luminance at the position of each point of interest. Recognition device.

  (13) A facial expression recognition apparatus that uses an expected value, a mode value (mode), or a median value (median) of the posterior probability density distribution as an estimated value of the head posture at each time. (14) A facial expression recognition apparatus that uses the facial expression having the maximum calculated posterior probability as an estimated facial expression. (15) A facial expression recognition device that estimates a simultaneous posterior probability distribution of a head posture state and a facial expression state at each time using a particle filter framework.

It is a block diagram which shows the structure of embodiment of the facial expression recognition apparatus of this invention. It is a flowchart for demonstrating the process by the template preparation part 12 of FIG. It is a conceptual diagram for demonstrating the luminance distribution model 128 of FIG. 3 is a flowchart for explaining processing by a point of interest extraction unit 122 in FIG. 1. FIG. 7 is a diagram illustrating an example of attention point arrangement for explaining processing by the attention point extraction unit 122 of FIG. 1. FIG. 10 is an explanatory diagram for explaining a modification of processing by the attention point extraction unit 122 of FIG. 1. It is a figure for demonstrating the process by the head posture and facial expression estimation part 13 of FIG. 1, Comprising: It is a figure which shows the dynamic Bayesian network which consists of a head posture, a facial expression, and a face image. 3 is a flowchart for explaining processing by a head posture and expression estimation unit 13 in FIG. 1. 3 is a flowchart for explaining processing by an updating unit 132 in FIG. 1. FIG. 10 is a flowchart for explaining a process of step S32 of FIG. 9. FIG. It is a conceptual diagram for demonstrating the process of FIG.

Explanation of symbols

1 Facial expression recognition device
11 templates
12 Template creation part
13 Head posture and facial expression estimation unit
9, 9-1, 9-2, 9a-9d, 61a, 62a, 61b, 62b
10, 10-1 point pair (point of interest pair)
125 shape model
128 luminance distribution model

Claims (8)

A shape model creation unit for creating a shape model that approximates the shape of the face based on the input face image;
On the shape model created by the shape model creation unit, by extracting a point of interest based on the edge of each facial part, a point of interest extraction unit that extracts a point of interest consisting of a plurality of points of interest;
A template creation apparatus comprising: a brightness distribution model creation unit that creates a template representing a brightness change model of a group of attention points extracted by a point of interest extraction unit for a plurality of facial expressions. 2. The template creation according to claim 1, wherein the attention point extraction unit extracts a attention point on a shape model created based on a front image of a face input by the shape model creation unit. apparatus. The brightness distribution model creation unit creates a template representing a brightness change model of a point of interest group extracted on a shape model created based on a front image of the face. 2. The template creation device according to 2. A shape model creation unit for creating a shape model that approximates the shape of the face based on the input face image;
A point-of-interest extraction unit that extracts a point-of-interest group consisting of a plurality of points of interest that satisfy a predetermined condition on the shape model created by the shape model creation unit;
A template creating unit having a brightness distribution model creating unit that creates a template representing a brightness change model of a group of points of interest extracted by the point of interest extraction unit for a plurality of facial expressions;
A prediction unit that predicts a head posture and a state distribution of facial expressions based on the input face image;
An update unit that evaluates the predicted state distribution using a template created by the template creation unit;
A facial expression recognition apparatus comprising: an facial expression and head posture estimation unit having an estimated value calculation unit that calculates an estimated value of a head posture and facial expression from the evaluated state distribution. A shape model creation process for creating a shape model that approximates the shape of the face based on the input face image;
On the shape model created in the shape model creation process, by extracting the point of interest based on the edge of each face part, the point of interest extraction process of extracting a point of interest consisting of a plurality of points of interest;
A template creation method, comprising: a brightness distribution model creating process for creating a template representing a brightness change model of a group of points of interest extracted in the point of interest extraction process for a plurality of facial expressions. A shape model creation process for creating a shape model that approximates the shape of the face based on the input face image;
A point of interest extraction process that extracts a point of interest consisting of a plurality of points of interest that satisfy a predetermined condition on the shape model created in the shape model creation process;
A luminance distribution model creation process for creating a template representing a luminance change model of the target point group extracted in the target point extraction process for a plurality of facial expressions;
A prediction process for predicting the head posture and facial expression state distribution based on the input face image;
An update process for evaluating the predicted state distribution using the template created in the template creation process,
An estimated value calculating process for calculating an estimated value of the head posture and expression from the evaluated state distribution.   The apparatus of any one of Claims 1-4 is comprised using a computer, Comprising: The program run in the computer.   The apparatus according to any one of claims 1 to 4, wherein the apparatus is configured using a computer, and is a computer-readable recording medium on which a program executed on the computer is recorded.

Images