KR101783453B1 - Method and Apparatus for extracting information of facial movement based on Action Unit - Google Patents

Abstract

An AU-based facial motion information extraction method will be described. Extraction method: defining a standard model in which a plurality of vertices corresponding to specific parts of the human face are arranged; Obtaining a facial image from the subject; Mapping (masking) the standard model to the facial image and mapping the vertices according to a portion of the facial image; Extracting a coordinate change value of the vertexes while continuously acquiring a face image obtained from the subject; And grouping the vertices into AUs (Action Units) defined for the face, and extracting AU facial motion information including a change value of a centroid of vertices of each AU group.

Description

[0001] The present invention relates to an AU-based facial motion information extracting method and apparatus,

The present invention relates to a method for extracting human motion information and an apparatus using the same, and more particularly, to a method and apparatus for extracting facial motion information based on an AU (Action Unit).

A variety of human motion information extraction methods can be found in many documents. Human motion information can be used to evaluate the subject's physiological condition.

It is desirable to study such human motion information so that it can be used not only for evaluation of physiological condition but also for evaluation of psychological condition.

The AU defines the movement and direction of the facial region, and the study of extracting the facial motion information of AU is desirable for expanding various applications in the future.

US 6834115 B2 US 6904347 B1 US 7352881 B2 EP 0735509 B1 WO 2002009038 A2 WO 2003017186 A1

The present invention provides a method for quantifying and extracting facial motion information of a specific AU and an apparatus for applying the method.

The present invention provides a method of accurately extracting facial motion information of an AU, quantifying or quantifying the facial motion information, and an apparatus for applying the method.

An Action Unit (AU) facial motion information extraction method according to the present invention:

Defining a standard model in which a plurality of vertices corresponding to specific parts of the face of the human body are arranged;

Obtaining a facial image from the subject;

Mapping (masking) the standard model to the facial image and mapping the vertices according to a portion of the facial image;

Extracting a coordinate change value of the vertexes while continuously acquiring a face image obtained from the subject; And

Grouping the vertices by AUs defined for the face, and extracting AU facial motion information including a change value of a centroid of vertices of each AU group.

)) 값 중 적어도 어느 하나를 더 포함할 수 있다.According to a specific embodiment of the present invention, the AU facial motion information includes at least one of a facial motion size, an area size, a facial motion angle of a polar concept,

) ≪ / RTI > value).

According to the present invention, the vertex includes X, Y coordinates in the X-Y plane in the direction parallel to the average plane of the subject's face, and Z coordinates perpendicular to the X-Y plane.

According to a specific embodiment of the present invention, the facial motion angle r * Sin ([theta]) of the polar coordinate concept can be obtained from the change of the Z coordinate perpendicular to the X-Y plane.

According to an embodiment of the present invention, the AU includes at least three vertices.

According to the embodiment of the present invention, the AU facial motion information can be extracted from the motion (change) in at least one of the X-Y plane, the X-Z plane and the Y-Z plane in the X, Y and Z coordinates.

The present invention proposes a method for extracting facial expression changes as quantified data for each AU. The data extracted by the present invention shows a characteristic change according to AU, and can be applied to various fields by using it.

1 is a schematic block diagram of an AU-based motion information extraction system according to the present invention.
2 is a flowchart of an AU-based facial motion information extraction method according to the present invention.
Figure 3 shows an example of a standard model applied to the method of the present invention.
FIG. 4 illustrates masking (mapping) the standard model to a facial image acquired from a subject.
Figure 5 shows the result of vertex mapping to the subject image using the standard model.
FIG. 6 illustrates vertex information around the eye, with the left side representing the left eye and the right side representing the right eye.
Figures 7 and 8 illustrate the movement in the AU6 where the ball (cadaver) is raised, the lower eyelid is pressed and the wrinkles of the eye are produced in the method according to the present invention.

Hereinafter, an embodiment of an AU-based facial motion information extraction method according to the present invention will be described with reference to the accompanying drawings.

The facial motion information extracting method according to the present invention includes a computer system 1 including an input / output device such as a keyboard 14, a mouse 15, a monitor 12, and a main body 11 to which the input / ), And may be implemented by a dedicated system in accordance with an embodiment of the present invention. Such a method of the present invention is not limited in its technical scope by a specific hardware system.

An apparatus or system for performing the method of extracting facial motion information according to the present invention may include a moving picture camera 13, for example, a so-called web cam, for photographing the face of the subject 20. In addition, such a system processes an image obtained from a subject through an analysis algorithm provided in software form, extracts facial motion information for each AU using the extracted image, and outputs the result to a database in the computer system 1 and a monitor 12 Lt; / RTI >

2 is a flowchart of an AU-based facial motion information extraction method according to the present invention.

First step S21: photographing the face of the subject in real time and inputting the moving image data frame by frame. In this step, a general pre-processing process such as amplifying a continuous input image and removing noise included therein may be included.

Second Step (S22): Mask the standard model preset in the inputted face image. This process is performed once at the first time, and masking is performed again if necessary when a facial image is missed during future facial tracking.

In the standard model, a plurality of vertices corresponding to various specific parts of the face on the basis of a reference point of the human body surface are appropriately arranged corresponding to the face standard shape or shape.

The reference point is, for example, a central part of the hair, and a vertex for specific parts of the face is determined based on the reference point. In the mapping process, the vertices are mapped to the face region of the subject by the portion of the face image based on the standard model. At this time, the comparison region of the subject's face region with respect to the standard model is the face contour, eye, nose, mouth, and the like.

Step 3 (S23): The coordinate change value of the vertices is extracted from the mapped facial image. In extracting the change value at this time, the reference values are compared with the peak values obtained through the masking in the second step.

Step 4 (S24): Map or classify (define) the mapped vertices according to AUs (Action Units).

Step S25: The quantified AU information for each AU including the center point or the centroid of each AU is extracted using the vertices grouped in units of AUs and stored as current AU information, At this time, if there is the AU information stored before that, it becomes the preceding AU information.

Step S6: In this step, it is determined whether or not the preceding AU information exists, and if there is no preceding AU information, the process returns to the first step S21 described above. On the other hand, if there is the preceding AU information, the process proceeds to the seventh step S27.

Step 7 (S27): In this step, the current AU information is compared with the preceding AU information to obtain a change amount of a specific component in the current AU information. At this stage, the specific component to be detected is the center point information per AU, the magnitude of the center point change, the change amount of the area formed by the vertices per AU, the angle of motion of the polar coordinates, and the curve movement r * SIN And either one of them is selectively extracted.

The amount of change in the extracted AU information can be applied to a variety of application fields, for example, to a system for estimating a change in facial expression or emotion of a subject.

Hereinafter, the steps described above will be described in more detail.

3 shows an example of a standard model. In the standard model, the face of the human face with average shape and size is modeled as a network. Each node of the network structure, ie, a vertex, is a part sensitive to facial muscle movement. AUs that define change can be grouped separately. In FIG. 3, a vertex 136 of the forehead is a reference point for extracting facial expression change amount, and vertices related to eyes, eyebrows, nose, mouth, tail, cheekbones, and jaw line are linked.

FIG. 4 illustrates masking (mapping) the standard model to a facial image acquired from a subject, and FIG. 5 shows a result of vertex mapping to a subject image using a standard model. When mapping is performed as shown in FIG. 5, information of all vertices is extracted based on the vertex of the point-to-point 136. Each vertex information includes the values of X, Y, and Z.

FIG. 6 illustrates vertex information around the eye, with the left side representing the left eye and the right side representing the right eye.

Table 1 below shows the result of mapping AU and vertex, which defines the movement of the facial part, which brings about changes in facial expression.

From the obtained facial image, various kinds of information change amount can be extracted from each AU information, especially AU change.

The midpoints (Cx, Cy) and area (Axy) of the AUs in the XY plane from the facial image can be obtained from the following equations 1, 2, and 3 using the coordinate values of the vertices of each AU. In the equation below, N is the total number of vertices per AU, and x and y are the X and Y coordinates respectively, where i is the number of the vertex in the corresponding AU.

(Cy, Cz), (Cx, Cz), and area (Cz) of the AU in the YZ plane and the XZ plane can be obtained by converting the coordinate values and areas in the XY plane into values in the YZ plane or the XZ plane, Ayz, Axz) can be obtained.

The amount of change in AU information required from the current AU information and the preceding AU information can be obtained using the continuous input facial image in the same manner as described above.

As described above, the amount of AU information change includes the magnitude of the facial movement (|| V ||), the angle of facial motion, the amount of change in the center point, and the amount of change in the area of a specific AU. It can be judged.

In each AU, the magnitude || V || of the nth AU (AUn) is obtained from the following equation.

The facial motion size of AU _n is extracted using the following formula (n = AU number).

(X1, y1) of the centroid of AU _{n in} the previous facial image frame (preceding AU information) and (x2, y2) the center coordinates of AU _n in the current facial image frame (current AU information) , The coordinate change amount (V _n ) of the center point is as shown in the following Equation 4.

Therefore, the magnitude (|| V _n ||) of the facial motion vector is calculated as shown in Equation 5 below.

)는 아래와 같이 계산된다. In addition, the center of gravity (radian,

) Is calculated as follows.

는 선행 AU의 벡터

와 현재의 AU의 벡터

내적을 의미한다. 여기에서 두 벡터의 외적을 통해서 회전 방향을 계산할 수 있다.In the above equation

Is the vector of the preceding AU

And the vector of the current AU

It means inner product. Here we can calculate the direction of rotation through the outer product of two vectors.

이면 회전 방향은

는

의 반시계 방향이며,

이면, 회전 방향은

는

의 시계 방향이 된다.In other words,

The direction of rotation

The

Counterclockwise,

, The direction of rotation

The

Clockwise direction.

는 시계방향 또는 반시계 방향으로 움직일 수 있다.Figure 7 illustrates the movement in the AU6 where the ball (cadaver) is raised, the lower eyelid is pressed, and the wrinkles of the eyes are produced. In Fig. 7, a solid line indicates a vector in a frontal facial image, and a dotted line indicates a vector generated in a current facial image. As shown in FIG. 7, according to the operation of the AU 6

Can be moved clockwise or counterclockwise.

In this state, the curvilinear motion (Mr) of the vertex can be calculated according to the motion of the AU6.

In Fig. 8, &thetas; is the movement angle of the center point of AU6.

과 움직임 각도 θ는 아래와 같은 식에 의해 계산된다.Radius here

And the motion angle? Are calculated by the following equations.

The AU-based facial motion information extracted in accordance with the present invention described above, for example, the facial motion size of the AU, the angle, and the facial motion of the apex can be applied in various forms. For example, information of AUs moving according to a facial expression can be extracted and converted into a database, and the information of the facial expression image inputted in real time can be compared with the information in the database to determine the emotion of the current facial expression, have.

As a result, according to the present invention, it becomes possible to develop a technique of recognizing facial expression recognition using facial motion information based on AU. That is, the human facial expression movement is an important factor that can infer the sensibility of the person at present, and the present invention provides a basis for extracting the quantitative change of the AU and applying it.

These and other embodiments of the present invention have been described and shown in the accompanying drawings. However, it should be understood that these embodiments are only a part of various embodiments. Since various other modifications could occur to those of ordinary skill in the art.

1: Computer system
11: Body
12: Monitor
13: Video camera
14: Keyboard
15: Mouse
20: Subjects

Claims (8)

Defining a standard model in which a plurality of vertices corresponding to specific parts of the face of the human body are arranged;
Obtaining a facial image from the subject;
Mapping (masking) the standard model to the facial image and mapping the vertices according to a portion of the facial image;
Extracting a coordinate change value of the vertexes while continuously acquiring a face image obtained from the subject; And
Grouping the vertices into AUs (Action Units) defined for the face, and extracting AU facial motion information including a change value of a centroid of vertices of each AU group, A method for extracting facial motion information. The method according to claim 1,
The AU facial motion information includes at least one of facial motion size according to a change in position of AU midpoints, area size formed by apexes per AU, facial motion angle of a polar concept, and a curve motion (Mr) value of a vertex, Wherein the curvilinear motion (Mr) of the apexes satisfies the following equation.
Mr = r * Sin (?)
Here, r is the radius of the midpoint movement and? Is the midpoint movement angle. 3. The method of claim 2,
Wherein the coordinates of the vertices include X, Y coordinates on the XY plane, and Z coordinates perpendicular to the XY plane. 4. The method according to any one of claims 1 to 3,
Wherein each of the AUs comprises at least three vertices (Vertex). 4. The method according to any one of claims 1 to 3,
Wherein the AU facial motion information is extracted from a motion (change) in at least one of XY plane, XZ plane, and YZ plane in X, Y, Z coordinate system. An AU-based facial motion information extracting apparatus for performing the method according to any one of claims 1 to 3,
A moving picture camera for acquiring a facial image from the subject;
And a computer-based system having an analysis algorithm for processing image data from the moving picture camera and extracting the AU facial motion information using the image data. The method according to claim 6,
Wherein each of the AUs includes at least three vertices (Vertex). The method according to claim 6,
Wherein the AU facial motion information is extracted from a motion (change) in at least one of an XY plane, an XZ plane, and a YZ plane in X, Y, and Z coordinate systems.

Images