KR20140009013A - Method and apparatus for modeling 3d face, method and apparatus for tracking face - Google Patents

Abstract

Disclosed are a method and a device for modeling a three-dimensional (3D) face and a method and a device for tracking a face. The method for modeling the 3D face sets a predetermined standard 3D face as a working model and generates and outputs a tracking result including at least one among face feature points, face parameters, and head pose parameters from a video frame based on the working model. [Reference numerals] (AA) Start; (BB) No; (CC) Yes; (DD) End; (S110) Setting a working model and a designated starting frame; (S120) Tracking a face from the designated starting frame based on the working model; (S130) Updating the working model based on a tracking result; (S140) Meeting an optimal model or last video frame?; (S150) Setting a next designated starting frame; (S160) Outputting the working model

Description

3D face modeling method and apparatus, face tracking method and apparatus {METHOD AND APPARATUS FOR MODELING 3D FACE, METHOD AND APPARATUS FOR TRACKING FACE}

The following description relates to a 3D face modeling method and apparatus, and a face tracking method and apparatus, and more specifically, by simultaneously performing face tracking and 3D face modeling in a continuously input face video frame, The present invention relates to a 3D face modeling method that provides a similar 3D face and outputs facial expression information with high accuracy.

By input of continuous video, conventional face tracking / modeling techniques can output results with varying levels of complexity. For example, the conventional face tracking / modeling technique outputs various results according to the type of facial expression parameter and the intensity of the facial expression, the 2D shape of the face, the low resolution face 3D shape, or the high resolution face 3D shape.

In general, face tracking / modeling techniques can be classified into three types: identification techniques of user faces, fitting techniques, and regeneration techniques for modeling. Some face tracking / modeling techniques use a binocular camera or a depth camera to identify the user's face. The user performs 3D modeling of the face through a process of setting a marked key point, registering a user, and maintaining a fixed facial expression during modeling.

According to an embodiment, a modeling method of a 3D face may include: setting a preset reference 3D face as a work model and performing face tracking on a video frame basis based on the work model; Generating a tracking result comprising at least one of a facial feature point, an expression parameter, and a head pose parameter from the video frame; Updating the work model based on the tracking result; And outputting the updated working model.

According to an embodiment, the method of modeling a 3D face may further include pre-training a reference 3D face through 3D face data offline and setting the trained reference 3D face as a working model.

According to an embodiment, an apparatus for modeling a 3D face performs face tracking on an input video frame based on a working model, and generates a tracking result including at least one of a facial feature point, an expression parameter, and a head pose parameter. Tracking unit; And a modeling unit to update the work model based on the tracking result.

The apparatus for modeling a 3D face according to an embodiment may further include a training unit training a reference 3D face in advance through 3D face data and setting the trained reference 3D face as a work model.

1A is a flowchart illustrating a method of modeling a 3D face, according to an exemplary embodiment.
1B is a flowchart illustrating a step of updating a work model in a 3D face modeling method according to an embodiment.
1C is a flowchart of a face tracking method, according to an exemplary embodiment.
2 is a diagram illustrating an example of generating a 3D face based on a general face.
3 is a diagram illustrating an example of extracting a face sketch from a video frame using feature points.
4 is a diagram illustrating an example of performing feature point matching and sketch matching between a 3D face and a face sketch.
5A and 5B are diagrams illustrating a modeling device / face tracking device according to one embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The specific structural or functional descriptions below are merely illustrative for purposes of illustrating embodiments of the invention and are not to be construed as limiting the scope of the invention to the embodiments described in the text. Like reference symbols in the drawings denote like elements.

The 3D face modeling method and the face tracking method may be performed in a general computer or a dedicated processor. The modeling method of the 3D face sets a preset high-precision reference 3D face as a working model for a continuously input video frame including a face shape (at a predetermined time, for example, within a few minutes), and sets the working model. Face tracking can be performed based on. Next, the 3D face modeling method may perform an update / modification process of the work model on a predetermined number of faces based on the face tracking result. The 3D face modeling method updates / corrects the working model, then acquires a 3D face that reaches a preset threshold or until the face tracking and update / correction of the working model are complete for all input video frames, Face tracking can be continuously performed on the video frame. If necessary, the face tracking result including the precise expression and the head pose information may be output during the update / correction process, or the 3D face generated after the update / correction process is finished may be output.

The continuously input video frames may be a plurality of pictures or video frames taken by a general digital camera and extracted or processed for digital video streaming, or may be a plurality of pictures or video frames taken continuously by a digital camera. Continuously input video frames may be input via a input / output interface to a general computer or dedicated processor for modeling 3D faces and tracking methods.

로 표현할 수 있다. 여기서, S는 3D 형상, a 는 외모 성분, e는 표정 성분, q는 헤드 포즈, T(S, q)는 헤드 포즈q에 따라 3D 형상S를 회전시키는 동작 또는 이동시키는 동작을 수행하는 함수를 나타낼 수 있다.4 shows an example of a 3D face generated based on any face. The 3D face may include, but is not limited to, a 3D shape of the face, an appearance parameter, an expression parameter, and a head pose parameter. The working model of the 3D face,

. Where S is a 3D shape, a is an appearance component, e is a facial expression component, q is a head pose, and T (S, q) is a function for rotating or moving the 3D shape S according to the head pose q. Can be represented.

According to an embodiment of the present disclosure, the reference 3D face may be trained in advance using high-precision facial data having different facial expressions and poses. According to another exemplary embodiment, the reference 3D face may be obtained through a general method, or the 3D face including the features of the reference face may be determined as the reference 3D face as needed.

, 표정 성분

, 및 헤드 포즈 qo를 포함할 수 있다. 평균 형상 So는 모든 트레이닝 샘플의 평균 값을 의미하고. 외모 성분

(i=1:N)에서 각 성분은 얼굴의 외모적 변화를 나타낸다. 표정 성분

(j=1:M)에서 각 성분은 얼굴의 표정 변화를 나타내고, 헤드 포즈 qo는 얼굴의 공간상 위치 및 회전 각도를 나타낸다.Standard 3D face is average shape So, looks components

Facial expression elements

, And head pose qo. Mean shape So means the mean value of all training samples. Appearance

At (i = 1: N) each component represents an appearance change of the face. Facial expression

In (j = 1: M), each component represents a facial expression change, and the head pose qo represents the spatial position and rotation angle of the face.

1A is a flowchart illustrating a method of modeling a 3D face, according to an exemplary embodiment.

In operation S110, the method of modeling a 3D face may set a preset reference 3D face as a work model and set a designated start frame as a first frame. The reference 3D face may be a 3D face that has been previously trained using face data of various expressions and poses.

In step S120, the 3D face modeling method performs face tracking from a designated starting frame of a plurality of video frames continuously input based on the working model, and facial feature points, facial expression parameters, and heads in the tracked video frame. Pose parameters can be extracted. The 3D face modeling method may generate a tracking result corresponding to a preset number of video frames according to a preset condition. The generated tracking result may include facial feature points, facial expression parameters, and head pose parameters extracted from each tracked video frame and each tracked video frame. According to an embodiment, the method of modeling a 3D face may determine the preset number based on input rates, noise characteristics, or tracking accuracy requirements of a plurality of consecutively input video frames, The preset number may be a constant or a variable.

According to an embodiment, the method of modeling the 3D face in step S120 may also output the tracking result generated through the input / output interface.

According to one embodiment, the modeling method of the 3D face in step S120 is an Active Appearance Model (AAM), an Active Shape Model (ASM), and a Composite Constraint AAM. At least one of the face feature point, the facial expression parameter, and the head pose parameter may be obtained.

Next, the method of modeling the 3D face in step S130 may update the work model based on the tracking result generated in step S120. The update of the working model will be described in detail with reference to FIG. 1B below.

According to an embodiment of the present disclosure, when the updating of the work model is completed in operation S130, the updated work model may be output through the input / output interface.

In step S140, the method of modeling the 3D face is a difference between the appearance parameter of the updated working model and the appearance parameter of the working model before the update is not smaller than a preset threshold, and the video frame after the preset number of video frames. If it is not the last video frame among the plurality of consecutively input video frames, the method of modeling the 3D face in step S150 may set the first video frame after the preset number of video frames as the designated start frame. Thereafter, the modeling method of the 3D face may return to step S120 to perform face tracking from the start frame set based on the updated working model.

The difference between the appearance parameters of the updated working model and the appearance parameters of the working model before being updated is smaller than a preset threshold, and the last video frame among the plurality of video frames in which video frames after a preset number of video frames are continuously input. In this case, the modeling method of the 3D face may perform step S150. In detail, the modeling method of the 3D face may terminate the update to the working model when the optimal 3D face is generated or meets the preset condition or the processing of the entire video frame is completed.

In operation S160, the modeling method of the 3D face may output the updated working model as the personalized 3D face.

FIG. 1B is a flowchart showing the process of step S130 shown in FIG. 1A.

를 계산할 수 있다. 여기서, K는 표정 파라미터의 종류의 수를 나타낸다. 3D 얼굴의 모델링 방법은 표정 파라미터 중 가장 높은 빈도로 나타내는 표정 파라미터 값

을 중립 표정 값으로 설정하고; 전체 K개의 표정 파라미터와 중립 표정 값의 편차가 미리 설정된 임계값보다는 작은 비디오 프레임을 중립 표정 프레임으로 선택할 수 있다.Referring to FIG. 1B, in step S132, the 3D face modeling method may select a video frame most similar to a neutral expression in the tracking result generated in step S120 as a neutral expression frame. According to a preferred embodiment, in step S132, a facial expression parameter for each tracked video frame, in order to select a neutral facial expression frame from the tracking result corresponding to the preset number of video frames whose number is T

Can be calculated. Here, K represents the number of types of facial expression parameters. Modeling method of the 3D face is the expression parameter value represented by the highest frequency among the expression parameters.

Set to a neutral facial expression value; A video frame whose deviation of the total K facial expression parameters and the neutral facial expression value is smaller than a predetermined threshold value may be selected as the neutral facial expression frame.

In operation S135, the 3D face modeling method may extract a face sketch from the neutral facial expression frame based on the facial feature points appearing in the neutral facial expression frame. The 3D face modeling method extracts information including facial feature points, facial expression parameters, head pose parameters, and the like for each video frame tracked in step S120, and uses an active contour model algorithm. In step S132, the face sketch may be extracted from the neutral expression frame selected in step S132.

Figures A, B, and C of Figure 3 illustrate an example of extracting a face sketch from a video frame using feature points. According to an embodiment, when a sketch is extracted from the video frame of FIG. A, a face sketch may be extracted from the video frame by referring to a feature point of the video frame (Fig. B) and using an active contour model algorithm. Face sketch shown in C) Through this process, the face sketch may be extracted from the neutral facial expression frame.

Referring back to FIG. 1B, in step S138, the 3D face modeling method may update the working model based on the feature points of the neutral facial expression frame and the extracted face sketch. In detail, the 3D face modeling method may update the head pose q of the work model to the head pose of the neutral facial expression frame and set the facial expression component e of the work model to zero. In addition, the 3D face modeling method matches the work model S (a, e, q) with the feature point position of the neutral facial expression frame, and then neutralizes the face sketch calculated through the work model S (a, e, q). By matching the face sketch extracted from the frame, the appearance component a of the working model can be modified. 4 illustrates a work model generated by adjusting the work model to match the feature points with the feature points of the neutral facial expression frame shown in FIG. 4 shows the steps of adjusting the work model or modifying appearance parameters such that the work model matches the extracted face sketch.

In the method of modeling a 3D face by modifying an appearance component to represent an appearance parameter, the 3D face may be compared in step S140 by recording the numerical value of the external appearance parameter and the numerical value of the external appearance parameter after the modification.

Face tracking and updating may be performed on video frames continuously input in steps S120 to S150 illustrated in FIG. 1A. According to one embodiment, step S120 and step S130 may be performed simultaneously. By performing face tracking using the current working model, extracting facial feature points, facial expression parameters and head pose parameters, and updating the working model using the extracted facial feature points and head pose parameters and corresponding video frames, A face model most similar to the user's face may be obtained. Also, a face tracking result for each input video frame may be output, and the face tracking result may include an expression parameter, an appearance parameter, and a head pose parameter.

1C is a flowchart of a face tracking method, according to an exemplary embodiment.

The face tracking method may have a main purpose of outputting a face tracking result. In FIG. 1C, when the optimal model meeting the preset condition is obtained, the face tracking method may not perform update processing on the working model but may perform face tracking on the next video frame.

Referring to FIG. 1C, when a 3D face modeling method is performed, the face tracking method sets a preset reference 3D face as a working model in step S110 and sets a specified start frame as a first frame. In this case, the variable indicating whether to continue updating the working model (eg, modeling instruction) may be set to 1 (or Yes). The reference 3D face may be a series of facial expressions, a 3D face that has been previously trained in a pose.

Step S120 illustrated in FIG. 1C may be the same as step S120 of FIG. 1A. However, in FIG. 1C, step S125 and step S128 may be performed after completing face tracking for a predetermined number of video frames. In step S125, the face tracking method may output a tracking result for each tracked video frame, and the tracking result may include an expression parameter, an appearance parameter, and a head pose parameter.

In step S128, the face tracking method may determine whether to continue to update the working model. (Determining whether modeling instruction = 1) If the modeling instruction = 1, the face tracking method proceeds to step S130. Can be done. Step S130 may be the same as step S130 of FIG. 1A.

In step S140, when the face tracking method performs an update on the work model, when the difference between the appearance parameter of the updated work model and the appearance parameter of the work model before being updated is not smaller than a preset threshold, in step S150. The first video frame after a preset number of video frames may be set as a specified start frame. Thereafter, the face tracking method may return to step S120 to perform face tracking from the designated start frame based on the updated working model.

According to another embodiment, when performing an update on the work model, if the difference between the appearance parameters of the updated work model and the appearance parameters of the work model before being updated is less than or equal to the preset threshold, in step S145. You can set the modeling instruction to 0 (or No) to indicate whether to continue updating the working model. In other words, if it is determined that the 3D face most similar to the face of the user has already been generated, the face tracking method may no longer perform an update process on the work model.

Next, in step S148, the face tracking method may determine whether the video frame after the preset number of video frames is the last video frame of the plurality of continuously input video frames. If the video frame after the preset number of video frames is not the last video frame among the plurality of continuously input video frames, the face tracking method performs step S150 to perform the first video frame after the preset number of video frames. May be set to the designated start frame and the process returns to step S120.

The face tracking method may end the face tracking method when the video frame after the preset number of video frames is the last video frame among a plurality of video frames continuously input.

According to one embodiment, the face tracking method may output the last updated working model before terminating the face tracking method.

As described above, the face tracking method performs face tracking using the current working model, extracts facial feature points, facial expression parameters, and head pose parameters, and uses extracted facial feature points and head pose parameters and corresponding video frames. By updating the work model, iterative tracking can be performed on the face model most similar to the user's face, and more accurate face tracking results can be output. The face tracking result may include an expression parameter, an appearance parameter, and a head pose parameter.

It is possible to provide a 3D face model more similar to the user's face by repeatedly performing face tracking on continuously input face video frames and updating the 3D face based on the result of the face tracking. Also, face tracking may be repeatedly performed on continuously input face video frames, and high-precision facial expression information may be output by updating the 3D face based on the tracking result.

FIG. 5A illustrates an apparatus for implementing a 3D face modeling method and / or a face tracking method, according to an exemplary embodiment.

An apparatus for implementing a 3D face modeling method and / or a face tracking method may include a tracking unit and a modeling unit. The tracking unit may perform steps S110 to S120 shown in FIG. 1A or steps S110 to S125 shown in FIG. 1C, and the modeling unit may perform steps S130 to S150 shown in FIG. 1A or FIG. 1C. Can be.

Referring to FIG. 5A, the tracking unit performs face tracking on video frames 0 to video frames t _2-1 using a working model (reference 3D face model M ₀ ), and performs video frames 0 to video frames t ₂ -1, And a tracking result (that is, result 0 to result t ₂ -1 shown in FIG. 5A) including a facial feature point, an expression parameter, and a head pose parameter extracted from each video frame. The tracking result is provided to the modeling unit and can be output to the user through an input / output interface as needed.

The modeling unit may update the work model based on the tracking result (result 0 to result t ₂ -1) output from the tracking unit. (Refer to the description of FIGS. 1A and 1B for the update operation.) For the sake of simplicity, the updated working model is represented by M ₁ .

Then, the tracking unit may perform face tracking on video frames t ₂ to video frames t ₃ based on the updated working model M ₁ according to a preset rule (see description in FIG. 1A), and the tracking result (Result t ₂ ~ Result t ₃ ) can be output. The modeling unit may update the work model M ₁ based on the result t ₂ -1 to the result t ₃ . The face tracking and updating of the working model may be repeatedly performed until an optimal model meeting a condition is obtained, or until all video frames are input. The tracking unit and the modeling unit may operate simultaneously.

According to a preferred embodiment, the apparatus for implementing the 3D face modeling method and / or face tracking method may include a training unit, the training model M ₀ in advance by training a reference 3D face through a series of 3D face data offline Can be set.

5B is a diagram illustrating an apparatus for implementing a 3D face modeling method and / or a face tracking method according to another exemplary embodiment.

Unlike in FIG. 5A, an apparatus for implementing the 3D face modeling method and / or face tracking method illustrated in FIG. 5B may include a plurality of modeling units (eg, modeling unit A and modeling unit B). By using the plurality of modeling parts alternately, step S130 may be repeatedly performed, and the results repeatedly performed may be integrated.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software.

Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

Setting a preset reference 3D face as a work model and performing face tracking on a video frame basis based on the work model;
Generating a tracking result comprising at least one of a facial feature point, an expression parameter, and a head pose parameter from the video frame;
Updating the work model based on the tracking result; And
Outputting the updated working model
3D face modeling method comprising a. The method of claim 1,
The 3D face,
3D face modeling method comprising at least one of a 3D shape of the face, appearance parameters, facial expression parameters, and head pose parameters The method of claim 1,
Wherein the generating comprises:
3. The method of modeling a 3D face that generates a tracking result corresponding to a preset number of video frames based on a specified start frame among input video frames. The method of claim 1,
Wherein the updating comprises:
And determining whether to update the work model based on a difference between the appearance parameter of the updated work model and the appearance parameter of the work model before being updated and a preset threshold. The method of claim 1,
The working model of the 3D face,

Lt; / RTI >
S is a 3D shape, a is an appearance component, e is an expression component, q is a head pose, and T (S, q) rotates or moves the 3D shape S according to head pose q. 3D face modeling method representing a function for performing at least one of the motions to. The method of claim 5,
The reference 3D face,
Average shape S ^o , appearance components

Facial expression elements

, And the reference head pose q ^o ,
i = 1: N,

Represents a change in facial appearance, j = 1: M,

Is a 3D face modeling method that represents a change in facial expression. The method of claim 1,
Pretraining the reference 3D face with 3D face data offline and setting the trained reference 3D face as the working model
Modeling method of the 3D face further comprising. The method of claim 1,
Generating the tracking result and updating the working model are performed at the same time. The method of claim 1,
Wherein the updating comprises:
Selecting a video frame most similar to a neutral expression in the generated tracking result as a neutral expression frame;
Extracting a face sketch from the neutral facial expression frame based on facial feature points included in the neutral facial expression frame; And
Updating the working model based on feature points of the neutral facial expression frame and the extracted face sketch
3D face modeling method comprising a. 10. The method of claim 9,
The selecting step,
Calculating facial expression parameters for each tracked video frame; Setting a facial expression value of the facial expression parameter that appears at the highest frequency to a neutral facial expression value; And
Selecting as a neutral facial expression frame a video frame having a difference between the total K facial expression parameters and the neutral facial expression value smaller than a preset threshold;
3D face modeling method comprising a. 10. The method of claim 9,
Wherein the extracting comprises:
3D face modeling method of extracting a face sketch from the neutral facial expression frame using an active contour model algorithm. The method of claim 5,
Wherein the updating comprises:
Updating the head pose q of the working model with the head pose of the neutral facial expression frame;
Setting the facial expression component e of the working model to zero; And
The work model S (a, e, q) is matched with the feature point position of the neutral facial expression frame, and the face sketch calculated through the work model S (a, e, q) is extracted from the neutral facial expression frame. Modifying appearance component a of the working model by matching with a sketch
3D face modeling method comprising a. The method of claim 1,
The 3D face modeling method,
And repeatedly updating the work model, and reflecting the repeatedly performed result to the work model. The method of claim 1,
Wherein the generating comprises:
3D face modeling method for outputting the generated tracking results through at least one of an input interface and an output interface. The method of claim 1,
Wherein the outputting step comprises:
And if the updating of the working model is completed, outputting the updated working model through at least one of an input interface and an output interface. The method of claim 1,
Wherein the generating comprises:
3. A method of modeling a 3D face that determines the number of video frames to perform face tracking based on at least one of an input rate, a noise characteristic, and an accuracy requirement of tracking of an input video frame. The method of claim 1,
Wherein the generating comprises:
Facial feature points, facial expression parameters, and head pose parameters using at least one of an Active Appearance Model (AAM), an Active Shape Model (ASM), and a Composite Constraint AAM. A method of modeling a 3D face to obtain at least one of. The method of claim 5,
Wherein the generating comprises:
And if the face tracking fails, resetting the appearance component a of the working model to zero and performing the generating again. A tracking unit which performs face tracking on the input video frame based on the working model and generates a tracking result including at least one of a facial feature point, an expression parameter, and a head pose parameter; And
A modeling unit for updating the work model based on the tracking result
3D face modeling apparatus comprising a. 20. The method of claim 19,
Training unit that pre-trains the reference 3D face with 3D face data offline and sets the trained reference 3D face as the working model
3D face modeling device further comprising.

Images