KR100512567B1 - Displaced Subdivision Mesh restoration method for face animation - Google Patents

Abstract

The present invention relates to a method for restoring a DSM for a face animation that provides convenience to a face animation by allowing a face model to be directly restored to a Displaced Subdivision Mesh (DSM) from a set of atypical three-dimensional points. As such, the present invention automatically generates a lip boundary and applies it to a scan model, and reduces the amount of mesh to the maximum to increase efficiency in model transmission. That is, when transmitting, a small amount of face model is transmitted, and the receiving side can restore the original state again using wavelets. In addition, the DSM step can be adjusted so that the model can be effectively animated by adjusting the step according to the viewpoint distance, fully animate at a short distance, and a main part such as the mouth or forehead at a certain distance or more.

Description

Displaced Subdivision Mesh restoration method for face animation}

The present invention relates to a method for restoring a Displaced Subdivision Mesh (DSM) for face animation, and in particular, a face model can be directly restored to a Displaced Subdivision Mesh (DSM) from a set of atypical three-dimensional points. The present invention relates to a method for restoring a DSM for facial animation to provide convenience for animation.

In the above, the DSM is applied to a mesh generated only by subdivisions to prevent specific shapes from gradually changing to smooth curves and to maintain subdivision meshes while maintaining sharp features of the object. Say.

In general, animation refers to a technique for continuously displaying a series of images generated inside a computer on a display screen so that the image looks similar to moving, or such an image. Recently, in computer-aided learning (CAI), such animation techniques have been used to enhance the understanding of learners, giving visual effects to the explanation of complex and difficult problems.

The process of creating human facial expressions requires a high degree of expertise, time-consuming investment and manual labor. In addition, in order to perform real-time face animation, the amount of mesh should be less than an appropriate level.

Conventional facial animations were created manually by experienced designers. This requires a considerable amount of time and the constraint of having to repeat the same task each time the model changes.

To solve this problem, various methods have been proposed. Among them, a method of transforming a source model into a target model and generating an animation of a target model using the source model are mainly used. Here, the source model is an animation model and the target model is a model to be animated.

The method of transforming a source model into a target model is a method of transforming the source model into a target model so as to animate the source model as if it were a target model.

However, this method requires the user to manually photograph hundreds of feature points, and it is rather awkward and inconvenient to use because it uses the animation of the source model.

In addition, the method of generating an animation of the target model using the source model is a method of performing the animation using the target model directly.

However, since this method can reduce the number of feature points to 30 inwards, the amount of manual work is greatly reduced, but the problem is that the model needs to have a lip boundary or a western face.

That is, the conventional methods used for facial animation have problems such as animation of an unnatural source model, constraints of a model that a lip boundary must exist, the same animation regardless of distance, and a long time for transferring a model. .

Therefore, the present invention has been proposed to solve various problems occurring in the conventional face animation as described above,

An object of the present invention is to restore a face model to a Displaced Subdivision Mesh (DSM) directly from a set of atypical three-dimensional points, thereby providing a convenience for facial animation. To provide a way.

"DSM restoration method for the face animation" according to the present invention for achieving the above object,

It automatically generates the edges of the lips and applies them to the scan model, reducing the amount of mesh to the maximum to increase efficiency in model transmission. That is, when transmitting, a small amount of face model is transmitted, and the receiving side can restore the original state again using wavelets.

In addition, since the DSM step can be adjusted, the model can be effectively animated by adjusting the step according to the viewpoint distance. Not only the LOD of the mesh itself, but also the LOD of the animation can be fully animated at a short distance, and only a certain part of the mouth or forehead can be animated at a certain distance or more.

Hereinafter, a preferred embodiment of the present invention according to the technical spirit as described above will be described in detail with reference to the accompanying drawings.

The present invention relates to a method for directly reconstructing a face model to a DSM and performing animation from a set of atypical three-dimensional points. The reconstructed DSM undergoes a wavelet transform, enabling progressive presentation, transfer and multilevel editing. For facial animation, expression retargeting is performed on the simplest control mesh, and motion vectors of different levels of detail are automatically generated using displacement maps and subdivision masks.

1 is a flowchart illustrating a method for restoring a Displaced Subdivision Mesh (DSM) for facial animation according to the present invention.

As shown therein, after receiving the 3D scan face data, generating the control mesh using the point cloud data (S10 to S30); Generating a displacement map using the reconstructed subdivision mask (S40); Generating a retargeted control mesh for animation of the generated control mesh (S50); Generating a retargeted subdivision mesh by giving an expression to the multi-level DSM from the generated retargeted control mesh (S60); Generating an analysis filter and a synthesis filter for adjusting the detail of the model using the generated control mesh and the displacement map (S70); Step S80 is performed by applying the analysis filter and the synthesis filter to the generated retargeted subdivision mesh to generate a retargeted DSM.

A method for restoring a Displaced Subdivision Mesh (DSM) for face animation according to the present invention as described above will be described in detail with reference to FIGS. 2 to 15.

First, in step S10, the 3D scan face data is input, and in step S20, data and subdivision masks in the form of a source mesh and a point cloud are restored from the received 3D scan face data.

In step S30, a control mesh is generated from the restored point cloud data.

That is, as shown in FIG. 7, a shrink warping is applied to cover the point cloud data with a bounding box having a dense surface and close to the surface of the model. A dense surface bounding box simplifies the shrink-wrapped mesh as closely as possible to create a control mesh.

2 is a flowchart illustrating an embodiment of a process of generating the control mesh.

As shown therein, after forming the bounding box surrounding the point cloud, subdividing the formed bounding box and performing shrink wrapping with the point cloud (S31); Generating a control mesh by applying a point-based simplified algorithm to the shrink wrapped bounding box (S32); Subdividing the generated control mesh to generate a parametric wholesale surface serving as a basic mesh for surface detailed sampling (S33); Sampling using local subdivision surface fitting (S34) to prevent shrinkage during the subdivision and to avoid undue computation due to global optimization.

First, in order to generate a control mesh from the point cloud as in step S31, a bounding box surrounding the point cloud is created. Then subdivid the bounding box and shrink wrap to the point cloud. Shrinking wrapping repeats the process of projection and smoothing. In the projection process, each vertex of the subdivided bounding box is drawn to the point at the closest point in the point cloud. The smoothing process is required for even sampling during surface detail sampling. To do this, a laplacian approximation is used.

Next, in step S32, a control mesh is generated by applying a point-based simplification algorithm to the shrink wrapped bounding box. In the point-based simplified algorithm, unlike the conventional QEM algorithm, the distance between the point cloud and the simplified mesh becomes an error metric.

Next, by subdividing the generated control mesh in step S33, a parametric domain surface serving as a basic mesh for surface detail sampling may be generated. To this end, the present invention uses a loop subdivision.

Finally, in step S34, sampling is performed using a regional subdivision surface fitting to prevent shrinkage during the subdivision and to avoid excessive calculation due to global optimization. Then, each vertex of the mesh shrinked by the subdivision is moved near the vertices of the point clouds that exist at the closest distance. That is, the mesh reduced in size by the subdivision is enlarged to the size of the mesh reconstructed initially.

Next, in step S40, a displacement map is generated. That is, after subdivisioning the control mesh, the displacement mesh is generated by comparing the shrink-wrapped mesh with the error mesh. If you add a displacement map to the subdivision mesh, you'll get a mesh that's pretty close to the original.

Next, in step S50, a retargeted control mesh is generated.

Use expression retargeting to animate the control mesh. Expression retargeting refers to animate different models with one animation. To use expression retargeting, find the same number of feature points in the original model with animation and the target model to be animated, transform the original model similar to the target model through volume morphing, and then target the animation data of the original model. You must modify it to fit your model. In the present invention, the control mesh is used as the target model. Animation takes the form of changing the motion vector of each point in the model every frame. 9 is an exemplary view showing an example of Expression Retargeting.

3 is a flowchart illustrating an embodiment of a process of generating the retargeted control mesh.

As shown therein, step S51 of performing volume morphing to transform the standard model into a target model; Performing a cylindrical projection that completely adheres the standard model to the target model after the step (S52); Converting directions and lengths of the motion vectors present in the standard model into motion vectors of the target model (S53 to S54); Cylindrical Projection of the target model as a deformation model (S55); Computing a motion vector of the control mesh (scan model) (S56).

First, in step S51, volume morphing is performed. Volume morphing is the process of transforming a standard model into a target model. To do this, we need a match point and use the Radial Basis Function (RBF). RBF is known to be very useful for fitting face models. This is because RBF has a strong fitting function. Volume morphing takes several steps. First, we compute a matrix for moving the match of the standard model to the match of the target model. Next, the matrix is used to calculate the weights for approximating all the vertices of the standard model to the target model. To calculate the weight, we need a value that minimizes the cost function that quantifies the difference in shape between the standard model and the target model. To obtain the value, Generalized Cross-Validation (GCV) is used, which takes an initial value and operates recursively. In other words, the result is written back as an input. If the result of the calculation converges to a constant, that is the value that minimizes the cost function. Next, the weights are derived from these, and the weights are put into RBF to form a morphing matrix to transform the standard model into a scan model.

Next, in step S52, the first cylindrical projection is performed. Even volume morphing does not completely transform the target model. Some approximation stops the deformation, leaving some gaps between the two models. Therefore, it is the cylindrical projection that closes this gap and keeps the standard model in close contact with the target model. The key to this step is to fully embed the vertices of the standard model into the face of the target model. The direction in which the vertices move is starting from the center of the model and going through each vertex. Barycentric coordinates are then used to determine whether each vertex in the standard model has reached the plane of the target model correctly. After all vertices have been determined, the standard model is completely in close contact with the scan model. FIG. 10 illustrates a deformation model generation process, FIG. 10A illustrates a volume morphing process, and FIG. 10B illustrates a cylindrical projection process. 11 is an exemplary view showing generation of a deformation model.

Next, in step S53, the direction of the motion vector is changed. The animation of the model is performed by the motion vectors present in each vertex. The motion vector means a direction and a distance for moving from the vertex of the first frame to the position of the vertex of the current frame. Each face has a different shape, so the motion vector is inevitably different. Therefore, it is necessary to replace the motion vector existing in the standard model with the motion vector of the target model. Vectors generally have size and direction. Therefore, to change the motion vector to match the target model, the size and direction must be changed.

First, find the coordinate system that exists in each vertex of the standard model, and find the matrix to fit it to the X, Y, and Z axes of the world coordinate system. To find the coordinate system that exists in a vertex, we use the normal vector as the X axis, find a plane perpendicular to that axis, and project any neighboring vertices onto that plane. The original vertex and the projected vertex are then specified on the Y axis. The Z axis is obtained by calculating the cross products of the X and Y axes. In addition, after obtaining a matrix matching the world coordinate system to the coordinate system existing in the vertex of the standard model (hereinafter, referred to as "deformation model") transformed into the target model, multiplying the vertex coordinate system of the standard model by multiplying these two matrices. Get the matrix transformed by.

Next, in step S54, the length of the motion vector is converted. Create a bounding box (BB) with each vertex and its neighbors, and compare the X, Y, and Z lengths of the corresponding BBs between the standard and deformed models to obtain the scaling values. This transformed the motion vector of the standard model into the motion vector of the deformation model. That is, the animation of the standard model can be used in the deformation model. 12A is an exemplary diagram illustrating a direction transformation of a motion vector, and FIG. 12B is an exemplary diagram illustrating a length transformation of a motion vector.

Next, in step S55, the second cylindrical projection is performed. That is, it is time to move the motion vector of the deformation model to the target model. First, as shown in FIG. 13A, the target model is cylindrically projected to the deformation model. This ensures that all vertices in the target model are on each side of the deformation model.

Next, the motion vector of the control mesh is calculated in step S56. That is, if the triangle of the deformed model including the vertex of the target model is found and the motion vectors of the three vertices of the triangle are combined at an appropriate ratio using barycenrtic coofdinates as shown in FIG. 13B, the motion vector of the scan model is generated.

Next, in step S60, a retargeted subdivision mesh is generated.

The application of facial expressions from the control mesh to the multi-level DSM is made in the order of the steps: the control mesh is applied to the primary DSM and the primary DSM is applied to the secondary DSM. For subdivisions, the position of the newly created point is determined by the mask value. At this time, the motion vector of the newly created point is calculated by applying the mask value to the motion vector of the previous model, as if the location of the point is found. The resulting animation gives facial expressions to the DSM at each stage.

4 is a flowchart illustrating an embodiment of a process of generating the retargeted subdivision mesh.

As shown therein, performing a general loop subdivision on the generated Retargeted control mesh (S61); Computing a point newly generated by the loop subdivision and a Mossen vector of the point (S62).

First, in step S61, a general loop subdivision is performed on the generated retargeted control mesh. In step S62, motion vector propagation is performed.

14A and 14B show a point newly generated by the subdivision and a mossen vector of the point. The dividing in two cases is due to the presence of a hole in the face. The motion vector of the new point is calculated using FIG. 14A for the point on the general edge and FIG. 14B for the point on the boundary edge like the boundary of the mouth. here (i = 1, 2, ..., n) gives the motion vector of that point, Is the motion vector of the new point.

Next, in step S70, an analysis filter and a synthesis filter are generated.

An analysis filter (P filter) and a synthesis filter (Q filter) are required to automatically adjust the detail level of the model. Such analysis filters and synthesis filters can be conveniently expressed as a block matrix. By pre-configuring an analysis filter and a synthesis filter for each subdivision step, each step can be freely moved by matrix operation alone. The process of applying the P matrix to create a parametric domain surface is the same as the conventional wavelet method. The Q matrix stores coefficients for modifying the generated parametric domain close to the original point cloud.

5 is a flowchart illustrating an embodiment of a process of generating the analysis filter and the synthesis filter.

Forming a subdivision mesh and shooting a ray in a vertex normal direction from a virtual smooth surface approximating a point cloud to obtain a wavelet coefficient (P filter) from the subdivision mesh (S71); Obtaining an inverse matrix (Q filter) of the wavelet coefficients (S72); In step S73, the obtained wavelet is analyzed.

First, in step S71, unlike the conventional wavelet coefficient storage method, the displacement map obtained during the DSM reconstruction process is used to construct the content of the Q matrix. In the present invention, once the subdivision mesh is formed, the ray is shot in the vertex normal direction to a virtual smooth surface approximating the point cloud to obtain a wavelet coefficient value (P filter).

Next, in step S72, an inverse matrix of the analysis filter (P filter) is obtained.

In step S73, the obtained wavelet is analyzed.

Next, in step S80, a retargeted DSM is generated.

The DSM of each subdivision stage does not have animation data, but after expression retargeting to the control mesh according to the proposed method, all of them are automatically generated by using a displacement map and a subdivision mask. A motion vector is generated in the subdivision step. Therefore, the same facial expression can be generated in all the subdivision models, and can be applied to any model. 15A-15D show an animation of the DSM.

6 is a flowchart illustrating an embodiment of a process for generating a retargeted DSM.

As shown therein, retrieving the level of the generated DSM (S81); Determining the generated DSM as a retargeted DSM when the search result is a desired level (S82); If the search result is at a high level, applying the synthesis filter to lower the detail level of the model (S83 and S84); In the case where the search result is a low level, step S85 is performed to increase the detail level of the model by applying an analysis filter.

First, in step S81, the level of the generated DSM is searched. If the search result is a desired level, the generated DSM is determined as the retargeted DSM in step S82. Next, when the search result is a high level, a coarse mesh is generated by lowering the DSM level by applying the synthesis filter in step S84. In addition, when the search result is a low level, as in step S85, the analysis filter is applied to increase the level of the DSM to generate a dense mesh, thereby creating a model of a detailed level desired by the user.

According to the present invention described above, the process of animate a human face using a computer in the prior art required designer skill and a long time manual work, but in the present invention, the effect of enabling such a task in a simple manual operation and a short time have.

In addition, since a scan model is used as a face model, even a beginner can easily animate his or her face model on a computer, and it is possible to apply various expressions by performing animation using motion capture data.

In addition, the present invention is effective for generating facial expressions of people appearing in movies or animations, and in particular, when attempting to transmit to a network, it is possible to implement original high quality animations by receiving only a small amount of data.

1 is a flowchart showing a method for restoring a Displaced Subdivision Mesh (DSM) for facial animation according to the present invention;

2 is a flowchart illustrating an embodiment of a process of generating a control mesh of FIG. 1;

3 is a flowchart illustrating an embodiment of a process of generating a retargeted control mesh of FIG. 1;

4 is a flowchart illustrating an embodiment of a process of generating a retargeted subdivision mesh of FIG. 1;

FIG. 5 is a flowchart illustrating an embodiment of a P and Q filter generation process of FIG. 1;

6 is a flowchart illustrating an embodiment of a process of generating a retargeted DSM of FIG. 1;

7 is an exemplary view showing a process of restoring a DSM from a point cloud in the present invention,

8 is a view showing a sampling process for generating a DSM in the present invention,

9 is an exemplary view showing an expression retargeting process in the present invention,

10a and 10b is a view showing a deformation model generation process in the present invention,

11 is an exemplary view for generating a deformation model in the present invention,

12A and 12B are exemplary views of transforming a motion vector in the present invention.

13A and 13B illustrate a process of converting a motion vector in the present invention.

14A and 14B illustrate a motion vector propagation process according to the present invention.

15A to 15D illustrate the results of performing animation with the restored DSM in the present invention.

Claims (6)

In the method of restoring Displaced Subdivision Mesh (DSM) for facial animation, A first step of generating a control mesh from the point cloud data after receiving the 3D scan face data; Generating a displacement map using the reconstructed subdivision mask; Generating a retargeted control mesh for animation of the generated control mesh; Generating a retargeted subdivision mesh by giving an expression to the multi-level DSM from the generated retargeted control mesh; Generating an analysis filter and a synthesis filter for adjusting the detail of the model using the generated control mesh and the displacement map; And a sixth step of generating a retargeted DSM by applying the analysis filter and the synthesis filter to the generated retargeted subdivision mesh. The method of claim 1, wherein the first step, After forming a bounding box surrounding the point cloud, subdividing the formed bounding box and performing shrink wrap to the point cloud; Generating a control mesh by applying a point-based simplified algorithm to the shrink wrapped bounding box; Subdividing the generated control mesh to generate a parametric wholesale surface that becomes a basic mesh for surface detailed sampling; Sampling using local subdivision surface fitting to prevent shrinkage during subdivision and to avoid undue calculation due to global optimization. The method of claim 1, wherein the third step, Performing volume morphing to transform the standard model into a target model; Performing a Cylindrical Projection that completely adheres the standard model to the target model after the step; Converting a direction and a length of a motion vector present in the standard model into a motion vector of the target model; Cylindrical Projection of the target model as a deformation model; Comprising a step of calculating the motion vector of the control mesh (scan model). The method of claim 1, wherein the fourth step, Performing general loop subdivision on the generated Retargeted control mesh; Computing a point newly created by the loop subdivision and a Mossen vector of the point (DSM) for facial animation. The method of claim 1, wherein the fifth step, Forming the subdivision mesh and shooting a ray in the vertex normal direction from the virtual smooth surface approximating a point cloud to obtain a wavelet coefficient (analysis filter: P filter); Obtaining an inverse matrix (synthetic filter: Q filter) of the wavelet coefficients; A method for restoring a DSM for facial animation, characterized in that the step of analyzing the obtained wavelet. The method of claim 1, wherein the sixth step is Retrieving the level of the generated DSM; Determining the generated DSM as a retargeted DSM when the search result is a desired level; Lowering the detail level of the model by applying a synthesis filter when the search result is a high level; And if the search result is a low level, applying the analysis filter to increase the detail level of the model.