KR101521358B1 - Method and apparatus for matching of multi model - Google Patents

Abstract

Disclosed are a method and an apparatus for matching a multi-model. The method includes the steps of: sorting a decorative model including at least one feature point into independent surfaces; setting at least one control point on each independent surface; and generating a control point set model by generating connection information between the control points of the independent surfaces and the feature points. The decorative model can be linearly changed according to a 3D model by the control point set model.

Description

[0001] METHOD AND APPARATUS FOR MATCHING MULTI MODEL [0002]

The present invention relates to a multi-model matching method and apparatus for classifying a decorative model into independent surfaces, extracting control points, and linearly deforming the model based on the extracted control points.

The face of a person can express very important information in various forms such as psychological state, social interaction, physiological signal, etc. in conversation with each other. Especially facial expressions have an important meaning to communicate with others in real life. On the other hand, since the human expression in the virtual reality expresses the user 's emotion through the 3D facial avatar, it can be said that it is possible to interact with the language by using an exaggerated facial expression rather than the actual user' s face. For this reason, research on faces has been actively studied in various fields such as face animation synthesis, robotics, and HCI.

As computing technology and optical technology developed, it became possible to obtain accurate data reconstructing the human face in three dimensions. However, it is still difficult to restore hair and beard, which is a feature of the face, even if the face is restored by using the latest optical equipment. In order to solve this problem, a method of extracting hair using a separate growth algorithm in 2D image, and reconstructing the 3D space using a multi-view stereo method has been proposed. It is not suitable for use in facial animation because the surface of the face must be deformed to construct a decorative model such as a hair obtained using a separate growth algorithm on the skin surface. Another solution is to make hair and beard models using commercial programs.

However, it is difficult to apply models of hair, beard and the like according to the conventional method to faces other than the face of a specific character, and it is very unnatural even if applied.

The present invention is to provide a multiple model matching method and a device for extracting control points after classifying an ornamental model into independent surfaces, and performing linear transformation on the basis of the extracted control points.

In addition, the present invention is to provide a method and apparatus for multiple model matching that can not only decorate a decorative model naturally in accordance with a face model but also reuse the decorative model to each face model.

The present invention provides a method of classifying a decorative model into independent surfaces, extracting control points, and linearly deforming the model based on the extracted control points.

According to an embodiment of the present invention, there is provided a method of designing a decorative model, the method comprising: classifying each decorative model including at least one feature point into independent surfaces; Setting at least one control point on each independent surface; And generating a control point set model by generating connection information of the control points of the respective independent surfaces and the minutiae points, wherein a matching method of multiple models in which the ornamental model is linearly deformed according to the 3D model by the control point set model Can be provided.

The independent surface is an independent geometric information unit in which the vertex-to-vertex interconnection information exists in the ornamental model.

The step of setting the control points may respectively set at least one vertex of the geometric information of the independent surface to the control point.

Wherein the linear transformation comprises: detecting at least one corresponding point of the 3D object for which the ornamental model is to be linearly transformed into the 3D object; Calculating a displacement between the control point set model and the corresponding point; And linearly transforming each vertex of the control point set model to the corresponding point by using the calculated displacement as a cotangent value of a weight matrix calculated in advance.

The step of detecting the corresponding point may detect a vertex located at a minimum distance between each vertex included in the control point set model and each vertex of the 3D model as a corresponding point.

The linear deformation may be performed such that a peripheral vertex of each vertex of the control point set model is linearly deformed to the face model using a cotangent value of the previously calculated weight matrix.

The peripheral vertex may be a vertex of the ornamental model included in the radius based on the distance between each vertex of the control point set model.

The step of generating the control point set model may include generating connection point information between the minutiae points and the control points so that the minutiae of the ornamental model and at least two control points set on the respective independent surfaces form one polygon, can do.

According to another aspect of the present invention, there is provided an apparatus for classifying a decorative model into independent surfaces, extracting control points, and linearly deforming the model based on the extracted control points.

According to an embodiment of the present invention, there is provided a surface classification apparatus comprising: a surface classifying unit for classifying, respectively, an ornamental model including at least one feature point into independent surfaces; A control point set model generation unit for setting at least one control point on each of the independent surfaces and generating control point set models by generating connection information of the control points and the minutiae on the respective independent surfaces; And a matching unit for linearly deforming the decorative model to a 3D object using the control point set model.

Wherein the matching unit detects at least one corresponding point of the 3D object for which the ornamental model is to be linearly transformed to the 3D object, calculates a displacement between the control point set model and the corresponding point, The vertex of the control point set model can be linearly transformed to the corresponding point by using the value as the cotangent value of the matrix.

The control point aggregation model generation unit generates connection point aggregation models by generating connection information between the minutiae points and the control points so that at least two control points set on the minutiae of the ornamental model and each of the independent control points form one polygon Lt; / RTI >

The control point set model generation unit may set at least one vertex of the geometric information of the independent surface as the control point.

By providing the multimodel matching method and apparatus according to an embodiment of the present invention, it is possible to classify the decorative model into independent surfaces, extract the control points, and linearly deform the face model based on the extracted control points.

Thus, not only can the present invention match the decorative model naturally according to the face model, but also the advantage that the decorative model can be reused for each face model.

1 is a flowchart for explaining a multi-model matching method according to an embodiment of the present invention;
Figure 2 illustrates an ornamental model comprised of discrete surfaces according to one embodiment of the present invention.
3 is a diagram illustrating a method for creating a control point aggregation model in a decoration model according to an embodiment of the present invention.
4 is a flowchart illustrating a method of linearly transforming a decorative model to a face model according to an embodiment of the present invention.
5 is a pseudo code for detecting a correspondence point of a face model corresponding to each vertex of a control point set model of an ornamental model according to an embodiment of the present invention.
FIG. 6 is a view for explaining vertexes of a control point set model and corresponding points of a face model and surrounding vertices according to an embodiment of the present invention; FIG.
7 is a diagram showing an initial state of a decoration model according to an embodiment of the present invention;
8 is a view showing a result of fitting a decoration model according to an embodiment of the present invention to a face model;
FIG. 9 is a block diagram schematically showing an internal configuration of a multi-model matching apparatus according to an embodiment of the present invention; FIG.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention is capable of various modifications and various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

The present invention is an invention for defining a 3D model of a face and a decorative model such as a hair, a beard, a box, etc., corresponding to a facial feature, and then aligning and matching the decorative model to the 3D face model.

Hereinafter, the 3D face model may include geometry information and texture information for a three-dimensional face. It should be understood that the geometric information includes the vertexes of the 3D face model and the connection information between the vertices.

In addition, the decoration model may include geometry information and texture information for decorating a 3D face model such as hair, beard, bracelet, and the like.

The 3D face model and the ornamental model may be composed of one component or a plurality of components, respectively.

In one embodiment of the present invention, one component refers to geometry information in which connection information between vertices exists, and will be referred to as an independent surface in the following description. That is, in the following, an independent surface should be understood as a component in which the connection information of vertices is connected together.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a flowchart illustrating a method of matching multiple models according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an ornamental model composed of independent surfaces according to an embodiment of the present invention. FIG. 4 is a diagram illustrating a method of generating a control point aggregation model in a decoration model according to an embodiment of the present invention.

In step 110, the multiple model registration device 100 analyzes the decorative models and classifies them into separate surfaces.

For example, the multi-model registration device 100 may analyze the connection information between the vertices of the ornamental model to classify the components having connection information between vertices (i.e., all vertices connected) into one independent surface.

Fig. 2 illustrates a whisker model among the decorative models. In general, the beard is composed of a single component of each of the beards. Accordingly, as shown in FIG. 2, the multi-model matching apparatus 100 can analyze the connection information between vertices of the ornamental model, classify the vertices into independent units connected to each vertex, and classify the ornamental models into independent surfaces.

Also, at least one feature point can be set for the decorative model. The feature point may be set to at least one of the vertices of the ornamental model.

In FIG. 2, the geometric information of the decorative model is shown as a triangular polygonal shape. However, the shape of the polygon may be varied, and it is natural that the connection information may be changed according to the shape of the polygon. However, in the present specification, it is assumed that the connection information of the geometry information is in the form of a triangular polygon in order to facilitate understanding and explanation.

In step 115, the multiple model registration device 100 sets at least one control point on each independent surface.

For example, the multi-model registration device 100 can set at least one vertex of geometric information of each classified independent surface as a control point.

Referring to FIG. 3, 310 of FIG. 3 illustrates an example of a decorative model including at least one feature point. An example of the connection between vertices in such a decorating model is shown in FIG. 320 shows an example in which at least one of the vertices of each classified independent surface is set as a control point. Reference numeral 320a shown at 320 in FIG. 3 represents control points respectively set on the classified independent surfaces, and 320b represents feature points set in the decorative model. An example 325 of FIG. 3 shows an example where control points are set on each independent surface of the ornamental model.

In step 120, the multiple model registration apparatus 100 generates connection point information between the minutiae of the ornamental model and each control point set on the classified independent surface to generate a control point set model.

More specifically, the multi-model matching apparatus 100 can generate a control point set model by connecting minutiae points (vertices) set in the ornamental model and respective control points (vertices) set on the classified independent surfaces.

In FIG. 3, an example of generating a control point set model by generating connection information between minutiae points (vertices) of the ornamental model and each control point (vertex) set on the classified independent surface is shown.

As shown in 330 of FIG. 3, connection information can be generated by connecting each control point (vertex) set on the classified independent surface and minutiae points (vertices) of the ornamental model, respectively.

In generating the control point set model, it is possible to generate the control point set model by generating the connection information so that at least two control points (vertices) form one polygon based on the minutiae of the ornament model (vertices).

For example, as shown in FIG. 3, the minutiae of the ornamental model are referred to as a first vertex 330a, and each control point set on each of the independent surfaces is sequentially referred to as a second vertex 330b, a third vertex 330c, The fourth vertex 330d, the fifth vertex 330e, and the sixth vertex 330f.

The multiple model matching apparatus 100 generates connection information so that the first vertex 330a, the second vertex 330b and the third vertex 330c form one polygon, and the first vertex 330a, the third vertex 330b, The first vertex 330c and the fourth vertex 330d form connection information so that the first vertex 330a and the fourth vertex 330d form a single polygon and the fourth vertex 330d and the fifth vertex 330e form a single polygon. And the connection information may be generated so that the first vertex 330a, the fifth vertex 330e, and the sixth vertex 330f form one polygon.

As shown in 330 of FIG. 3, when there are one minutiae of the ornamental model, connection information between each control point set on the surface independent of minutiae can be generated corresponding to the number of control points.

For example, suppose that there is one feature point of the ornamental model, six independent surfaces, and one control point for each independent surface. At this time, six pieces of connection information between the minutiae point and the control point can be generated. That is, connection information in the form of six triangular polygons can be generated.

In step 125, the multiple model matching apparatus 100 linearly transforms the decorative model to fit the face model using the control point set model.

The linear transformation will be described in more detail with reference to FIG.

FIG. 4 is a flowchart illustrating a method of linearly deforming a decoration model according to an embodiment of the present invention to a face model, FIG. 5 is a flowchart illustrating a method of linearly deforming a face corresponding to each vertex of a control point set model of a decoration model according to an embodiment of the present invention, 6 is a view for explaining the vertexes of the control point set model, the corresponding points of the face model and the surrounding vertexes according to an embodiment of the present invention, and Fig. 7 is a view FIG. 8 is a diagram showing a result of matching a decoration model according to an embodiment of the present invention with a face model. FIG.

In step 410, the multiple model matching apparatus 100 detects a corresponding point of the face model corresponding to the control point set model of the ornamental model.

Referring to FIG. 6, reference numeral 610 denotes apexes of the control point set model of the decorative model, reference numeral 615 denotes a corresponding point of the face model, and reference numeral 620 denotes a peripheral apex to a vertex of each control point set model.

The corresponding points (target vertices) of the face models corresponding to the respective vertexes of the control point set model of the ornamental model must be respectively detected as shown in Fig.

To this end, the multiple model registration device 100 can detect apexes of facial surfaces closest to each vertex of the control point set model of the ornamental model as corresponding points. To this end, the multi-model matching apparatus 100 repeatedly circles the vertex of the face model at each vertex of the control point set model of the ornamental model to obtain the distance between the two vertices. Then, the vertices of the face model corresponding to the minimum distance are set as corresponding points Can be detected.

The multiple model matching apparatus 100 can detect vertices located at the minimum distance of the face model corresponding to each vertex with respect to all the vertices of the control point set model of the ornamental model as corresponding points.

In order to detect the correspondence point of the face model corresponding to the control point set model of such a decorative model, it is natural that the decorative model must be located at a specific position in order to match the face model.

5 illustrates pseudo code for detecting a corresponding point of the face model corresponding to each vertex of the control point set model of the ornamental model.

In step 415, the multiple model registration device 100 calculates the displacement between the vertices of the set of control points of the ornamental model and the corresponding points of the face model.

일 때, 제어점집합의 정점들은

, 얼굴 모델의 대응점은

로 표현될 수 있다. 이에 따라 장식 모델의 제어점집합 모델의 정점들과 3D 얼굴 모델의 표면 대응점 변위는 하기 수 1과 같이 정의할 수 있다.For example, if a feature point for linear transformation is

, The vertices of the set of control points

, The corresponding point of the face model

. ≪ / RTI > Accordingly, the vertices of the control point set model of the decorative model and the surface correspondence point displacement of the 3D face model can be defined as follows.

Here, the calculated displacement is applied to the cotangent value of the weighting matrix.

In step 420, the multiple model matching apparatus 100 linearly transforms the control point set model to fit the face model using a pre-calculated weighting matrix.

For the linear transformation, a weight matrix must be created in advance. To generate the weighting matrix, the vertices of the ornamental model should be separated into vertices, not feature points and feature points. Hereinafter, in order to facilitate understanding and explanation, vertices that are not characteristic points will be collectively referred to as peripheral vertices.

-좌표계로 정의한다. 이때, 장식 모델의 하나의 특징점이 영향을 주는 주변 정점의 개수는 장식 모델 표면의 반경 곡률을 계산한 후 반경에 의해 결정될 수 있다.Also, among the vertices of the ornamental model, the vertexes included in the control point set model can be classified as feature points. Then, the multi-model registration device 100 registers each feature point and peripheral vertices of the ornamental model

- Defined as a coordinate system. At this time, the number of peripheral vertices to which one feature point of the ornamental model affects can be determined by the radius after calculating the radius curvature of the ornamental model surface.

에 의해 결정되며, 각 특징점들간의 거리가 가장 먼 두 거리의 합으로 결정된다. 이어, 특징점과 주변 정점의 가중치는 코탄젠트 가중치를 이용하여 계산된다. 계산된 가중치는 대각행렬인 라플라시안 그래프 행렬에 저장되고, 저장된 가중치 행렬을 이용하여 장식 모델의 제어점집합 모델의 각 정점들이 3D 얼굴 모델의 대응점으로 이동할 때 정의된 주변 정점들도 제어점집합 모델의 각 정점들이 가중치 행렬에 영향을 받은 만큼 이동할 수 있다. For example, the radius of the surrounding vertex is

, And the distance between each feature point is determined as the sum of the two distances that are the farthest from each other. Then, the weights of feature points and peripheral vertices are calculated using the cotangent weights. The calculated weights are stored in a Laplacian graph matrix, which is a diagonal matrix. When the vertexes of the control point set model of the ornamental model move to the corresponding points of the 3D face model using the stored weight matrix, the peripheral vertices defined by each vertex of the control point set model Can be moved as long as they are affected by the weighting matrix.

로 구할 수 있다. 본 발명의 일 실시예에서는 라플라시안 그래프에 저장된 내용을 가중치 행렬로 저장하였으며, 이 행렬은 장식 모델 변환 과정에서 고정된 상수값이다. 따라서, x=B에서 3차원 얼굴 모델과 유사한 형태의 장식 모델B를 구할 수 있다.For example, assuming that the vertex set of the ornamental model is A, assuming that the weight matrix is x, and that the vertex set of the ornamental model transformed into the face shape is B, Ax = B. In order to obtain B, we used a set of vertexes of the ornamentation model and a modified ornamentation model to define the weighting matrix x first. That is, let A be the vertex set of the decorating model before transformation, and A 'be the set of vertices of the decorative model to which the transformation is applied. Since A and A' are known, the weighting matrix x

. In an embodiment of the present invention, the contents stored in the Laplacian graph are stored as a weight matrix, which is a fixed constant value in the decor model conversion process. Therefore, a decoration model B similar to the three-dimensional face model can be obtained at x = B.

FIG. 7 is a diagram illustrating an initial state in which an ornamental model is matched to a face model, and FIG. 8 is a diagram illustrating an initial state in which the ornamental model is classified into independent surfaces according to an embodiment of the present invention, A model of feature point and connection information is generated to generate a control point set model, and then the result is shown as a result of linearly deforming the face model based on each vertex of the control point set model.

As shown in FIG. 8, when the decorative model is matched to the face model according to the multi-model matching method according to an embodiment of the present invention, it can be seen that the decorative model is linearly deformed to fit the final 3D face model.

FIG. 9 is a block diagram schematically illustrating an internal configuration of a multi-model matching apparatus according to an embodiment of the present invention.

9, the multiple model matching apparatus 100 according to an embodiment of the present invention includes a surface classifying unit 910, a control point aggregation model generating unit 915, a matching unit 920, a memory 925, (930).

The surface classifying unit 910 analyzes the connection information between vertices of the ornamental model and classifies them into independent surfaces on which the vertex-to-vertex connection information exists. As already mentioned above, the ornamental model may include at least one feature point.

The control point set model generation unit 915 sets at least one of the classified independent surface vertices as control points, respectively. The control points may be set for a plurality of independent surfaces for each classification, but it is assumed that the control points are set one by one in order to facilitate understanding and explanation in one embodiment of the present invention.

The control point set model generation unit 915 generates control point set models by generating connection information for the control points set on the respective independent surfaces and the minutiae points of the ornament model. As described above, the control point set model can be generated by generating the connection information in the form of a triangular mesh connecting the minutiae points of the ornamental model and at least two control points.

The matching unit 920 is means for aligning the decorative model with the face model by linearly deforming the face model based on each vertex of the control point set model of the ornamental model. This is the same as that described above with reference to FIG. 4, and a duplicate description will be omitted.

The memory 925 is a means for storing various algorithms, data, and the like necessary for operating the multi-model matching apparatus 100 according to an embodiment of the present invention.

The control unit 930 controls the internal components (e.g., the surface classifying unit 910, the control point aggregation model generating unit 915, the matching unit 920, and the control unit 920) of the multiple model matching apparatus 100 according to an embodiment of the present invention. ), The memory 925, and the like).

Meanwhile, the multi-model matching method for extracting control points after classifying the decorative model according to the embodiment of the present invention into independent surfaces, and linearly deforming the face models based on the extracted control points, May be implemented in the form of program instructions that can be executed and recorded in a storage medium. The storage medium may include program instructions, data files, data structures, and the like, alone or in combination.

Program instructions to be recorded on the storage medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of software. Examples of storage media include magnetic media such as hard disks, floppy disks and magnetic tape, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, magneto-optical media and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as devices for processing information electronically using an interpreter or the like, for example, a high-level language code that can be executed by a computer.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention as defined in the appended claims. It will be understood that the invention may be varied and varied without departing from the scope of the invention.

910:
915: Control point set model generation unit
920:
925: Memory
930:

Claims (13)

Classifying each of the decorative models including at least one feature point into independent surfaces;
Setting at least one control point on each independent surface; And
Generating a control point set model by generating connection information of the control points of the respective independent surfaces and the minutiae points,
Wherein the ornamental model is linearly deformed according to the 3D model by the control point set model. The method according to claim 1,
Wherein the independent surface is an independent geometric information unit in which the vertex-to-vertex interconnect information is present in the ornamental model.

The method according to claim 1,
The step of setting the control points, respectively,
Wherein at least one vertex of the geometric information of the independent surface is set as the control point. The method according to claim 1,
The linear strain may be,
Detecting at least one corresponding point of the 3D object for which the ornamental model is to be linearly transformed into the 3D object;
Calculating displacement of the control point set model and the corresponding point; And
And linearly transforming each vertex of the control point set model to the corresponding point by using the calculated displacement as a cotangent value of a weight matrix calculated in advance. 5. The method of claim 4,
Wherein the step of detecting the corresponding point comprises:
Wherein a vertex located at a minimum distance between each vertex included in the control point set model and each vertex of the 3D model is detected as a corresponding point. 5. The method of claim 4,
Wherein the step of linearly deforming comprises:
Wherein the peripheral vertexes of the respective vertices of the control point set model are linearly deformed in the face model using the previously calculated cotangent value of the weight matrix. The method according to claim 6,
Wherein the peripheral vertexes are vertices of the ornamental model included in the radius based on the distance between each vertex of the control point set model. The method according to claim 1,
Wherein the step of generating the control point set model comprises:
Wherein the control point set model is generated by generating connection information between the minutiae points and the control points so that the minutiae of the ornamental model and at least two control points set on each independent surface form one polygon. A computer-readable recording medium storing a program for executing the method according to any one of claims 1 to 8. A surface classifying unit for classifying the decorative models including at least one feature point into independent surfaces;
A control point set model generation unit for setting at least one control point on each of the independent surfaces and generating control point set models by generating connection information of the control points and the minutiae on the respective independent surfaces; And
And a matching unit that linearly deforms the ornamental model to match the 3D object using the control point set model. 11. The method of claim 10,
The matching unit may include:
And a controller for detecting at least one corresponding point of the 3D object for which the ornamental model is to be linearly transformed into the 3D object, computing the displacement of the control point set model and the corresponding point and then computing the calculated displacement as a cotangent value , And linearly transforming each vertex of the control point set model to the corresponding point. 11. The method of claim 10,
The control point set model generation unit may include:
Wherein the control point set model generating unit generates the control point set model by generating connection information between the minutiae points and the control points so that the minutiae of the ornamental model and at least two control points set on each independent surface form one polygon. 11. The method of claim 10,
The control point set model generation unit may include:
And sets at least one vertex of the geometric information of the independent surface to the control point.

Images