CN116665284B - Face modeling and mask model partition matching method, device, terminal and medium - Google Patents

Face modeling and mask model partition matching method, device, terminal and medium Download PDF

Info

Publication number
CN116665284B
CN116665284B CN202310961964.1A CN202310961964A CN116665284B CN 116665284 B CN116665284 B CN 116665284B CN 202310961964 A CN202310961964 A CN 202310961964A CN 116665284 B CN116665284 B CN 116665284B
Authority
CN
China
Prior art keywords
face
point cloud
points
feature
matching
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202310961964.1A
Other languages
Chinese (zh)
Other versions
CN116665284A (en
Inventor
程泽
黄少墅
高翔
宋志诚
江再风
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Yushi Technology Co ltd
Original Assignee
Shenzhen Yushi Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Yushi Technology Co ltd filed Critical Shenzhen Yushi Technology Co ltd
Priority to CN202310961964.1A priority Critical patent/CN116665284B/en
Publication of CN116665284A publication Critical patent/CN116665284A/en
Application granted granted Critical
Publication of CN116665284B publication Critical patent/CN116665284B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/172—Classification, e.g. identification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects
    • G06V20/647—Three-dimensional objects by matching two-dimensional images to three-dimensional objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Computing Systems (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Human Computer Interaction (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Processing Or Creating Images (AREA)
  • Image Processing (AREA)

Abstract

The application discloses a method, a device, a terminal and a medium for facial modeling and mask model partition matching, wherein the method comprises the following steps: carrying out characteristic vectorization on the texture, shape and expression of the face in the 2D face picture; dividing the region of the 3D face model corresponding to the face characteristic points into forehead, cheek, chin, eyebrow, nose and lip regions; extracting feature vectors of the feature points to finish matching; calculating and optimizing the sum of the distances of the corresponding positions between the feature points; and carrying out partition matching on the 3D face model and the 3D face mask model. According to the application, when the loss function is optimized through a random gradient descent algorithm, the region with high weight value setting can be more strongly optimized, and the region with small weight value setting has smaller optimizing force, so that the influence of the non-key region of the face on the matching result during matching is reduced; the human face is partitioned, so that the matching effect of the key area of the human face and the face mask is better.

Description

Face modeling and mask model partition matching method, device, terminal and medium
Technical Field
The application relates to the technical field of beauty masks, in particular to a method, a device, a terminal and a medium for facial modeling and mask model partition matching.
Background
At present, the working principle of the face modeling and beautifying method is that a corresponding three-dimensional face model is generated through a two-dimensional image, so that the face modeling and beautifying method is convenient to match with the three-dimensional structure of a face mask, and the effect of better controlling the lamp beads on the face mask to irradiate light is achieved. Therefore, the accuracy of face modeling and the accuracy of matching with the face mask in three-dimensional space are key factors for determining the cosmetic effect, and are of great importance for the subsequent use effect. Therefore, the texture, shape and expression of the face need to be accurately modeled and accurately matched with the face mask.
However, most of traditional face model modeling adopts coarse granularity modeling, accurate modeling can not be performed on fine parts such as textures, expressions and the like of the face, effects on shielding and large expressions are poor, and the generated 3D face model is low in precision. Meanwhile, the traditional 3D face model is mostly matched by adopting one-to-one comparison of face feature points and feature points on the face mask, the matching of key treatment areas of the face, such as forehead, cheek and chin, is omitted, and the matching accuracy of non-key treatment areas of the face, such as eyebrows, nose, lips and the like, has less influence on actual use, so that the matching effect of the key areas of the face and the face mask is poor, and the treatment effect of the face mask is influenced.
In view of the foregoing, it is desirable to provide a face modeling and mask model partition matching method, apparatus, terminal and medium that overcomes the above-mentioned drawbacks.
Disclosure of Invention
The application aims to provide a method, a device, a terminal and a medium for facial modeling and mask model partition matching, which aim to solve the problems of low facial modeling precision and poor matching effect with a facial mask at present, so that the facial mask can provide better and targeted treatment for key areas of the face subsequently, and the treatment effect of the facial mask is improved.
In order to achieve the above object, a first aspect of the present application provides a method for partition matching between face modeling and mask model, including:
step S10: acquiring pixel coordinate values of key feature points of a face region in a 2D face picture, and pre-constructing a 3D face model based on a convolutional neural network;
step S20: carrying out feature vectorization on the texture, shape and expression of the face in the 2D face picture, and inputting the feature vectorization into the 3D face model for parameter training; the loss function of the step is set to output the distance between corresponding feature points of the projection of the 3D face to the 2D face picture;
step S30: generating three-dimensional coordinate points of the face feature points according to the 2D face picture, and dividing the region of the 3D face model corresponding to the face feature points into forehead, cheek, chin, eyebrow, nose and lip regions through lower standard values of the three-dimensional coordinate points;
step S40: extracting feature points of the 3D face model and feature vectors of the feature points generated by the 2D face picture, matching the feature points and the feature vectors, and obtaining an initial position of the 3D face model after matching, so that matching of the 3D face model and the 2D face picture is completed;
step S50: calculating the sum of distances between the corresponding positions of the two parts of feature points of the 3D face model and the 2D face picture, and optimizing the sum of the distances by adopting a random gradient descent algorithm through a preset loss function; the weight values of the areas of the forehead, the cheeks and the chin are larger than the weight values of the areas of the eyebrows, the nose and the lips;
step S60: and carrying out partition matching on the 3D face model and the 3D beauty mask model through a preset 3D model matching system, and storing matching results, so that special treatment is carried out on a specific area of a user when the beauty mask is used.
In a preferred embodiment, in step S20, performing feature vectorization on the texture, shape and expression of the face in the 2D face picture includes:
the feature vectors of the texture T and the shape S are obtained by the following formula:
，/>is the average face shape,/->Is the average face texture and is used to determine,
、/>、/>the feature, expression and texture are respectively the basis vectors processed by PCA; />、/>、/>Generating corresponding coefficient vectors of the 3D face model; the shape-related parameters are obtained through MLP multi-layer perceptron training, and the texture-related coefficients are obtained through CNN convolutional neural network training.
In a preferred embodiment, before said step S40, the method further comprises the steps of:
reducing the number of vertexes of the 3D face model generated according to the 2D face picture through downsampling; the method specifically comprises the following steps:
the size of a unit voxel is preset, the point cloud is put into a unit voxel box with the preset size, and if the local point group with a preset range falls into a unit voxel box, all points in the unit voxel box are represented by a gravity center point.
In a preferred embodiment, in the step S30, generating the three-dimensional coordinate point of the face feature point according to the 2D face picture includes:
and positioning 468 characteristic points of the face of the 2D face picture clock by adopting a mediaapipe face recognition algorithm, and returning to the three-dimensional coordinate point of each characteristic point.
In a preferred embodiment, in said step S40,
the specific steps of feature matching include S411-S412;
step S411: performing feature extraction on the three-dimensional point cloud by using the FPFH features of the fast point feature histogram;
step S412: calculating a simplified point feature histogram SPFH of each point in k adjacent to the query point respectively, and weighting all the SPFHs into a final fast point feature histogram through weighting statistics;
the specific steps of initial position acquisition comprise S421-S425;
step S421: randomly selecting at least three non-collinear data points from a source point cloud S, and selecting a corresponding point from a target point cloud T; the corresponding points are the closest points obtained by comparing the calculated FPFH feature vectors;
step S422: according to the data points selected by the source point cloud S and the corresponding points selected by the target point cloud T, a rotation translation matrix H is calculated through a least square algorithm;
step S423: all points in the source point cloud S are subjected to a rotation translation matrix H to obtain a point cloud S ', and all points with the distance smaller than a given threshold value T are extracted to form a consistency point cloud set S by comparing the point cloud S' with a target point cloud T 1 And record this point cloud set S 1 The number of points N owned by the host;
step S424: iterating steps S421-S423 by continuously cycling, and ending the cycle if the consistency set cannot be obtained after the cycle number K is reached; otherwise, when the iteration times are smaller than K, consistency sets are found, and the consistency set with the most data point sets in the consistency sets is returned;
step S425: calculating the optimal rotation translation matrix H through the least square algorithm by all points in the consistency set of the maximum data point set _best Passing the source point cloud S through H _best The transformed position is used as an initial position for ICP registration of the source point cloud.
In a preferred embodiment, in the step S50, the preset loss function is:
，/>is the weight value of the feature point, +.>Landmark values for feature points in 2D face pictures, < >>Landmark values projected onto 2D face pictures for 3D face model landmarks.
In a preferred embodiment, in said step S50, the optimization specific step comprises S511-S513;
step S511: finding a corresponding point of the source point cloud S corresponding to the target point cloud T by calculating the distance between the points of the source point cloud S and the target point cloud T;
step S512: in the case of the known point correspondence, set，/>Respectively representing the mass centers of the source point cloud S and the target point cloud T, let +.>Let->Calculating the optimal rotational translation matrix H by SVD decomposition to obtain +.>The optimal rotation matrix is +.>The optimal translation matrix isThe method comprises the steps of carrying out a first treatment on the surface of the Wherein U, V and Σ are matrices, U is a left singular vector obtained by singular value decomposition, V is a right singular vector obtained by singular value decomposition, Σ is a matrix composed of singular values, T is a matrix transposition, and->Is the point of the source point cloud S,a point which is a target point cloud T;
step S513: obtaining a point cloud S' by transforming the source point cloud S according to a rotation and translation matrix; and (S511-S512) of loop iteration, wherein when the iteration times are smaller than the preset maximum iteration times K, if the variation of the mean square error loss of the point cloud S' and the target point cloud T is smaller than a given threshold or the calculated matching degree is larger than the given threshold, the iteration is stopped, and otherwise, the iteration is stopped after the maximum iteration times K are reached.
The second aspect of the present application provides a device for facial modeling and mask model partition matching, comprising:
the preprocessing module is used for acquiring pixel coordinate values of key feature points of a face region in the 2D face picture and a 3D face model which is pre-constructed based on a convolutional neural network;
the feature acquisition module is used for carrying out feature vectorization on the textures, the shapes and the expressions of the faces in the 2D face picture, and inputting the feature vectorization into the 3D face model for parameter training; the loss function of the step is set to output the distance between corresponding feature points of the projection of the 3D face to the 2D face picture;
the region dividing module is used for generating three-dimensional coordinate points of the face feature points according to the 2D face picture, and dividing the region of the face feature points corresponding to the 3D face model into forehead, cheek, chin, eyebrow, nose and lip regions through lower standard values of the three-dimensional coordinate points;
the feature matching module is used for extracting feature points of the 3D face model and feature vectors of the feature points generated by the 2D face picture, matching the feature points and obtaining an initial position of the 3D face model after matching, and therefore matching of the 3D face model and the 2D face picture is completed;
the optimized registration module is used for calculating the sum of the distances between the corresponding positions of the two part of feature points of the 3D face model and the 2D face picture, and optimizing the sum of the distances by adopting a random gradient descent algorithm through a preset loss function; the weight values of the areas of the forehead, the cheeks and the chin are larger than the weight values of the areas of the eyebrows, the nose and the lips;
the mask matching module is used for carrying out partition matching on the 3D face model and the 3D face mask model through a preset 3D model matching system, and storing matching results, so that special treatment is carried out on a specific area of a user when the face mask is used.
A third aspect of the present application provides a terminal comprising a memory, a processor and a computer program stored in the memory, which when executed by the processor, performs the steps of the face modeling and mask model partition matching method according to any of the embodiments described above.
A fourth aspect of the present application provides a computer readable storage medium storing a computer program which, when executed by a processor, performs the steps of a face modeling and mask model partition matching method according to any one of the embodiments described above.
A fifth aspect of the application provides a computer program product comprising a computer program or instructions which, when processed for execution, implement the steps of a face modeling and mask model partition matching method as described in any of the embodiments above.
According to the face modeling and mask model partition matching method, device, terminal and medium, the sum of distances between corresponding positions of the 3D face model and the two part characteristic points of the face picture is calculated, and the sum of the distances is optimized by a preset loss function through a random gradient descent algorithm, so that the point with the best matching effect of the 3D face and the 2D image is obtained, and the matching effect of the 3D model and the 2D image is completed. By giving higher weight values to forehead, cheek and chin areas and giving lower weight to points of eyebrow, nose, lips and other areas, when a loss function is optimized through a random gradient descent algorithm, the areas with high weight value setting can be more strongly optimized, the area with small weight value setting has smaller optimizing force, and the influence of non-key areas of the face on a matching result during matching is reduced.
Meanwhile, the 3D face model is matched with the face mask, the face is partitioned, the key treatment area is strongly matched with the face mask, and the non-key area is weakly matched with the face mask, so that the matching effect of the key area of the face and the face mask is better, the influence of the non-key area on the matching result is reduced, the face mask can provide better and targeted treatment for the key area of the face, and the treatment effect of the face mask is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a face modeling and mask model partition matching method provided by the application;
FIG. 2 is a schematic diagram of feature points of a face mask 3D model and a face 3D model in an initial position;
FIG. 3 is a schematic diagram of the positions of the facial mask 3D model after feature points of the facial mask 3D model are matched;
FIG. 4 is a schematic diagram of feature point screening after a 3D model of a face mask is matched with feature points of a 3D face;
fig. 5 is a frame diagram of a method and a device for facial modeling and mask model partition matching provided by the application.
Detailed Description
In order to make the objects, technical solutions and advantageous technical effects of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and detailed description. It should be understood that the detailed description is intended to illustrate the application, and not to limit the application.
It is also to be understood that the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
Example 1
In this embodiment of the present application, a method for partition matching of a face modeling and a mask model is provided, which is used for building a 3D face model of a user according to 2D face pictures in a targeted manner, and precisely matching the face model with a cosmetic mask, so that when the cosmetic mask is used, special treatment is performed for a specific area of the user, and the treatment effect of the cosmetic mask is improved.
As shown in FIG. 1, the face modeling and mask model partition matching method includes steps S10-S60.
Step S10: and acquiring pixel coordinate values of key feature points of a face region in the 2D face picture, and pre-constructing a 3D face model based on a convolutional neural network. The application takes the 2D face picture as a data set, and trains to obtain a model for generating the corresponding 3D face.
The 2D face picture may be obtained by capturing a user's face, including but not limited to frontal shooting, large area shielding (e.g., wearing a mask), small area shielding (e.g., hair shielding a side face), and the like.
Step S20: carrying out characteristic vectorization on the texture, shape and expression of a face in the 2D face picture, and inputting the characteristic vectorization into a 3D face model for parameter training; the loss function in the step is set to output the distance between corresponding feature points of the projection of the 3D face to the 2D face picture.
Specifically, feature vectorizing the texture, shape and expression of the face in the 2D face picture includes: the feature vectors of the texture T and the shape S are obtained by the following formula:
，/>is the average face shape,/->Is the average facial texture,/->、/>、The basis vectors of features, expressions and textures processed by PCA (Principal components analysis, principal component analysis) respectively can refer to the data of the prior art, such as the data obtained by 3D scanning 100 men and 100 women. />、/>、/>Generating corresponding coefficient vectors of the 3D face model; wherein the shape-related parameters are obtained by MLP (Multilayer Perceptron, multi-layer perceptron) training, and the texture-related coefficients are obtained by CNN (Convolutional Neural Network ) training.
Therefore, by vectorizing the characteristics of the face in the 2D face picture, such as texture, shape, expression and the like, training parameters of the face by using a convolutional neural network, and obtaining the parameters which can be fully fitted with the characteristics of the 2D face picture after training, the output 3D face model is closer to the characteristics of the face, such as texture, shape, expression and the like of the 2D face picture.
Step S30: and generating three-dimensional coordinate points of the face characteristic points according to the 2D face picture, and dividing the region of the 3D face model corresponding to the face characteristic points into forehead, cheek, chin, eyebrow, nose and lip regions according to lower standard values of the three-dimensional coordinate points.
Specifically, 468 feature points of the face of the 2D face picture clock can be positioned by adopting a mediaapipe face recognition algorithm, and three-dimensional coordinate points of each feature point are returned.
Step S40: extracting feature points of the 3D face model and feature vectors of feature points generated by the 2D face picture, matching, and obtaining an initial position of the 3D face model after matching, so that matching of the 3D face model and the 2D face picture is completed.
Further, before step S40, the method further includes the steps of: and reducing the number of vertexes of the 3D face model generated according to the 2D face picture through downsampling. Specifically, downsampling is performed by adopting a point cloud voxelization mode, the size of a proper unit voxel is preset, the point cloud is put into a unit voxel box with the preset size, and if a local point group with a preset range falls into a unit voxel box, all points in the unit voxel box are represented by a gravity center point. Therefore, the number of vertexes of the preliminarily generated 3D face model is reduced through downsampling, so that the subsequent calculation amount is reduced, point cloud data can be greatly filtered through the downsampling process, and the model body state still maintains integrity.
In this step S40, specific steps of feature matching include S411 to S412.
Step S411: the three-dimensional point cloud is feature extracted using the fast point feature histogram FPFH (Fast Point Feature Histograms) features.
Step S412: and calculating a simplified point characteristic histogram SPFH of each point in the k neighborhood of the query point respectively, and weighting all the SPFHs into a final fast point characteristic histogram through weighting statistics.
Thus, the computational complexity of the algorithm is reduced to O (nk) by FPFH, but the majority of the recognition features are retained anyway. The default FPFH implementation uses 11 statistical subintervals, the feature histograms are computed separately, and then combined to obtain a 33-element feature vector for the floating point value.
In this step S40, specific steps of initial position acquisition include S421 to S425.
Step S421: randomly selecting at least three non-collinear data points from a source point cloud S, and selecting a corresponding point from a target point cloud T; the corresponding point is the closest point obtained by comparing the calculated FPFH characteristic vectors.
Step S422: and calculating a rotation translation matrix H through a least square algorithm according to the data points selected by the source point cloud S and the corresponding points selected by the target point cloud T.
Step S423: all points in the source point cloud S are subjected to a rotation translation matrix H to obtain a point cloud S ', and all points with the distance smaller than a given threshold value T are extracted to form a consistency point cloud set S by comparing the point cloud S' with a target point cloud T 1 And record this point cloud set S 1 N, the number of points owned by the host.
Step S424: iterating steps S421-S423 by continuously cycling, and ending the cycle if the consistency set cannot be obtained after the cycle number K is reached; otherwise, when the iteration number is smaller than K, consistency sets are found, and the consistency set with the most data point sets in the consistency sets is returned. The number of cycles K may be predetermined to avoid trapping dead cycles.
Step S425: calculating the optimal rotation translation matrix H through the least square algorithm by all points in the consistency set of the maximum data point set _best Passing the source point cloud S through H _best The transformed position is used as the initial position for ICP (Iterative Closest Point, based on the data registration method, using the nearest point search method, thereby solving the problem of registration based on an algorithm of the freeform curved surface) in the follow-up of the source point cloud.
Step S50: calculating the sum of distances between corresponding positions of the two parts of feature points of the 3D face model and the 2D face picture, and optimizing the sum of the distances by adopting a random gradient descent algorithm through a preset loss function; wherein the weight value of the given forehead, cheek and chin area is larger than the weight of the given eyebrow, nose and lip area.
Specifically, in this embodiment, the preset loss function is:，is the weight value of the feature point, +.>Landmark values for feature points in 2D face pictures, < >>Landmark values projected onto 2D face pictures for 3D face model landmarks. By giving higher weight values to forehead, cheek and chin areas and giving lower weight to points of eyebrow, nose, lips and other areas, when a loss function is optimized by a random gradient descent algorithm, the areas with high weight value setting can be more strongly optimized, the area with small weight value setting has smaller optimizing force, and the influence of non-key areas of the face on a matching result during matching is reduced.
In this step S50, the optimization specific step includes S511-S513. The optimization process is a process of performing ICP registration based on the initial position. The point with the best matching effect between the 3D face model and the 2D face picture is obtained by continuously optimizing the sum value of the distances, so that the effect of more accurate matching between the 3D face model and the 2D face picture is realized.
Step S511: finding a corresponding point of the source point cloud S corresponding to the target point cloud T by calculating the distance between the points of the source point cloud S and the target point cloud T;
step S512: in the case of the known point correspondence, set，/>Respectively representing the mass centers of the source point cloud S and the target point cloud T, let +.>Let->Calculating an optimal rotational translation matrix H by SVD decomposition (singular value decomposition ) to obtain +.>The optimal rotation matrix is +.>The optimal translation matrix is +.>The method comprises the steps of carrying out a first treatment on the surface of the Wherein U, V and Σ are matrices, U is a left singular vector obtained by singular value decomposition, V is a right singular vector obtained by singular value decomposition, Σ is a matrix composed of singular values, T is a matrix transposition, and->For the point of the source point cloud S +.>A point which is a target point cloud T;
step S513: obtaining a point cloud S' by transforming the source point cloud S according to a rotation and translation matrix; and (S511-S512) of loop iteration, wherein when the iteration times are smaller than the preset maximum iteration times K, if the variation of the mean square error loss of the point cloud S' and the target point cloud T is smaller than a given threshold or the calculated matching degree is larger than the given threshold, the iteration is stopped, and otherwise, the iteration is stopped after the maximum iteration times K are reached.
Step S60: and carrying out partition matching on the 3D face model and the 3D beauty mask model through a preset 3D model matching system, and storing matching results, so that special treatment is carried out on a specific area of a user when the beauty mask is used.
The 3D face model is matched with the face mask as shown in fig. 2-4, the face is partitioned, the key treatment area is strongly matched with the face mask, and the non-key area is weakly matched with the face mask, so that the matching effect of the key area of the face and the face mask is better, the influence of the non-key area matching result is reduced, the face mask can provide better and targeted treatment for the key area of the face, and the treatment effect of the face mask is improved.
Example two
The embodiment provides a facial modeling and mask model partition matching device 100, which is in one-to-one correspondence with the steps of the method, and is used for constructing a 3D facial model of a user in a targeted manner according to a 2D facial picture, and accurately matching the facial model with a cosmetic mask. It should be noted that the functional roles, implementation principles and specific embodiments of the device are consistent with the above-mentioned facial modeling and mask model partition matching method, so that the following description is omitted.
As shown in fig. 5, the face modeling and mask model partition matching apparatus 100 includes:
the preprocessing module 10 is used for acquiring pixel coordinate values of key feature points of a face region in the 2D face picture and a 3D face model pre-constructed based on a convolutional neural network;
the feature acquisition module 20 is configured to perform feature vectorization on textures, shapes and expressions of faces in the 2D face picture, and input the feature vectorization into a 3D face model for parameter training; the loss function of the step is set to output the distance between corresponding feature points of the projection of the 3D face to the 2D face picture;
the region dividing module 30 is configured to generate three-dimensional coordinate points of face feature points according to the 2D face picture, and divide the region of the 3D face model corresponding to the face feature points into forehead, cheek, chin, eyebrow, nose and lip regions according to lower label values of the three-dimensional coordinate points;
the feature matching module 40 is configured to extract feature points of the 3D face model and feature vectors of feature points generated by the 2D face picture, match the feature points and obtain an initial position of the 3D face model after matching, thereby completing matching between the 3D face model and the 2D face picture;
the optimized registration module 50 is configured to calculate a sum of distances between corresponding positions of two feature points of the 3D face model and the 2D face picture, and optimize the sum of distances by using a random gradient descent algorithm through a preset loss function; the weight values of the areas of the forehead, the cheeks and the chin are larger than the weight values of the areas of the eyebrows, the nose and the lips;
the mask matching module 60 is configured to perform partition matching on the 3D face model and the 3D cosmetic mask model through a preset 3D model matching system, and save the matching result, so that when the cosmetic mask is used, special treatment is performed on a specific area of a user.
Example III
The present embodiment provides a terminal, where the terminal includes a memory, a processor, and a computer program stored in the memory, where the computer program when executed by the processor implements the steps of the face modeling and mask model partition matching method according to any one of the foregoing embodiments.
Example IV
The present embodiment provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the face modeling and mask model partition matching method according to any one of the above embodiments.
Example five
The present embodiments provide a computer program product comprising a computer program or instructions which, when processed for execution, implement the steps of a face modeling and mask model partition matching method as described in any one of the above embodiments.
In summary, according to the face modeling and mask model partition matching method, device, terminal and medium provided by the application, the point with the best matching effect between the 3D face and the 2D image is obtained by calculating the sum of the distances between the corresponding positions of the two part of feature points of the 3D face model and the face picture and optimizing the sum of the distances by adopting a random gradient descent algorithm through a preset loss function, so that the matching effect between the 3D model and the 2D image is completed. By giving higher weight values to forehead, cheek and chin areas and giving lower weight to points of eyebrow, nose, lips and other areas, when a loss function is optimized through a random gradient descent algorithm, the areas with high weight value setting can be more strongly optimized, the area with small weight value setting has smaller optimizing force, and the influence of non-key areas of the face on a matching result during matching is reduced.
Meanwhile, the 3D face model is matched with the face mask, the face is partitioned, the key treatment area is strongly matched with the face mask, and the non-key area is weakly matched with the face mask, so that the matching effect of the key area of the face and the face mask is better, the influence of the non-key area on the matching result is reduced, the face mask can provide better and targeted treatment for the key area of the face, and the treatment effect of the face mask is improved.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the system is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the elements and method steps of the examples described in connection with the embodiments disclosed herein can be implemented as electronic hardware, or as a combination of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed system or apparatus/terminal device and method may be implemented in other manners. For example, the system or apparatus/terminal device embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical function division, and there may be additional divisions in actual implementation, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, systems or units, which may be in electrical, mechanical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The present application is not limited to the details and embodiments described herein, and thus additional advantages and modifications may readily be made by those skilled in the art, without departing from the spirit and scope of the general concepts defined in the claims and the equivalents thereof, and the application is not limited to the specific details, representative apparatus and illustrative examples shown and described herein.

Claims (9)

1. A method for partition matching of face modeling and mask model, comprising:
step S10: acquiring pixel coordinate values of key feature points of a face region in a 2D face picture, and pre-constructing a 3D face model based on a convolutional neural network;
step S20: carrying out feature vectorization on the texture, shape and expression of the face in the 2D face picture, and inputting the feature vectorization into the 3D face model for parameter training; the loss function of the step is set to output the distance between corresponding feature points of the projection of the 3D face to the 2D face picture;
step S30: generating three-dimensional coordinate points of the face feature points according to the 2D face picture, and dividing the region of the 3D face model corresponding to the face feature points into forehead, cheek, chin, eyebrow, nose and lip regions through lower standard values of the three-dimensional coordinate points;
step S40: extracting feature points of the 3D face model and feature vectors of the feature points generated by the 2D face picture, matching the feature points and the feature vectors, and obtaining an initial position of the 3D face model after matching, so that matching of the 3D face model and the 2D face picture is completed;
step S50: calculating the sum of distances between the corresponding positions of the two parts of feature points of the 3D face model and the 2D face picture, and optimizing the sum of the distances by adopting a random gradient descent algorithm through a preset loss function; the weight values of the areas of the forehead, the cheeks and the chin are larger than the weight values of the areas of the eyebrows, the nose and the lips;
step S60: the 3D face model and the 3D beauty mask model are subjected to partition matching through a preset 3D model matching system, and matching results are stored, so that special treatment is conveniently carried out on a specific area of a user when the beauty mask is used;
specifically, in the step S40,
the specific steps of feature matching include S411-S412;
step S411: performing feature extraction on the three-dimensional point cloud by using the FPFH features of the fast point feature histogram;
step S412: calculating a simplified point feature histogram SPFH of each point in k adjacent to the query point respectively, and weighting all the SPFHs into a final fast point feature histogram through weighting statistics;
the specific steps of initial position acquisition comprise S421-S425;
step S421: randomly selecting at least three non-collinear data points from a source point cloud S, and selecting a corresponding point from a target point cloud T; the corresponding points are the closest points obtained by comparing the calculated FPFH feature vectors;
step S422: according to the data points selected by the source point cloud S and the corresponding points selected by the target point cloud T, a rotation translation matrix H is calculated through a least square algorithm;
step S423: all points in the source point cloud S are subjected to a rotation translation matrix H to obtain a point cloud S ', and all points with the distance smaller than a given threshold value T are extracted to form a consistency point cloud set S by comparing the point cloud S' with a target point cloud T 1 And record this point cloud set S 1 The number of points N owned by the host;
step S424: iterating steps S421-S423 by continuously cycling, and ending the cycle if the consistency set cannot be obtained after the cycle number K is reached; otherwise, when the iteration times are smaller than K, consistency sets are found, and the consistency set with the most data point sets in the consistency sets is returned;
step S425: calculating the optimal rotation translation matrix H through the least square algorithm by all points in the consistency set of the maximum data point set _best Passing the source point cloud S through H _best The transformed position is used as an initial position for ICP registration of the source point cloud.
2. The method for face modeling and mask model partition matching according to claim 1, wherein in step S20, feature vectorizing the texture, shape and expression of the face in the 2D face picture comprises:
the feature vectors of the texture T and the shape S are obtained by the following formula:
，/>is the average face shape,/->Is the average face texture and is used to determine,
、/>、/>the feature, expression and texture are respectively the basis vectors processed by PCA; />、/>、/>Generating corresponding coefficient vectors of the 3D face model; the shape-related parameters are obtained through MLP multi-layer perceptron training, and the texture-related coefficients are obtained through CNN convolutional neural network training.
3. The method for face modeling and mask model partition matching according to claim 1, further comprising, prior to said step S40, the steps of:
reducing the number of vertexes of the 3D face model generated according to the 2D face picture through downsampling; the method specifically comprises the following steps:
the size of a unit voxel is preset, the point cloud is put into a unit voxel box with the preset size, and if the local point group with a preset range falls into a unit voxel box, all points in the unit voxel box are represented by a gravity center point.
4. The method for face modeling and mask model partition matching according to claim 1, wherein in the step S30, generating three-dimensional coordinate points of face feature points from the 2D face picture includes:
and positioning 468 characteristic points of the face of the 2D face picture clock by adopting a mediaapipe face recognition algorithm, and returning to the three-dimensional coordinate point of each characteristic point.
5. The method for partition matching between face modeling and mask model according to claim 1, wherein in step S50, the predetermined loss function is:
，/>is the weight value of the feature point, +.>Landmark values for feature points in 2D face pictures, < >>Landmark values projected onto 2D face pictures for 3D face model landmarks。
6. The face modeling and mask model partition matching method according to claim 5, wherein in the step S50, the optimizing step includes S511-S513;
step S511: finding a corresponding point of the source point cloud S corresponding to the target point cloud T by calculating the distance between the points of the source point cloud S and the target point cloud T;
step S512: in the case of the known point correspondence, set ，/>Respectively representing the mass centers of the source point cloud S and the target point cloud T, let +.>Let->Calculating the optimal rotational translation matrix H by SVD decomposition to obtain +.>The optimal rotation matrix is +.>The optimal translation matrix isThe method comprises the steps of carrying out a first treatment on the surface of the Wherein U, V and Σ are matrices, U is a left singular vector obtained by singular value decomposition, V is a right singular vector obtained by singular value decomposition, Σ is a matrix composed of singular values, T is a matrix transposition, and->Is the point of the source point cloud S,a point which is a target point cloud T;
step S513: obtaining a point cloud S' by transforming the source point cloud S according to a rotation and translation matrix; and (S511-S512) of loop iteration, wherein when the iteration times are smaller than the preset maximum iteration times K, if the variation of the mean square error loss of the point cloud S' and the target point cloud T is smaller than a given threshold or the calculated matching degree is larger than the given threshold, the iteration is stopped, and otherwise, the iteration is stopped after the maximum iteration times K are reached.
7. A facial modeling and mask model partition matching device, comprising:
the preprocessing module is used for acquiring pixel coordinate values of key feature points of a face region in the 2D face picture and a 3D face model which is pre-constructed based on a convolutional neural network;
the feature acquisition module is used for carrying out feature vectorization on the textures, the shapes and the expressions of the faces in the 2D face picture, and inputting the feature vectorization into the 3D face model for parameter training; the loss function of the step is set to output the distance between corresponding feature points of the projection of the 3D face to the 2D face picture;
the region dividing module is used for generating three-dimensional coordinate points of the face feature points according to the 2D face picture, and dividing the region of the face feature points corresponding to the 3D face model into forehead, cheek, chin, eyebrow, nose and lip regions through lower standard values of the three-dimensional coordinate points;
the feature matching module is used for extracting feature points of the 3D face model and feature vectors of the feature points generated by the 2D face picture, matching the feature points and obtaining an initial position of the 3D face model after matching, and therefore matching of the 3D face model and the 2D face picture is completed;
the optimized registration module is used for calculating the sum of the distances between the corresponding positions of the two part of feature points of the 3D face model and the 2D face picture, and optimizing the sum of the distances by adopting a random gradient descent algorithm through a preset loss function; the weight values of the areas of the forehead, the cheeks and the chin are larger than the weight values of the areas of the eyebrows, the nose and the lips;
the mask matching module is used for carrying out partition matching on the 3D face model and the 3D cosmetic mask model through a preset 3D model matching system, and storing matching results, so that special treatment is carried out on a specific area of a user when the cosmetic mask is used;
wherein, in the feature matching module:
the specific steps of feature matching include S411-S412;
step S411: performing feature extraction on the three-dimensional point cloud by using the FPFH features of the fast point feature histogram;
step S412: calculating a simplified point feature histogram SPFH of each point in k adjacent to the query point respectively, and weighting all the SPFHs into a final fast point feature histogram through weighting statistics;
the specific steps of initial position acquisition comprise S421-S425;
step S421: randomly selecting at least three non-collinear data points from a source point cloud S, and selecting a corresponding point from a target point cloud T; the corresponding points are the closest points obtained by comparing the calculated FPFH feature vectors;
step S422: according to the data points selected by the source point cloud S and the corresponding points selected by the target point cloud T, a rotation translation matrix H is calculated through a least square algorithm;
step S423: all points in the source point cloud S are subjected to a rotation translation matrix H to obtain a point cloud S ', and all points with the distance smaller than a given threshold value T are extracted to form a consistency point cloud set S by comparing the point cloud S' with a target point cloud T 1 And record this point cloud set S 1 The number of points N owned by the host;
step S424: iterating steps S421-S423 by continuously cycling, and ending the cycle if the consistency set cannot be obtained after the cycle number K is reached; otherwise, when the iteration times are smaller than K, consistency sets are found, and the consistency set with the most data point sets in the consistency sets is returned;
step S425: all of the consistency sets through this maximum set of data pointsThe optimal rotation translation matrix H is calculated by the point through a least square algorithm _best Passing the source point cloud S through H _best The transformed position is used as an initial position for ICP registration of the source point cloud.
8. A terminal comprising a memory, a processor, and a computer program stored in the memory, which when executed by the processor, performs the steps of the face modeling and mask model partition matching method of any one of claims 1-6.
9. A computer readable storage medium storing a computer program which when executed by a processor performs the steps of the face modeling and mask model partition matching method of any of claims 1-6.
CN202310961964.1A 2023-08-02 2023-08-02 Face modeling and mask model partition matching method, device, terminal and medium Active CN116665284B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310961964.1A CN116665284B (en) 2023-08-02 2023-08-02 Face modeling and mask model partition matching method, device, terminal and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310961964.1A CN116665284B (en) 2023-08-02 2023-08-02 Face modeling and mask model partition matching method, device, terminal and medium

Publications (2)

Publication Number Publication Date
CN116665284A CN116665284A (en) 2023-08-29
CN116665284B true CN116665284B (en) 2023-11-28

Family

ID=87714007

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310961964.1A Active CN116665284B (en) 2023-08-02 2023-08-02 Face modeling and mask model partition matching method, device, terminal and medium

Country Status (1)

Country Link
CN (1) CN116665284B (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112002014A (en) * 2020-08-31 2020-11-27 中国科学院自动化研究所 Three-dimensional face reconstruction method, system and device for fine structure
CN113628263A (en) * 2021-09-06 2021-11-09 西安热工研究院有限公司 Point cloud registration method based on local curvature and neighbor characteristics thereof
WO2022089360A1 (en) * 2020-10-28 2022-05-05 广州虎牙科技有限公司 Face detection neural network and training method, face detection method, and storage medium
CN116311474A (en) * 2023-04-18 2023-06-23 厦门美图之家科技有限公司 Face image face filling method, system and storage medium
CN116433812A (en) * 2023-06-08 2023-07-14 海马云（天津）信息技术有限公司 Method and device for generating virtual character by using 2D face picture

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11443484B2 (en) * 2020-05-15 2022-09-13 Microsoft Technology Licensing, Llc Reinforced differentiable attribute for 3D face reconstruction

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112002014A (en) * 2020-08-31 2020-11-27 中国科学院自动化研究所 Three-dimensional face reconstruction method, system and device for fine structure
WO2022089360A1 (en) * 2020-10-28 2022-05-05 广州虎牙科技有限公司 Face detection neural network and training method, face detection method, and storage medium
CN113628263A (en) * 2021-09-06 2021-11-09 西安热工研究院有限公司 Point cloud registration method based on local curvature and neighbor characteristics thereof
CN116311474A (en) * 2023-04-18 2023-06-23 厦门美图之家科技有限公司 Face image face filling method, system and storage medium
CN116433812A (en) * 2023-06-08 2023-07-14 海马云（天津）信息技术有限公司 Method and device for generating virtual character by using 2D face picture

Also Published As

Publication number Publication date
CN116665284A (en) 2023-08-29

Similar Documents

Publication Publication Date Title
Jeni et al. Dense 3D face alignment from 2D videos in real-time
Roth et al. Adaptive 3D face reconstruction from unconstrained photo collections
KR101007276B1 (en) Three dimensional face recognition
JP4653606B2 (en) Image recognition apparatus, method and program
JP4950787B2 (en) Image processing apparatus and method
US20100295854A1 (en) Viewpoint-invariant image matching and generation of three-dimensional models from two-dimensional imagery
US10489977B2 (en) Method for establishing a deformable 3D model of an element, and associated system
US20080309662A1 (en) Example Based 3D Reconstruction
JP6207210B2 (en) Information processing apparatus and method
CN109684969B (en) Gaze position estimation method, computer device, and storage medium
KR20090092473A (en) 3D Face Modeling Method based on 3D Morphable Shape Model
CN108564619B (en) Realistic three-dimensional face reconstruction method based on two photos
CN111815768B (en) Three-dimensional face reconstruction method and device
Anbarjafari et al. 3D face reconstruction with region based best fit blending using mobile phone for virtual reality based social media
Achenbach et al. Accurate Face Reconstruction through Anisotropic Fitting and Eye Correction.
Amstutz et al. PCA based 3D shape reconstruction of human foot using multiple viewpoint cameras
Ferková et al. Age and gender-based human face reconstruction from single frontal image
KR20190069750A (en) Enhancement of augmented reality using posit algorithm and 2d to 3d transform technique
Chen et al. Single and sparse view 3d reconstruction by learning shape priors
CN111862278A (en) Animation obtaining method and device, electronic equipment and storage medium
Liu et al. Face geometry and appearance modeling: concepts and applications
Kang et al. Appearance-based structure from motion using linear classes of 3-d models
Kahraman et al. Robust face alignment for illumination and pose invariant face recognition
Stylianou et al. Image based 3d face reconstruction: a survey
CN116665284B (en) Face modeling and mask model partition matching method, device, terminal and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant