KR20230098313A - Facial reconstruction methods, devices, computer devices and storage media - Google Patents

Abstract

The present invention provides a face reconstruction method, apparatus, computer device, and storage medium, wherein the method comprises: obtaining dense point cloud data of an original face included in a target image; Fitting the dense point cloud data of the original face using the dense point cloud data of the first reference face corresponding to each of the plurality of reference images to obtain a fitting coefficient corresponding to each of the dense point cloud data of the first reference face of the plurality of groups step; Determining dense point cloud data of a target facial model based on fitting coefficients corresponding to each of the dense point cloud data of a plurality of groups of second reference faces and the dense point cloud data of a plurality of groups of first reference faces having a preset style step, the second reference faces of the plurality of groups are generated based on the first reference faces in the plurality of reference images; and generating a target face model corresponding to the original face of the target image based on the dense point cloud data of the target face model.

Description

Facial reconstruction methods, devices, computer devices and storage media

[Cross-citation of related applications]

This application claims priority of the Chinese patent application filed on November 25, 2020, application number 202011337942.0, titled "Facial reconstruction method, device, computer device and storage medium", all contents of which are hereby incorporated by reference. incorporated into this application through

[Technical field]

TECHNICAL FIELD The present invention relates to the field of image processing technology, and more particularly to facial reconstruction methods, apparatus, computer devices, and storage media.

In general, by constructing a 3D model of a real face or a virtual face according to one's preference and realizing face reconstruction, it can be widely applied to fields such as games, animations, and virtual socials. For example, in a game, a game player generates a virtual face 3D model according to a real face included in an image provided by the game player through a face reconstruction system provided by a game program, and uses the generated virtual face 3D model. This allows you to be immersed in the game.

Currently, the similarity between a virtual face 3D model obtained based on a face reconstruction method and a real face is relatively low.

Embodiments of the present invention provide at least a facial reconstruction method, apparatus, computer device, and storage medium.

In a first aspect, an embodiment of the present invention provides a face reconstruction method, which includes: obtaining dense point cloud data of an original face included in a target image; Fitting coefficients corresponding to each of the dense point cloud data of the first reference face of a plurality of groups by fitting the dense point cloud data of the original face using the dense point cloud data of the first reference face corresponding to each of the plurality of reference images obtaining; Determining dense point cloud data of a target face model based on dense point cloud data of a plurality of groups of second reference faces having a preset style and fitting coefficients corresponding to each of the plurality of groups of dense point cloud data of the first reference face but; second reference faces of the plurality of groups are generated based on first reference faces in the plurality of reference images; and generating a target face model corresponding to the original face of the target image based on dense point cloud data of the target face model.

In the above embodiment, a correlation between the original face's dense point cloud data and the plurality of first reference face's dense point cloud data is established using the fitting coefficient as a medium. The target facial model generated by characterizing the association between the dense point cloud data of the second reference face determined based on the point cloud data and the dense point cloud data of the target facial model built based on the original face is the original face in the target image. By having features (eg, shape features, etc.), it is possible to have a higher degree of similarity with the original face, and it is possible to make the created target face model have a preset style.

In one optional embodiment, a target face based on a fitting coefficient corresponding to each of the dense point cloud data of a plurality of groups of second reference faces having a preset style and the plurality of groups of dense point cloud data of the first reference face. The determining of the dense point cloud data of the model may include generating average value data of a plurality of groups of the second reference face dense point cloud data based on the plurality of groups of the second reference face dense point cloud data; Based on the dense point cloud data of the second reference face of the plurality of groups, the average value data, and the fitting coefficient corresponding to each of the plurality of groups of the dense point cloud data of the first reference face of the plurality of groups, the dense point cloud data of the target face model It includes the steps of creating

In one optional embodiment, the target based on a fitting coefficient corresponding to each of the plurality of groups of dense point cloud data of the second reference face, the average value data, and the plurality of groups of dense point cloud data of the first reference face The generating of the dense point cloud data of the facial model includes the dense point cloud data of the second reference face of each group among the dense point cloud data of the second reference face of a plurality of groups and the average value data of each group based on the average value data. determining difference value data of dense point cloud data of the second reference face; Performing interpolation on difference value data corresponding to each of the plurality of groups of dense point cloud data of the second reference face based on fitting coefficients corresponding to each of the plurality of groups of dense point cloud data of the first reference face. ; and generating dense point cloud data of the target facial model based on a result of the interpolation process and the average value data.

In the above embodiment, the average feature of the dense point cloud data of the plurality of groups of the second reference face can be accurately expressed using the average value data; Each difference between the dense point cloud data of the second reference face of each group and the average feature of the dense point cloud data of the second reference face of the plurality of groups using the difference value data of the dense point cloud data of the second reference face of each group , it is possible to more simply determine the dense point cloud data of the target facial model by adjusting the average value data using more accurate difference data.

In one optional embodiment, acquiring the dense point cloud data of the original face included in the target image includes acquiring the target image including the original face; and obtaining dense point cloud data of the original face in the target image by processing the target image with a pretrained neural network.

In the above embodiment, the facial features of the original face in the target image can be represented more accurately by using the dense point cloud data of the original face.

In one optional embodiment, the method includes a method of obtaining a plurality of reference images including a first reference face; For each of the reference images in the plurality of reference images, a method of obtaining dense point cloud data of the first reference face in each reference image by processing each reference image with a pre-trained neural network Reference of the plurality of sheets Dense point cloud data of the first reference face corresponding to each image is acquired.

In the above embodiment, facial features respectively corresponding to the first reference face in the reference image may be more accurately represented using dense point cloud data of the first reference face. At the same time, it is possible to cover facial appearance features as broadly as possible by using a plurality of reference images including the first reference face.

In one optional embodiment, the dense point cloud data of the original face is fitted using the dense point cloud data of the first reference face corresponding to each of the plurality of reference images to obtain a plurality of groups of dense point cloud data of the first reference face The obtaining of fitting coefficients corresponding to each of the cloud data may include performing least square processing on the density point cloud data of the original face and the dense point cloud data of the first reference face to obtain a plurality of groups of dense point cloud data of the first reference face obtaining intermediate coefficients corresponding to each point cloud data; and determining fitting coefficients corresponding to the dense point cloud data of the first reference face of each group based on intermediate coefficients corresponding to the dense point cloud data of the first reference face of each group.

In the above embodiment, it is possible to accurately represent a fitting situation when fitting the dense point cloud data of the original face using the dense point cloud data of the plurality of first reference faces using the fitting coefficient.

In one optional embodiment, determining a fitting coefficient corresponding to the dense point cloud data of the first reference face of each group based on an intermediate coefficient corresponding to the dense point cloud data of the first reference face of each group. Determining a first type of dense point cloud data corresponding to a part of a target facial model among the first reference faces from the dense point cloud data of the first reference faces of each group; obtaining first fitting coefficients by adjusting intermediate coefficients corresponding to the first type of dense point cloud data among the dense point cloud data of the first reference face; determining an intermediate coefficient corresponding to a second type of dense point cloud data among the dense point cloud data of the first reference face as a second fitting coefficient; the second type of dense point cloud data is dense point cloud data excluding the first type of dense point cloud data from the dense point cloud data of the first reference face; and obtaining a fitting coefficient of the dense point cloud data of the first reference face of each group based on the first fitting coefficient and the second fitting coefficient.

In one optional embodiment, the method includes a method of obtaining dense point cloud data of a second reference face having the preset style by adjusting dense point cloud data of a first reference face in the reference image; or, generating a virtual face image including a second reference face having the preset style based on a first reference face in the reference image; Dense point cloud data of the second reference face having a preset style is acquired through a method of generating dense point cloud data of the second reference face in the virtual face image using a pre-trained neural network.

In the above embodiment, by adjusting the fitting coefficients corresponding to some facial parts, the dense point cloud data obtained by fitting based on the fitting coefficients and the plurality of corresponding dense point cloud data of the first reference face correspond to the original face corresponding to the target image , that is, the obtained fitting coefficient can more accurately represent the coefficient when the first reference facial dense point cloud fits the original facial dense point cloud.

In one optional embodiment, training the neural network comprises: obtaining a set of sample images; The sample image set includes a plurality of first sample images including a first sample face; The plurality of first sample images are partitioned into a plurality of subsets of first sample images, and in each subset of first sample images, first sample facial expressions having the same type of expression obtained by collecting each from a plurality of predetermined collection angles are included. Steps with images; obtaining dense point cloud data of a first sample face of a first sample image in the sample image set; obtaining predicted density point cloud data of a first sample face of the first sample image by performing feature learning on a first sample image in the sample image set with the neural network; and training the neural network using the dense point cloud data of the first sample face and the predicted dense point cloud data.

In one optional embodiment, the sample image set further includes a plurality of second sample images including a second sample face and a background, and the step of training the neural network includes a face of each of the second sample images. obtaining keypoint data; fitting and generating dense point cloud data of a second sample face of the second sample image using facial keypoint data of a second sample image and the second sample image; obtaining predicted density point cloud data of a second sample face of the second sample image by performing feature learning on a second sample image in the sample image set with the neural network; The method may further include training the neural network using the dense point cloud data of the second sample face and the predicted dense point cloud data.

In one optional embodiment, the set of sample images further includes a third sample image; the third sample image is obtained by performing data augmentation processing on the first sample image;

Training the neural network,

obtaining dense point cloud data of a third sample face of the third sample image; obtaining predicted density point cloud data of a third sample face of the third sample image by performing feature learning on the third sample image with the neural network;

The method may further include training the neural network using the dense point cloud data of the third sample face and the predicted dense point cloud data.

In one optional embodiment, the data augmentation process includes at least one of random occlusion process, Gaussian noise process, motion blur process and color gamut channel change process.

In the above embodiments, a better neural network is obtained according to actual needs by adjusting the quantity of the first sample image, the second sample image, and the third sample image to obtain neural networks having different advantages; At the same time, since the third sample image is obtained through data augmentation processing, when the sample image includes the third sample image, the ability of the neural network obtained by training to process data becomes stronger.

At the same time, since the face included in the second sample image can cover facial appearance features as broadly as possible, the acquired neural network can have relatively good generalization ability.

In a second aspect, an embodiment of the present invention further provides a facial reconstruction device, which includes:

a first acquiring module for obtaining dense point cloud data of an original face included in a target image;

Fitting coefficients corresponding to each of the dense point cloud data of the first reference face of a plurality of groups by fitting the dense point cloud data of the original face using the dense point cloud data of the first reference face corresponding to each of the plurality of reference images A first processing module to obtain;

Determining dense point cloud data of a target face model based on dense point cloud data of a plurality of groups of second reference faces having a preset style and fitting coefficients corresponding to each of the plurality of groups of dense point cloud data of the first reference face but; a determination module, wherein each of the plurality of groups of second reference faces is generated based on a first reference face in the plurality of reference images;

and a generation module for generating a target face model corresponding to the original face of the target image based on dense point cloud data of the target face model.

In one optional embodiment, the determining module determines the fitting coefficient corresponding to each of the plurality of groups of dense point cloud data of the second reference face and the plurality of groups of the dense point cloud data of the first reference face having a preset style. when determining the dense point cloud data of the target facial model based on the above, generating average value data of the dense point cloud data of the second reference face of the plurality of groups based on the dense point cloud data of the second reference face of the plurality of groups; Based on the dense point cloud data of the second reference face of the plurality of groups, the average value data, and the fitting coefficient corresponding to each of the plurality of groups of the dense point cloud data of the first reference face of the plurality of groups, the dense point cloud data of the target face model generate

In one optional embodiment, the determining module determines a plurality of groups of dense point cloud data of the second reference face, the average value data, and a plurality of groups of fitting coefficients corresponding to each of the dense point cloud data of the first reference face. Based on the dense point cloud data of the second reference face of each group among the dense point cloud data of the second reference face of a plurality of groups and the average value data determine difference value data of the dense point cloud data of the second reference face of each group; performing interpolation on difference value data corresponding to each of the plurality of groups of dense point cloud data of the second reference face based on fitting coefficients corresponding to each of the plurality of groups of dense point cloud data of the first reference face; Dense point cloud data of the target facial model is generated based on the result of the interpolation process and the average value data.

In one optional embodiment, the first acquiring module, when obtaining dense point cloud data of an original face included in the target image, acquires the target image including the original face; Dense point cloud data of the original face in the target image is obtained by processing the target image with a pre-trained neural network.

In one optional embodiment, the device includes: a method of obtaining a plurality of reference images including a first reference face; For each of the reference images in the plurality of reference images, a method of obtaining dense point cloud data of the first reference face in each reference image by processing each reference image with a pre-trained neural network Reference of the plurality of sheets A second processing module for obtaining dense point cloud data of the first reference face corresponding to each image is further included.

In one optional embodiment, the first processing module uses dense point cloud data of a first reference face corresponding to each of a plurality of reference images to fit the original face to the dense point cloud data to obtain the first plurality of groups. 1 When obtaining fitting coefficients corresponding to each of the dense point cloud data of the reference face, least square processing is performed on the dense point cloud data of the original face and the dense point cloud data of the first reference face to obtain the first of the plurality of groups 1 obtain intermediate coefficients corresponding to each of the dense point cloud data of the reference face; A fitting coefficient corresponding to the dense point cloud data of the first reference face of each group is determined based on an intermediate coefficient corresponding to the dense point cloud data of the first reference face of each group.

In one optional embodiment, the first processing module is configured to: based on a median coefficient corresponding to the dense point cloud data of the first reference face of each group, the dense point cloud data corresponding to the first reference face of each group; determining a first type of dense point cloud data corresponding to a part of a target facial model among the first reference faces from the dense point cloud data of the first reference faces of each group when determining the fitting coefficient; obtaining first fitting coefficients by adjusting intermediate coefficients corresponding to the first type of dense point cloud data of the first reference face among the dense point cloud data; determining an intermediate coefficient corresponding to a second type of dense point cloud data among the dense point cloud data of the first reference face as a second fitting coefficient; the second type of dense point cloud data is dense point cloud data excluding the first type of dense point cloud data from the dense point cloud data of the first reference face; A fitting coefficient of the dense point cloud data of the first reference face of each group is obtained based on the first fitting coefficient and the second fitting coefficient.

In one optional embodiment, the device further comprises: adjusting dense point cloud data of a first reference face in the reference image to obtain dense point cloud data of a second reference face having the preset style; or, generating a virtual face image including a second reference face having the preset style based on a first reference face in the reference image; An adjustment module for obtaining dense point cloud data of the second reference face having a preset style through a method of generating dense point cloud data of the second reference face in the virtual face image using a pre-trained neural network. contains more

In one optional embodiment, when training the neural network, the device acquires a set of sample images; the sample image set includes a plurality of first sample images including a first sample face; The plurality of first sample images are partitioned into a plurality of subsets of first sample images, and in each subset of first sample images, first sample facial expressions having the same type of expression obtained by collecting each from a plurality of predetermined collection angles are included. images are included; acquire dense point cloud data of a first sample face of a first sample image in the sample image set; perform feature learning on a first sample image in the sample image set with a neural network to obtain predicted density point cloud data of a first sample face of the first sample image; and a training module configured to train the neural network using the dense point cloud data of the first sample face and the predicted dense point cloud data.

In one optional embodiment, the sample image set further includes a plurality of second sample images including a second sample face and a background, and the training module further includes facial keypoint data of each of the second sample images. obtain; generating by fitting dense point cloud data of a second sample face of the second sample image using facial keypoint data of the second sample image and the second sample image; perform feature learning on a second sample image in the sample image set with the neural network to obtain predicted density point cloud data of a second sample face of the second sample image; The neural network is trained using the dense point cloud data of the second sample face and the predicted dense point cloud data.

In one optional embodiment, the set of sample images further includes a third sample image; the third sample image is obtained by performing data augmentation processing on the first sample image;

The training module further acquires dense point cloud data of a third sample face of the third sample image; perform feature learning on the third sample image with the neural network to obtain predicted dense point cloud data of a third sample face of the third sample image;

The neural network is trained using the dense point cloud data of the third sample face and the predicted dense point cloud data.

In one optional embodiment, the data augmentation process includes at least one of random occlusion process, Gaussian noise process, motion blur process and color gamut channel change process.

In a third aspect, an alternative embodiment of the present invention further provides a computer device comprising a processor and a memory, wherein the processor executes machine-readable instructions stored in the memory, and the machine-readable instructions correspond to the processor When performed by, steps according to the first aspect or any one possible embodiment of the first aspect are performed.

In a fourth aspect, an alternative embodiment of the present invention further provides a computer readable storage medium wherein, when a computer program stored therein is executed, according to the first aspect or any one possible embodiment of the first aspect. do the steps

A description of the effects of the face reconstruction device, computer device, and computer readable storage medium may refer to the description of the face reconstruction method, which is not described herein any further.

In order to make the above objects, features and advantages of the present invention clearer and easier to understand, particularly preferred embodiments are enumerated below and described in detail below in conjunction with the accompanying drawings.

In order to more clearly explain the technical solutions of the embodiments of the present invention, the drawings to be used in the following embodiments are briefly introduced. These drawings show embodiments consistent with the present invention, and together with the specification, the technical Describe the solution. It should be understood that the drawings below only illustrate some embodiments of the present invention and should not be regarded as limiting in scope, and a person skilled in the art may obtain other drawings based on these drawings without creative efforts.
1 is a flowchart of a facial reconstruction method provided by an embodiment of the present invention;
2 is a flowchart of a specific method for training a neural network provided by an embodiment of the present invention;
3 is a schematic diagram of a first sample image provided by an embodiment of the present invention and a third sample image determined using the first sample image;
4 is a flowchart of a specific method for obtaining dense point cloud data of a second sample face of a second sample image provided by an embodiment of the present invention;
5 shows a specific illustration of the structure of a neural network provided by an embodiment of the present invention;
6 is a flowchart of a method for determining fitting coefficients corresponding to each of a plurality of reference images;
7 is a flowchart of a specific method for determining fitting coefficients corresponding to dense point cloud data of a first reference face of each group provided by an embodiment of the present invention;
8 is a flowchart of a specific method for determining dense point cloud data of a target facial model provided by an embodiment of the present invention;
9 is a flowchart of a specific method for generating dense point cloud data of a target facial model provided by an embodiment of the present invention;
10 is a schematic diagram of a facial reconstruction device provided by an embodiment of the present invention;
11 is a schematic diagram of a computer device provided in an embodiment of the present invention.

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the following is intended to clearly and completely explain the technical solutions of the embodiments of the present invention in conjunction with the drawings of the embodiments of the present invention. It is obvious that only some embodiments of the present invention, but not all embodiments. In general, the components of the embodiments of the invention described and illustrated in the figures herein may be arranged and designed in a number of different configurations. Accordingly, the detailed description of the embodiments of the present invention provided in the drawings below does not limit the scope of the present invention to be protected, but only indicates selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present invention.

After research, when face reconstruction is performed on a face included in a person image to obtain a virtual facial 3D model, a face dense point cloud corresponding to the face is usually acquired based on the face image, and then the virtual face 3D model is obtained. It was found that a virtual image was created by adjusting the facial dense point cloud multiple times based on the specific style of the model. Since the face density point clouds corresponding to different faces are different, that is, even if the styles of the virtual faces to be reconstructed are the same, the adjustments to be performed when performing face reconstruction are not the same for the face dense point clouds corresponding to different faces according to the determined style. It is difficult to control the specific direction of the adjustment during the adjustment process because there is a large uncertainty in the adjustment, and since the generated 3D model of the virtual face may have a large difference from the real face, the degree of similarity between the 3D virtual face model and the real face cause low problems.

In addition, when face reconstruction is performed for different faces, adjustment schemes for different faces must be set according to the needs of the face dense point cloud corresponding to the different faces and the face dense point cloud of the style, so that different adjustment schemes If the facial density point cloud of each face is adjusted based on the above, a lot of time is consumed and the efficiency of face reconstruction is low.

Based on the above research, the present invention provides a face reconstruction method, apparatus, computer device, and storage medium, which use fitting coefficients as a medium to obtain dense point cloud data of an original face and dense point clouds of a plurality of first reference faces. An association relationship between the data is established, wherein the association relationship is the dense point cloud data of the second reference face determined based on the dense point cloud data of the first reference face and the dense point of the target face model built based on the original face By characterizing the association between cloud data, the generated target facial model has features (eg shape features, etc.) of the original face in the target image, so that it can have a higher degree of similarity with the original face and the generated target facial model has a preset style. can be provided.

In addition, the present solution generates dense point cloud data of a second reference face corresponding to each of a plurality of reference images for different styles, and uses the dense point cloud data of the second reference face having a preset style to generate a target face model for the original face, and use the dense point cloud data of the same second reference face and the fitted coefficients of the different original face without the need to determine the adjustment method aimably for the different original face. Since the target face model of the original face is determined, processing efficiency is higher.

Since all of the defects in the above means are the result obtained by the inventor through practice and meticulous research, the discovery process of the above problems and the solutions proposed for the above problems in the present invention to be described later are all related to the present invention during the process of the present invention. It must be the contribution of the inventor.

It should be noted that similar symbols and letters indicate similar items in the drawings below, so that items defined once in one drawing do not need to be defined and commented on in subsequent drawings.

In order to make the present embodiment easier to understand, the face reconstruction method disclosed in the embodiment of the present invention is first introduced in detail, and the execution subject of the face reconstruction method provided in the embodiment of the present invention generally has a certain computing power. A computer device having a computer device including, for example, a terminal device or a server or other processing device, wherein the terminal device includes a user equipment (UE), a mobile device, a user terminal, a cellular phone, a wireless phone, a personal use It may be a Personal Digital Assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, and the like. In some possible implementations, the face reconstruction method may be implemented in a way of calling computer readable instructions stored in a memory through a processor.

Hereinafter, a facial reconstruction method provided by an embodiment of the present invention will be further described.

Referring to the content shown in FIG. 1 , an embodiment of the present invention provides a facial reconstruction method, which includes steps S101 to S104, wherein:

Step S101: Acquiring dense point cloud data of an original face included in a target image;

Step S102: Fitting the dense point cloud data of the original face using the dense point cloud data of the first reference face corresponding to each of the plurality of reference images, and fitting corresponding to each of the dense point cloud data of the plurality of groups of first reference faces get coefficients;

Step S103: Dense point cloud data of the target facial model based on fitting coefficients corresponding to each of the dense point cloud data of the plurality of groups of second reference faces and the plurality of groups of dense point cloud data of the first reference face having a preset style. Determine; The plurality of groups of second reference faces are each generated based on the first reference faces in the plurality of reference images;

Step S104: A target face model corresponding to the original face of the target image is generated according to the dense point cloud data of the target face model.

In an embodiment of the present invention, a fitting coefficient determined when fitting dense point cloud data of an original face in a target image through dense point cloud data of a first reference face corresponding to each of a plurality of reference images and a first reference face in the reference image Obtaining dense point cloud data of a target facial model using dense point cloud data of a second reference face generated by and having a preset style, and a target corresponding to the target image based on the dense point cloud data of the target facial model. A face model is generated, and in this process, an association relationship is established between the dense point cloud data of the original face and the dense point cloud data of a plurality of first reference faces using the fitting coefficient as a medium, and the association relationship is the first A target face generated by showing a correlation between the dense point cloud of the second reference face determined based on the dense point cloud of the reference face and the dense point cloud data of the target face model built based on the dense point cloud data of the original face Make the model's dense point cloud data have features (eg shape features, etc.) of the original face in the target image, have a higher degree of similarity with the original face, and also make the generated target face model have a preset style do.

Steps S101 to S104 will be described in further detail below.

Regarding the above step S101,

The target image is, for example, a pre-obtained image including a face, which is obtained when an object is photographed using a photographing device such as a camera, for example, and includes a face. In this case, for example, an arbitrary face included in the image may be determined as the original face and used as the original face reconstruction object.

When the face reconstruction method provided in the embodiment of the present invention is applied to different scenarios, the method of acquiring the target image is slightly different.

For example, when this facial reconstruction method is applied to a game, an image including the game player's face is acquired through an image acquisition device mounted on the game device, or an image including the game player's face is obtained from an album of the game device. is selected, and an already acquired image including the game player's face can be used as the target image.

In addition, for example, when the facial reconstruction method is applied to a terminal device such as a mobile phone, an image containing a user's face is collected by a camera of the terminal device, or an image containing the user's face is selected from an album of the terminal device, or Other application programs installed in the terminal device may receive an image including the user's face.

Further, for example, when the face reconstruction method is applied to a live broadcast scenario, a video frame image including a face may be obtained from a multi-frame video frame image included in a video stream obtained by a live broadcast device; A video frame image including a face may be used as a target image. Here, the target image may be multi-frame, for example; For example, the multi-frame target image may be obtained by performing sampling on multi-frame video frame images in a video stream.

When obtaining dense point cloud data of an original face included in the target image, for example, a method of obtaining a target image including the original face; A method of obtaining dense point cloud data of an original face in the target image by processing the target image with a pre-trained neural network may be used.

Here, when the target image is processed with a pre-trained neural network to obtain dense point cloud data of the original face, the pre-trained neural network is a convolutional neural network (CNN), back propagation neural network (BP), backbone It includes at least one of neural networks (Backbone).

When determining the neural network structure, for example, the backbone network of the neural network may be determined as the subject architecture of the neural network. By way of example, the backbone network is, for example, an inception network and a residual network transformation network. (the next dimension to RESNET, ResNeXt), a starting network transformation network (Xception), a squeeze-and-excitation network (SENet), a lightweight network (MobileNet), and a lightweight network (ShuffleNet). can

Exemplarily, when the neural network includes a convolutional neural network, a lightweight network may be selected as a convolutional neural network base model, and on the basis of the lightweight network, other network structures may be increased to construct a convolutional neural network, Convolutional neural networks can be trained. This process uses the lightweight network as a part of the convolutional neural network, and the lightweight network is small in volume and has high data processing speed, so the training speed is faster; In addition, the neural network obtained by training is suitable for deployment in an embedded device because it has a small volume and a fast data processing speed.

Here, the network structure of the neural network is just an example; The specific construction method and structure of the network structure may be determined according to actual circumstances, and are not described herein any further, and the above examples do not limit the embodiments of the present invention.

Referring to the content shown in FIG. 2 , an embodiment of the present invention shows a specific method for training a neural network, which includes the following steps:

Step S201: Obtain a sample image set; The sample image set includes a plurality of first sample images including first sample faces; The plurality of first sample images are partitioned into a plurality of subsets of first sample images, and in each subset of first sample images, first sample facial images having the same type of facial expression obtained by collecting each from a plurality of predetermined collection angles are included. image is included.

Here, for the plurality of first sample images of the first sample face included in the sample image set, the corresponding first sample face is, for example, at least one entity object for acquiring a predetermined face image and training the neural network. is the face of

When the first sample face is photographed and the first sample image subset is determined, for example, a plurality of first sample image subsets are determined for a plurality of different facial expressions. Here, the plurality of different facial expressions are, for example, pleasure, excitement, disappointment, sadness, and the like. If the first sample face is photographed to obtain the first sample image subset, for example, the first sample face is photographed for a "pleasure" expression, the first sample face exhibiting a "pleasure" expression is photographed from different angles, A plurality of first sample images corresponding to the “pleasure” expression may be acquired and used as a first sample image subset. Here, when the first sample images are obtained, the backgrounds of the first sample faces in different first sample images may be the same.

Similarly, corresponding subsets of first sample images in different expressions of the first sample face may be obtained using a similar method, which is not further described herein.

After determining the subset of first sample images corresponding to each of a plurality of different facial expressions, a plurality of first sample images of the first sample face may be determined.

Exemplarily, when acquiring the first sample image, the image may be obtained by photographing with an image collection device, wherein the image collection device includes, for example, at least one of a depth camera and a color camera.

For example, a plurality of first sample images may be obtained by photographing the faces of I objects with E facial expressions. For example, first, the face of any one entity object A may be determined as the first sample face. When the first sample face exhibits a “sadness” expression, the first sample face is photographed from P (P is an integer greater than 1) different angles to obtain a plurality of images corresponding to P different angles in the “sadness” expression. obtained and used as the first sample image subset corresponding to the “sad” expression. Next, P length images corresponding to P angles of the first sample face in other facial expressions (including, for example, E-1 different facial expressions) are obtained and used as the first sample image subset corresponding to the other facial expressions. Correspondingly, E different subsets of first sample images may be determined for E different facial expressions of the entity object A, and correspondingly, PХE field images of the entity object A may be obtained. Then, it is possible to acquire PХE images corresponding to each of the remaining I-1 objects by photographing the remaining I-1 objects, that is, by acquiring all M (M=IХEХP) images, 1 can be used as a sample image.

Since the obtained plurality of first sample images represent faces corresponding to a plurality of photographic object objects with different facial expressions, and there are few other parts such as backgrounds, the neural network uses the first sample images to detect faces corresponding to faces at a plurality of angles in the image. Ability to acquire dense point cloud data can be trained.

The second sample image may be obtained by randomly photographing different objects, or may be obtained by randomly crawling a plurality of images including faces on a predetermined network platform and using the crawled images as the second sample face image.

Exemplarily, when obtaining second sample images by randomly photographing different objects, obtaining second sample images by photographing a plurality of second sample faces using a camera or other image capturing device; Alternatively, a plurality of pre-captured second sample images may be directly obtained. The second sample image includes, for example, already acquired H face images including a background; Here, the background included in the second sample image is a part that interferes with face recognition, and when the neural network recognizes a face in the target image to obtain dense point cloud data, the interference prevention ability of other parts except the face is trained. .

In another embodiment of the present invention, the sample image set further includes a third sample image, and the third sample image may be obtained by performing data augmentation processing on the first sample image, for example. Here, the data augmentation process includes at least one of random occlusion process, Gaussian noise process, fuzz process, and color gamut channel change process. It is possible to reduce the learning of irrelevant features of the neural network in the feature set of the data by using a method of data augmentation processing, thereby improving the performance of the neural network when obtaining dense point cloud data.

Exemplarily, when random occlusion processing is selected, a third sample image may be obtained by performing occlusion processing on a partial region in the first sample image; Here, the size of the occlusion portion may be determined according to the size of the first sample image and actual demand, but is not limited thereto; When Gaussian noise processing is selected, for example, at least one of fluctuation noise, cosmic noise, thermal noise, and shot noise is selected and added to the first sample image so that the third sample image includes at least one Gaussian noise. there is; When fuzzy processing is selected, for example, motion blur processing may be performed on at least some pixel points in the image to obtain a third sample image having a motion blur effect; When color gamut channel change processing is selected, for example, the first sample image is processed using global color gamut channel change processing or random color gamut channel change processing to obtain a third sample image in which the color gamut channels of the global or partial area are changed. You can get it.

3 is a schematic diagram of a first sample image provided by an embodiment of the present invention and a third sample image determined using the first sample image; As shown in Fig. 3, 31 represents a first sample image; 32 denotes a third sample image obtained by performing data augmentation processing for performing fuzzy processing on the first sample image 31; 33 denotes a third sample image obtained by performing data augmentation processing for performing partial occlusion on the first sample image 31, where 34 denotes a position of partial occlusion.

In this case, the dense point cloud data of the sample face corresponding to the third sample image is the same as the dense point cloud data of the sample face corresponding to the first sample image generating the third sample image.

Following step S201, a specific method for training a neural network is,

A step S202 of acquiring dense point cloud data of a first sample face of a first sample image in the sample image set is further included.

Specifically, when obtaining the dense point cloud data of the first sample face of the first sample image in the sample image set and the dense point cloud data of the second sample face of the second sample image in the sample image set, for example, the first of the sample image set Dense point cloud data of a first sample face of a sample image, dense point cloud data of a second sample face of a second sample image, and dense point cloud data of a third sample face of a third sample image may be obtained. .

Exemplarily, with respect to the first sample image in the sample image set, a first sample face is photographed and a corresponding first sample image is photographed, and then dense point cloud data of the first sample face in each first sample image is obtained. can do. When the first sample image is acquired using a color camera, dense point cloud data of the first sample face may be obtained using a model such as a facial 3D Morphable Model (3DMM); When the first sample image is acquired using the depth camera, for example, dense point cloud data of the first sample face may be obtained based on the depth image acquired by the depth camera.

Regarding the second sample image in the sample image set, referring to the content shown in FIG. 4 , an embodiment of the present invention provides a specific method for obtaining dense point cloud data of a second sample face of the second sample image. , this is,

Step S401 of acquiring facial keypoint data of each second sample image;

Step S402 of fitting and generating dense point cloud data of a second sample face of a second sample image using the facial keypoint data of the second sample image and the second sample image;

Step S403 of performing feature learning on a second sample image in the sample image set with a neural network to obtain predicted density point cloud data of a second sample face of the second sample image;

In step S404, the neural network is trained using the dense point cloud data of the second sample face and the predicted dense point cloud data.

Here, the second sample image includes, for example, a face image including an acquired background of H sheets, and each second sample image includes the determined face keypoints. Here, the facial keypoints included in the second sample image include dense point cloud data of the sample face corresponding to the second sample image, for example, a plurality of keypoints indicating directly labeled five optic tracts, cheekbones, and browbones among the second sample images, and determine keypoints representing the same facial features; Alternatively, a keypoint corresponding to the face determined using the keypoint detection method is included. Here, the keypoint detection method includes at least one of Active Shape Model (ASM), Active Appearance Models (AAM), and Cascaded Pose Regression (CPR).

When facial keypoint data of the second sample image is acquired, dense point cloud data of the second sample face of the second sample image may be generated using the fitting model. Here, the fitting model includes a facial 3DMM, for example.

Following step S202, the method for training a neural network,

A step S203 of performing feature learning on a first sample image in the sample image set using a neural network to obtain predicted density point cloud data of a first sample face of the first sample image.

Specifically, when determining the predicted dense point cloud data of the first sample face of the first sample image and the predicted dense point cloud data of the second sample face of the second sample image, for example, by using an initial neural network, among the sample image set By performing feature learning on the first sample image, the second sample image, and the third sample image, the predicted dense point cloud data of the first sample face of the first sample image and the predicted dense point of the second sample face of the second sample image are obtained. A method of obtaining cloud data and predicted dense point cloud data of the third sample face of the third sample image may be used.

At this time, it should be noted that the step of acquiring at least one sample image in the first sample image, the second sample image, and the third sample image includes the first sample image, the second sample image, and the third sample image by using the initial neural network. It may be performed synchronously with the step of performing feature learning on at least one of the sample images, that is, feature learning on at least one of the first sample image, the second sample image, and the third sample image using the initial neural network. After performing, the corresponding predicted dense point cloud data can be directly obtained.

In addition, when feature learning is performed on at least one of the first sample image, the second sample image, and the third sample image with the initial neural network, at least one of the first sample image, the second sample image, and the third sample image may perform feature learning synchronously for; Alternatively, according to actual demand, feature learning is performed sequentially on at least one of the first sample image, the second sample image, and the third sample image to obtain predicted dense point cloud data of the first sample face of the first sample image, Predicted dense point cloud data of a second sample face of the 2 sample image and predicted dense point cloud data of a third sample face of the third sample image may be obtained.

Embodiments of the present invention do not limit the sequential order of performing the sample processing process, and may be specifically set according to actual demand.

Exemplarily, with respect to the first sample image and the second sample image included in the sample image set, different quantities of the first sample image and the second sample image are selected according to a preset ratio, and the selected first sample image and the second sample image are selected. 2 sample images can be input to the initial neural network; When the sample image set includes the first sample image, the second sample image, and the third sample image, different numbers of the first sample image, the second sample image, and the third sample image are selected according to a predetermined ratio, and the selected The first sample image, the second sample image, and the third sample image may be input to the initial neural network. When the ratio selection is different, the focus of training the neural network is also different. When the ratio of the first sample image and/or the third sample image is large, the ability to acquire face dense point clouds corresponding to faces of different angles in the trained neural network image is strong; When the ratio occupied by the second sample image is large, the anti-interference capability for other background parts other than the face in the trained neural network image is stronger, thereby satisfying different usage needs.

After inputting the sample images to the initial neural network, the initial neural network may perform feature learning on the sample images, and output dense point cloud data of face prediction of each sample image; The loss of the neural network can be determined using the dense point cloud data of the face prediction and the dense point cloud data of the sample face corresponding to each sample image. The loss is the accuracy of generating the dense point cloud data of the face of the neural network to estimate

The face prediction dense point cloud data obtained using the neural network includes, for example, a coordinate value representing the position of the face dense point cloud in a preset coordinate system, or x representing the position of the face dense point cloud in a preset coordinate system. It may include coordinate values corresponding to each of the axis, y-axis, and z-axis. Here, the preset coordinate system may be, for example, a preset facial coordinate system.

를 포함할 수 있거나, 또는 x축에서의 좌표값

,y축에서의 좌표값

및 z축에서의 좌표값

를 포함할 수 있고, 상이한 안면 예측의 덴스 포인트 클라우드 데이터에 포함된 x축, y축 및 z축에서의 좌표값은 출력된 안면 예측의 덴스 포인트 클라우드 데이터에서 기설정된 배열 순서를 가질 수 있다.Exemplarily, in the case of including coordinate values indicating the location of the facial dense point cloud for any face prediction dense point cloud data, the face prediction dense point cloud data is a coordinate value

Can include, or coordinate values on the x-axis

, the coordinate value on the y-axis

and coordinate values on the z-axis

, and coordinate values on the x-axis, y-axis, and z-axis included in the different dense point cloud data of face prediction may have a preset arrangement order in the output dense point cloud data of face prediction.

Referring to FIG. 5 , an embodiment of the present invention further provides a specific exemplary diagram of a neural network structure, wherein the neural network structure includes a backbone network 51, a first fully connected layer 52 and three and a second fully connected layer 53 of the group.

When the sample image is processed by this neural network to determine the dense point cloud data of the face prediction of the face in the sample, for example, the sample image can be input to the backbone network to obtain feature data of the sample image, and the feature data is the first complete After passing through the connection layer, they are respectively input into three groups of second fully connected layers; The three groups of second fully connected layers may predict coordinate values in the face coordinate system of the dense point cloud of the face in the sample image. For example, the second fully connected layer of the first group may output the coordinate value of the x-axis in the face coordinate system of the dense point cloud, and the second fully connected layer of the second group may output the coordinate value of the x-axis in the face coordinate system of the dense point cloud. The y-axis coordinate value may be output, and the second fully connected layer of the third group may output the z-axis coordinate value in the facial coordinate system of the dense point cloud. Coordinate values in the face coordinate system of the dense point cloud constitute dense point cloud data of face prediction of faces in the sample image.

When determining the plurality of first sample images and the plurality of second sample images, the selection rate of the first sample image is 40% and the selection rate of the second sample image is 60%. Choose a sample image. For example, when training a neural network using 100 sample images, 40 first sample images and 60 second sample images are selected as a plurality of sample images.

Here, a convolutional neural network is selected and trained using it as an initial neural network, a lightweight network (t) is used as a base model, and a convolutional neural network is built using a split FC as an output layer, thereby constructing each of a plurality of sample images. Dense point cloud data of face prediction corresponding to can be obtained, and the obtained dense point cloud data of face prediction includes a plurality of coordinate values on the x-axis, y-axis, and z-axis of each point in the dense point cloud.

,

및

로 나타낼 수 있는 바, 즉 R개의 상이한 안면 부위 중 임의의 부위에 대응하는 안면 예측의 덴스 포인트 클라우드 데이터에 포함된 좌표값

은 x축, y축 및 z축에서의 좌표값

,

및

로 분해되고, 세 개의 상이한 매트릭스 중 i번째 요소로서 출력될 수 있다.At this time, with respect to the dense point cloud data of facial prediction corresponding to an arbitrary sample image, if R different facial parts are included in the facial dense point cloud, the output dense point cloud data of face prediction is in the form of a matrix, for example can be output,

,

and

, that is, the coordinate values included in the dense point cloud data of face prediction corresponding to any part among the R different face parts

is the coordinate value on the x-axis, y-axis and z-axis

,

and

can be decomposed into , and output as the i-th element of three different matrices.

Following step S203, the method for training a neural network,

A step S204 of training the neural network using the dense point cloud data of the first sample face and the predicted dense point cloud data is further included.

Specifically, when an already trained neural network is obtained by training an initial neural network, dense point cloud data and predicted dense point cloud data of the first sample face, dense point cloud data and predicted dense point cloud data of the second sample face, and third A method of training a neural network using the dense point cloud data of the sample face and the predicted dense point cloud data, and obtaining an already trained neural network after the training is completed may be used.

Exemplarily, the loss of the neural network is determined based on the difference between the dense point cloud data of the face prediction and the dense point cloud data of the sample face, and the neural network is trained using the loss, the direction of the training is to reduce the loss As a direction, it makes the dense point cloud data of the face prediction obtained when the neural network processes the image sufficiently close to the dense point cloud data of the real face.

In the case of obtaining an already trained neural network, it is possible to obtain dense point cloud data corresponding to an original face among target pictures by inputting a target picture to the neural network.

Regarding the above step S102,

The reference image may be, for example, a face corresponding to each of the different entity objects, and the faces corresponding to the different entity objects are different; Illustratively, a plurality of persons having different at least one of gender, age, skin color, obesity, etc. may be determined, a facial image of each person among the plurality of persons may be obtained, and the acquired facial image may be used as a second sample. Can be used as an image. In this way, it is possible to cover a wide range of facial appearance features based on the dense point cloud data of the second sample face generated based on the second sample image.

When obtaining dense point cloud data of a plurality of first reference faces corresponding to the plurality of reference images, for example, a method of obtaining a plurality of reference images including the first reference face; For each reference image in the plurality of reference images, a method of obtaining dense point cloud data of the first reference face in each reference image with a pretrained neural network may be used. Here, a method of acquiring a plurality of first reference facial dense point clouds with a pre-trained neural network is similar to a method of acquiring an original facial dense point cloud with the pre-trained neural network, and thus is not further described herein.

When determining the dense point cloud data of the original face and the dense point cloud data of the first reference face, the dense point cloud data of the original face is fitted using the dense point cloud data of the first reference face to each of the plurality of reference images A fitting coefficient of the corresponding dense point cloud data of the first reference face may be acquired. Here, the fitting coefficient may establish an association between dense point cloud data of an original face in a target image and dense point cloud data of a first reference face corresponding to each of a plurality of reference pictures as a medium.

Referring to FIG. 6 , an embodiment of the present invention provides a method for determining fitting coefficients corresponding to each of a plurality of reference images, including steps S601 and S602 below.

Step S601: Least square processing is performed on the dense point cloud data of the original face and the dense point cloud data of the first reference face to obtain intermediate coefficients respectively corresponding to the plurality of groups of dense point cloud data of the first reference face.

로 나타내고, 제1 참조 안면의 덴스 포인트 클라우드 데이터를

로 나타낸다. 여기서, 제1 참조 안면 덴스 포인트 클라우드가 복수의 참조 이미지에 의해 결정된 것이므로, N장의 참조 이미지가 존재하는 경우, 제1 참조 안면의 덴스 포인트 클라우드 데이터

중 대응하는 N 그룹의 안면을 포함하는 덴스 포인트 클라우드 데이터는

로 나타내고, 여기서,

는 i번째 장의 참조 이미지에 대응하는 안면의 덴스 포인트 클라우드 데이터를 나타낸다.Illustratively, the dense point cloud data of the original face

, and the dense point cloud data of the first reference face

represented by Here, since the first reference face dense point cloud is determined by a plurality of reference images, when N reference images exist, the dense point cloud data of the first reference face

Dense point cloud data including faces of corresponding N groups of

, where,

represents dense point cloud data of a face corresponding to the reference image of the i-th chapter.

및

내지

를 이용하여 최소 제곱 처리를 수행하여 N개의 피팅값을 얻을 수 있고,

으로 나타낸다. 여기서,

i번째 그룹의 제1 참조 안면의 덴스 포인트 클라우드 데이터에 대응하는 피팅값을 나타낸다. N개의 피팅값을 통해 피팅 계수 Alpha를 결정할 수 있는 바, 예를 들면 계수 매트릭스, 즉

으로 나타낼 수 있다.

and

pay

It is possible to obtain N fitting values by performing least square processing using

represented by here,

Indicates a fitting value corresponding to the dense point cloud data of the first reference face of the i-th group. The fitting coefficient Alpha can be determined through N fitting values, for example, a coefficient matrix, that is,

can be expressed as

Here, in the process of fitting the dense point cloud data of the original face through the dense point cloud data of the first reference face, the data after performing a weighted sum on the dense point cloud data of the first reference face through the fitting coefficient is the original It should be as close as possible to the data of the facial dense point cloud data.

The fitting coefficient is also considered as an expression coefficient of each of the dense point cloud data of the first reference face when the original face dense point cloud data is represented by the dense point cloud data of the first reference face corresponding to the plurality of reference images. can do.

Step S602: Corresponding to the dense point cloud data of the first reference face of each group based on intermediate coefficients corresponding to the dense point cloud data of the first reference face of each group in the dense point cloud data of the first reference face of the plurality of groups. Determine the fitting coefficient for

Specifically, referring to the content shown in FIG. 7 , an embodiment of the present invention further provides a specific method for determining fitting coefficients corresponding to dense point cloud data of a first reference face of each group, which includes the following steps It includes steps S701 to S704.

Step S701: In the dense point cloud data of the first reference face of each group, a first type of dense point cloud data corresponding to a part of the target face model among the first reference faces is determined.

에 대하여,

는 i번째 장의 참조 이미지에 기반하여 얻은 것이므로,

에는 i번째 장의 참조 이미지 내의 안면 부위에 대응하는 덴스 포인트 클라우드 데이터가 포함된다. 여기서, 안면 부위는 예를 들면 눈썹, 코, 눈, 입, 광대뼈 부위, 아래턱 부위를 포함한다. 각 안면 부위에 대하여, 추가로 더 상세하게 구획할 수 있는 바, 예를 들면 눈썹은 눈썹 끝 부위, 눈썹 중앙 부위 및 눈썹 피크 부위로 더 구획될 수 있다.Dense point cloud data of any one group among the plurality of groups of dense point cloud data of the first reference face

about,

is obtained based on the reference image of the ith chapter,

includes dense point cloud data corresponding to a facial part in the reference image of the i-th chapter. Here, the facial parts include, for example, eyebrows, nose, eyes, mouth, cheekbones, and lower jaw. For each facial region, further detailed partitioning may be performed, for example, the eyebrows may be further divided into an eyebrow tip region, an eyebrow central region, and an eyebrow peak region.

In one possible embodiment, by adjusting the fitting coefficients corresponding to some facial parts, so that the fitting coefficients more accurately represent the case where the dense point cloud data of the first reference face fits the dense point cloud data of the original face , the fitting coefficients, and the plurality of corresponding dense point cloud data of the first reference face to approximate the obtained dense point cloud data to the original face corresponding to the target image. At this time, some facial parts corresponding to the fitting coefficients, that is, target facial model parts such as eyes and mouth need to be adjusted. A specific target facial model part can be determined according to specific cases or experience, and is not described herein any further.

를 예로 들면, 안면 부위는 예를 들어 R개로 구획될 수 있는 바, 이때 R개의 안면 부위에 대응하는 덴스 포인트 클라우드 데이터는 예를 들면

,

, ……,

로 나타낼 수 있다. R개의 안면 부위 중의 타깃 안면 부위가 눈, 입을 포함하는 경우, 타깃 안면 모델 부위에 대응하는 덴스 포인트 클라우드 데이터는 예를 들면

및

즉 제1 유형의 덴스 포인트 클라우드 데이터를 포함한다.When determining the first type of dense point cloud data corresponding to the target facial model part in the first reference face indicated by the dense point cloud data of the first reference face, the dense point cloud data of a group of first reference faces

For example, the facial parts can be divided into R pieces, for example, and the dense point cloud data corresponding to the R facial parts is, for example

,

, … … ,

can be expressed as When a target facial part among the R facial parts includes eyes and a mouth, the dense point cloud data corresponding to the target facial model part is, for example,

and

That is, it includes the first type of dense point cloud data.

Step S702: Among the dense point cloud data of the first reference face, intermediate coefficients corresponding to the first type of dense point cloud data are adjusted to obtain first fitting coefficients.

에 대하여,

는

및

를 이용하여 최소 제곱 처리를 수행하여 얻은 것이므로,

에는 복수의 타깃 안면 모델 각각에 대응하는 복수의 일부 부위의 중간 계수가 포함된다.When the first type of dense point cloud data is determined, first fitting coefficients corresponding to some parts corresponding to the target facial model, that is, intermediate coefficients that achieve a better fitting effect by adjustment may be determined. At the same time, any one fitting value among the intermediate coefficients

about,

Is

and

Since it was obtained by performing least square processing using

includes intermediate coefficients of a plurality of partial parts corresponding to each of a plurality of target facial models.

중 복수의 타깃 안면 모델 부위에 대응하는 중간 계수는 예를 들면

및

, 즉 제1 유형의 덴스 포인트 클라우드 데이터

및

에 대응하는 중간 계수로 나타낼 수 있다. 중간 계수

과

에 대해 수치 조정을 수행하여, 조정하여 얻은 제1 피팅 계수에 기반하여 원본 안면의 덴스 포인트 클라우드 데이터를 피팅한 후, 피팅 결과가 원본 안면의 덴스 포인트 클라우드 데이터에 보다 근접하도록 한다. 여기서, 수치 조정은 예를 들면 수치 증가 및/또는 수치 감소를 포함한다.Exemplarily, when the target facial model parts are eyes and mouth, respectively,

Among them, the intermediate coefficient corresponding to a plurality of target facial model parts is, for example,

and

, that is, the first type of dense point cloud data

and

It can be expressed as an intermediate coefficient corresponding to median coefficient

class

Numerical adjustment is performed on , and after fitting the dense point cloud data of the original face based on the first fitting coefficient obtained by the adjustment, the fitting result is brought closer to the dense point cloud data of the original face. Here, numerical adjustment includes, for example, numerical increase and/or numerical decrease.

Step S703: Determine an intermediate coefficient corresponding to a second type of dense point cloud data of the first reference face as a second fitting coefficient; The second type of dense point cloud data is dense point cloud data other than the first type of dense point cloud data of the first reference face.

In this case, an intermediate coefficient corresponding to the second type of dense point cloud data among the dense point cloud data of the first reference face may be determined as the second fitting coefficient. Also, among the dense point cloud data of the first reference face of each group, dense point cloud data other than the first type of dense point cloud data may be used as the second type of dense point cloud data. Since the fitting coefficient corresponding to the second type of dense point cloud data has a small effect on the fitting result or the fitting result is relatively good when fitting, the fitting coefficient corresponding to the second type of dense point cloud data is not adjusted. As a result, the efficiency can be improved under the premise of securing the fitting effect.

Step S704: Obtain fitting coefficients of the dense point cloud data of the first reference face of each group according to the first fitting coefficient and the second fitting coefficient.

Since the first fitting coefficient corresponds to the target facial model part and the second fitting coefficient corresponds to other facial parts other than the target facial model part among the plurality of facial parts, the first fitting coefficient and the second fitting coefficient are combined to obtain a plurality of A fitting coefficient corresponding to a face region, that is, a fitting coefficient of dense point cloud data of a first reference face of each group may be determined.

Regarding the above step S103,

The preset style may be, for example, a cartoon style, an ancient style, or an abstract style, and may be specifically set according to actual needs. For example, when the preset style is a cartoon style, the second reference face having the preset style may be a cartoon face.

When generating the dense point cloud data of the second reference face having a preset style using the first reference face, for example, by adjusting the dense point cloud data of the first reference face in the reference image to obtain the preset style a method of obtaining dense point cloud data of a second reference face provided; Alternatively, a virtual face image including a second reference face having a preset style is generated based on the first reference face in the reference image, and a dense point of the second reference face among the virtual face images is generated using a pre-trained neural network. You can use a way to create cloud data.

In the case of obtaining the dense point cloud data of the second reference face having a preset style by adjusting the first reference face dense point cloud, illustratively, all the denses in the dense point cloud data of the first reference face according to the preset style By adjusting the point cloud data or some of the dense point cloud data, the face reflected by the acquired dense point cloud data of the second reference face may have a preset style.

Taking for example that the preset style is a cartoon style, in one possible embodiment, when the cartoon style includes, for example, enlarging the eyes, when adjusting the dense point cloud data of the first reference face, corresponding to the eyes The second reference obtained by adjusting the dense point cloud data by moving the positional coordinates of the dense point cloud corresponding to the eye part upward and/or moving the positional coordinates of the dense point cloud corresponding to the lower eyelid part downward Among the dense point cloud data of the face, the dense point cloud data corresponding to the eye indicates that the eye is enlarged.

When a virtual face image is generated based on the first reference face and dense point cloud data of the second reference face is generated using a neural network obtained by pre-training, exemplarily, according to a preset style, the first in the reference image A virtual face image of a second reference face having a preset style may be generated by performing graphic image processing on the reference face. Here, graphic image processing may include, for example, picture editing, picture production, and picture design.

When a virtual face image of the second reference face having a preset style is acquired, dense point cloud data of the corresponding second reference face may be determined using a neural network obtained through pre-training. Here, the method for determining the dense point cloud data of the second reference face using the neural network obtained by pre-training includes the dense point cloud data of the original face and the dense point cloud data of the first reference face using the pre-trained neural network. Similar to the determining method, it is not further described here.

로 나타낼 수 있다.At this time, for example, the obtained second reference face's dense point cloud data

can be expressed as

After determining a fitting coefficient corresponding to each of the dense point cloud data of the second reference face having a preset style and the dense point cloud data of the plurality of groups of the first reference face, the dense point cloud data of the target face model may be determined. .

Referring to FIG. 8 , an embodiment of the present invention further provides a specific method for determining dense point cloud data of a target facial model, which includes steps S801 to S802 below.

Step S801: Based on the dense point cloud data of the plurality of groups of second reference faces, average data of the dense point cloud data of the second reference faces of the plurality of groups is generated.

중 대응하는 부위의 좌표값에 기반하여 평균값을 구하는 처리를 수행함으로써 제2 참조 안면의 덴스 포인트 클라우드 데이터의 평균값 데이터를 생성할 수 있다. 여기서, 평균값 데이터는 복수 그룹의 제2 참조 안면의 덴스 포인트 클라우드 데이터의 평균 특징을 나타낸다. 예시적으로, 제2 참조 안면의 덴스 포인트 클라우드 데이터에 N 그룹의 안면 덴스 포인트 클라우드가 포함된 경우, 제2 참조 안면에 대한 덴스 포인트 클라우드 데이터 중의 복수 그룹의 안면의 덴스 포인트 클라우드 데이터를

,

, ……,

으로 나타낼 수 있다.In one possible embodiment, the dense point cloud data of the second reference face

Average value data of the dense point cloud data of the second reference face may be generated by performing a process of obtaining an average value based on the coordinate values of the corresponding part. Here, the average value data represents an average feature of dense point cloud data of a plurality of groups of second reference faces. Exemplarily, when the dense point cloud data of the second reference face includes N groups of facial dense point clouds, the dense point cloud data of a plurality of groups of faces among the dense point cloud data for the second reference face

,

, … … ,

can be expressed as

에 대하여, 상이한 부위 위치에 대응하는 복수의 안면 덴스 포인트 클라우드를 결정할 수 있는 바, W개의 상이한 안면 덴스 포인트 클라우드를 포함하는 경우, 예를 들면 W개의 상이한 안면 덴스 포인트 클라우드를

로 나타낼 수 있고, 각각 대응하는 좌표값은

,

로 나타낼 수 있다. 각 그룹의

중 대응하는 부위 위치의 좌표값을 이용하여 평균값을 구하여 대응하는 부위 위치의 평균값 데이터를 얻을 수 있다. 이때, 상이한 부위 위치의 첫 번째 안면 덴스 포인트 클라우드

에 대하여, 예를 들면 아래 수학식(1)을 사용하여 평균값 데이터

을 얻을 수 있다.Here, the dense point cloud data of any one group of second reference faces

, it is possible to determine a plurality of facial dense point clouds corresponding to different body positions, where W different facial dense point clouds are included, for example, W different facial dense point clouds

, and the corresponding coordinate values are

,

can be expressed as of each group

Average value data of the corresponding part position can be obtained by obtaining an average value using the coordinate values of the corresponding part position. At this time, the first facial dense point cloud at the location of the different parts

For, for example, the average value data using Equation (1) below

can be obtained.

A method of determining average value data for facial dense point clouds at other locations of different parts is similar to a method of obtaining average value data for the first facial dense point cloud, and is not further described herein.

즉 제2 참조 안면의 덴스 포인트 클라우드 데이터의 평균값 데이터를 얻을 수 있고,

로 나타낼 수 있다.At this time, the average value data of different site locations

That is, average value data of dense point cloud data of the second reference face can be obtained,

can be expressed as

Step S802: Dense point cloud data of the target face model is generated based on the dense point cloud data of the second reference face of the plurality of groups, the mean value data, and the fitting coefficient corresponding to each of the plurality of groups of the dense point cloud data of the first reference face of the plurality of groups. do.

Specifically, referring to the content shown in FIG. 9 , an embodiment of the present invention further provides a specific method for generating dense point cloud data of a target facial model, which includes the following steps:

Step S901: difference value data of the dense point cloud data of the second reference face of each group based on the average value data and the dense point cloud data of the second reference face of each group among the dense point cloud data of the second reference face of the plurality of groups decide

, 즉

로 나타낼 수 있다. 여기서, 차이값 데이터는 각 그룹의 제2 참조 안면의 덴스 포인트 클라우드 데이터와 복수 그룹의 제2 참조 안면의 덴스 포인트 클라우드 데이터의 평균 특징의 차이를 나타낼 수 있다.Exemplarily, the difference between the dense point cloud data of the second reference face of each group and the coordinate values corresponding to the positions of the different parts of the mean value data is obtained to obtain difference data between the dense point cloud data of the second reference face of each group. decide,

, in other words

can be expressed as Here, the difference value data may indicate a difference in average characteristics between the dense point cloud data of the second reference face of each group and the dense point cloud data of the second reference face of a plurality of groups.

에는 복수의 제2 참조 안면의 덴스 포인트 클라우드 데이터 각각에 대응하는 복수의 서브 차이값 데이터가 포함된다. 예시적으로, N 그룹의 제2 참조 안면의 덴스 포인트 클라우드 데이터가 존재하는 경우, 대응하는 차이값 데이터

는 예를 들면 N 그룹의 제2 참조 안면의 덴스 포인트 클라우드 데이터에 포함된

,

, ……,

각각에 대응하는 서브 차이값 데이터

,

, ……,

가 포함될 수 있는 바, 즉 차이값 데이터

에는 N개의 서브 차이값 데이터가 포함되고

로 나타낼 수 있다.Here, since there is a plurality of second reference face dense point cloud data, difference value data

includes a plurality of sub-difference value data corresponding to each of the plurality of second reference face dense point cloud data. Exemplarily, when there is dense point cloud data of N groups of second reference faces, the corresponding difference value data

is included in the dense point cloud data of the second reference face of the N group, for example.

,

, … … ,

Sub difference value data corresponding to each

,

, … … ,

Can be included, i.e. difference value data

contains N sub-difference data,

can be expressed as

Step S902: Interpolation is performed on difference value data corresponding to each of the dense point cloud data of the second reference face of the plurality of groups based on fitting coefficients corresponding to each of the dense point cloud data of the plurality of groups of the first reference face. .

Exemplarily, a weighted sum of difference value data corresponding to each of the dense point cloud data of the second reference face of the plurality of groups by using the fitting coefficient as a weight corresponding to the dense point cloud data of the second reference face of the plurality of groups. Interpolation processing can be implemented by performing processing.

이 포함된 경우, 복수 장의 참조 이미지 각각에 대응하는 제2 참조 안면의 덴스 포인트 클라우드 데이터의 가중치는

이다. 피팅 계수를 이용하여 복수 그룹의 제2 참조 안면의 덴스 포인트 클라우드 데이터 각각에 대응하는 차이값 데이터에 대해 가중 합계를 수행하여 얻은 결과는

로 나타낼 수 있으며, 원본 안면의 덴스 포인트 클라우드와 제1 참조 안면의 덴스 포인트 클라우드 사이의 연관 관계를 원본 안면의 덴스 포인트 클라우드와 제2 참조 안면의 덴스 포인트 클라우드 사이로 전이시켜, 얻은 안면 덴스 포인트 클라우드가 원본 안면의 덴스 포인트 클라우드의 특징을 가질 뿐만 아니라 제2 참조 안면의 덴스 포인트 클라우드의 스타일을 가지도록 하고; 이러한 경우, 복수 그룹의 제2 참조 안면의 덴스 포인트 클라우드 데이터 각각에 대응하는 차이값 데이터에 대해 가중 합계를 수행하여 얻은 결과

는 아래 수학식(2)를 만족시킨다:to the fitting coefficient

When is included, the weight of the dense point cloud data of the second reference face corresponding to each of the plurality of reference images is

am. The result obtained by performing the weighted sum on the difference value data corresponding to each of the dense point cloud data of the second reference face of the plurality of groups using the fitting coefficient is

It can be expressed as, and the facial dense point cloud obtained by transferring the association between the dense point cloud of the original face and the dense point cloud of the first reference face between the dense point cloud of the original face and the dense point cloud of the second reference face have the style of the dense point cloud of the second reference face as well as the characteristics of the dense point cloud of the original face; In this case, a result obtained by performing a weighted sum on the difference value data corresponding to each of the dense point cloud data of the second reference face of the plurality of groups

satisfies equation (2) below:

은 제2 참조 안면의 덴스 포인트 클라우드 데이터의 가중치

및/또는 복수 그룹의 제2 참조 안면의 덴스 포인트 클라우드 데이터 각각에 대응하는 차이값 데이터

,

, ……,

에 각각 대응하는 가중치를 나타내고, 이는 상이한 차이값 데이터가 가중 합계를 수행하여 얻은 결과의 기여도 및/또는 중요성을 나타내며, 미리 설정하거나 조정할 수 있는 바, 구체적인 설정 및 조정 방법은 여기서 더 이상 설명하지 않는다.here,

is the weight of the dense point cloud data of the second reference face

and/or difference value data corresponding to each of the plurality of groups of second reference face dense point cloud data.

,

, … … ,

Indicates a weight corresponding to each, which indicates the contribution and / or importance of the result obtained by performing a weighted sum of different difference value data, and can be set or adjusted in advance, and specific setting and adjustment methods are not further described herein. .

Step S903: Dense point cloud data of the target facial model is generated according to the result of the interpolation process and the mean value data.

로 나타낼 수 있는 바, 즉 아래 수학식(3)을 이용하여 타깃 안면 모델의 덴스 포인트 클라우드 데이터

를 얻을 수 있다:In one possible embodiment, the result of the weighted sum is directly superimposed on the average value data to generate dense point cloud data of the target facial model,

, that is, the dense point cloud data of the target facial model using Equation (3) below

can be obtained:

를 얻을 수 있다.At this time, the dense point cloud data of the target facial model including not only the facial features in the target image but also the preset style reflected by the dense point cloud data of the second reference face

can be obtained.

In step S104, a target face model corresponding to the target image may be generated using dense point cloud data of the target face model.

에 기반하여, 렌더링의 방식으로 대응하는 타깃 안면 모델을 생성하거나 또는 타깃 안면 모델의 덴스 포인트 클라우드 데이터

와 연관 관계를 가지는 스킨을 결정하고, 타깃 안면 모델의 덴스 포인트 클라우드 데이터

를 이용하여 타깃 이미지에 대응하는 타깃 안면 모델을 생성할 수 있다. 구체적인 방법은 실제 상황에 따라 선택할 수 있는 바, 여기서 더 이상 설명하지 않는다.Exemplarily, dense point cloud data of a target facial model

Based on this, a corresponding target facial model is generated by way of rendering or dense point cloud data of the target facial model.

Determine the skin having a relationship with, and the dense point cloud data of the target facial model

A target face model corresponding to the target image may be generated using . The specific method can be selected according to the actual situation, and is not further described herein.

내의 원본 안면 A에 대응하는 타깃 가상 안면 모델

을 획득하는 구체적인 과정의 설명을 더 제공한다.An embodiment of the present invention uses a facial reconstruction method provided by an embodiment of the present invention to target image

Target virtual face model corresponding to original face A within

A detailed description of the process of obtaining is further provided.

을 결정하는 단계는 아래 단계 (1) 내지 단계 (4)를 포함한다.Target virtual facial model

The step of determining includes steps (1) to (4) below.

를 획득한다.Step (1): preparing materials; Multiple reference images

Acquire

Step (2): prepare dense point cloud data; This includes steps (2-1) and (2-2) below.

중의 원본 안면 A에 대응하는 덴스 포인트 클라우드 데이터

를 결정한다.Step (2-1): target image using neural network

Dense point cloud data corresponding to original face A in

decide

각각에 대응하는 제1 참조 안면의 덴스 포인트 클라우드 데이터

를 결정한다.Step (2-2): Multiple reference images using neural networks

Dense point cloud data of the first reference face corresponding to each

decide

Step (3): determine the fitting coefficient Alpha; This includes steps (3-1) to (3-4) below.

및

를 결정한다.Step (3-1): Dense point cloud data of the first type corresponding to the part of the target facial model

and

decide

Step (3-2): use the dense point cloud data corresponding to the target face model to fit the original face's dense point cloud data to determine an intermediate coefficient;

및

를 조정하여 제1 피팅 계수를 얻는다; Step (3-3): Among the intermediate coefficients corresponding to the first type of dense point cloud data

and

to obtain a first fitting coefficient;

Step (3-4): determining an intermediate coefficient corresponding to a second type of dense point cloud excluding the first type of dense point cloud data from among the dense point cloud data of the first reference face as a second fitting coefficient; A fitting coefficient Alpha is determined using the coefficient and the second fitting coefficient.

을 결정하되; 여기에는 아래 단계 (4-1) 내지 단계 (4-3)가 포함된다.Step (4): Target facial model

determine; This includes steps (4-1) to (4-3) below.

를 결정한다.Step (4-1): average value data of dense point cloud data of second reference faces of multiple groups

decide

, 평균값 데이터

및 복수 그룹의 제1 참조 안면의 덴스 포인트 클라우드 데이터 각각에 대응하는 피팅 계수 Alpha에 기반하여 타깃 안면 모델의 덴스 포인트 클라우드 데이터

를 생성한다.Step (4-2): determine dense point cloud data of the target facial model; Dense point cloud data of multiple groups of second reference faces

, mean data

and dense point cloud data of a target face model based on a fitting coefficient Alpha corresponding to each of the plurality of groups of first reference face dense point cloud data.

generate

를 이용하여 타깃 안면 모델

을 결정한다.Step (4-3): Dense point cloud data of the generated target face model

Target facial model using

decide

Here, it should be noted that steps (1) to (4) are only one specific example of completing the facial reconstruction method, and the facial reconstruction method provided in the embodiment of the present invention is not limited.

A person skilled in the art does not imply that, in the described method of a particular implementation, the order in which the drafting of each step is strictly executed forms any limitation to the implementation process, and the order of drafting each step It can be appreciated that the specific order of execution should be determined by its functionality and possibly its own logic.

Based on the same idea of the present invention, the embodiment of the present invention further provides a face reconstruction device corresponding to the face reconstruction method. Similar to the reconstruction method, the implementation of the device may refer to the implementation of the method, and overlapping parts are not further described.

Referring to FIG. 10 , which is a schematic diagram of a facial reconstruction device provided by an embodiment of the present invention, the device includes a first acquisition module 10, a first processing module 20, and a determination module 30 and a generating module 40; here,

The first acquisition module 10 acquires the original face dense point cloud data included in the target image;

The first processing module 20 uses the dense point cloud data of the first reference face corresponding to each of the plurality of reference images to fit the original face's dense point cloud data to obtain a plurality of groups of the first reference face's dense point cloud data. obtaining fitting coefficients corresponding to each point cloud data;

The determination module 30 determines a target face model based on a fitting coefficient corresponding to each of the dense point cloud data of a plurality of groups of second reference faces having a preset style and the plurality of groups of dense point cloud data of the first reference face. Determine the dense point cloud data of; the second reference faces of the plurality of groups are each generated based on the first reference faces in the plurality of reference images;

The generation module 40 generates a target face model corresponding to the original face of the target image based on dense point cloud data of the target face model.

In one optional embodiment, the determining module 30 determines the plurality of groups of dense point cloud data of the second reference face and the plurality of groups of dense point cloud data of the first reference face having a preset style, respectively. When determining the dense point cloud data of the target facial model based on the fitting coefficient, average value data of the dense point cloud data of the second reference face of the plurality of groups based on the dense point cloud data of the second reference face of the plurality of groups create; Based on the dense point cloud data of the second reference face of the plurality of groups, the average value data, and the fitting coefficient corresponding to each of the plurality of groups of the dense point cloud data of the first reference face of the plurality of groups, the dense point cloud data of the target face model generate

In one optional embodiment, the determining module 30 includes a plurality of groups of dense point cloud data of the second reference face, the average value data, and a plurality of groups of dense point cloud data of the first reference face, respectively. When generating the dense point cloud data of the target face model based on the fitting coefficient, among the dense point cloud data of the second reference face of a plurality of groups, the dense point cloud data of the second reference face of each group and the average value data determining difference value data of dense point cloud data of the second reference face of each group based on; performing interpolation on difference value data corresponding to each of the plurality of groups of dense point cloud data of the second reference face based on fitting coefficients corresponding to each of the plurality of groups of dense point cloud data of the first reference face; Dense point cloud data of the target facial model is generated based on the result of the interpolation process and the average value data.

In one optional embodiment, the first acquisition module 10, when acquiring dense point cloud data of an original face included in the target image, acquires the target image including the original face; Dense point cloud data of the original face in the target image is obtained by processing the target image with a pre-trained neural network.

In one optional embodiment, the device includes: a method of obtaining a plurality of reference images including a first reference face; For each of the reference images in the plurality of reference images, a method of obtaining dense point cloud data of the first reference face in each reference image by processing each reference image with a pre-trained neural network Reference of the plurality of sheets A second processing module 50 for obtaining dense point cloud data of the first reference face corresponding to each image is further included.

In one optional embodiment, the first processing module 20 uses the dense point cloud data of the first reference face corresponding to each of the plurality of reference images to fit the dense point cloud data of the original face to form a plurality of groups When fitting coefficients corresponding to each of the dense point cloud data of the first reference face are obtained, least square processing is performed on the dense point cloud data of the original face and the dense point cloud data of the first reference face to obtain a plurality of groups obtaining intermediate coefficients corresponding to each of the dense point cloud data of the first reference face of ; A fitting coefficient corresponding to the dense point cloud data of the first reference face of each group is determined based on an intermediate coefficient corresponding to the dense point cloud data of the first reference face of each group.

In one optional embodiment, the first processing module 20 determines the dense point cloud data of the first reference face of each group based on the intermediate coefficient corresponding to the dense point cloud data of the first reference face of each group. determining a first type of dense point cloud data corresponding to a part of a target facial model among the first reference faces from among the dense point cloud data of the first reference faces of each group when fitting coefficients corresponding to are determined; obtaining first fitting coefficients by adjusting intermediate coefficients corresponding to the first type of dense point cloud data of the first reference face among the dense point cloud data; determining an intermediate coefficient corresponding to a second type of dense point cloud data among the dense point cloud data of the first reference face as a second fitting coefficient; the second type of dense point cloud data is dense point cloud data excluding the first type of dense point cloud data from the dense point cloud data of the first reference face; A fitting coefficient of the dense point cloud data of the first reference face of each group is obtained based on the first fitting coefficient and the second fitting coefficient.

In one optional embodiment, the device further comprises: adjusting dense point cloud data of a first reference face in the reference image to obtain dense point cloud data of a second reference face having the preset style; or, generating a virtual face image including a second reference face having the preset style based on a first reference face in the reference image; An adjustment module for obtaining dense point cloud data of the second reference face having a preset style through a method of generating dense point cloud data of the second reference face in the virtual face image using a pre-trained neural network ( 60) is further included.

In one optional embodiment, when training the neural network, the device acquires a set of sample images; the sample image set includes a plurality of first sample images including a first sample face; The plurality of first sample images are partitioned into a plurality of subsets of first sample images, and in each subset of first sample images, first sample facial expressions having the same type of expression obtained by collecting each from a plurality of predetermined collection angles are included. images are included; acquire dense point cloud data of a first sample face of a first sample image in the sample image set; perform feature learning on a first sample image in the sample image set with an initial neural network to obtain predicted density point cloud data of a first sample face of the first sample image; The method further includes a training module 70 configured to train the neural network using the dense point cloud data of the first sample face and the predicted dense point cloud data.

In one optional embodiment, the sample image set further includes a plurality of second sample images including a second sample face and a background, and the training module 70 further includes each of the second sample images. acquire facial keypoint data; fitting and generating dense point cloud data of a second sample face of the second sample image using the facial keypoint data of the second sample image and the second sample image; perform feature learning on a second sample image in the sample image set with the neural network to obtain predicted density point cloud data of a second sample face of the second sample image; The neural network is trained using the dense point cloud data of the second sample face and the predicted dense point cloud data.

In one optional embodiment, the set of sample images further includes a third sample image; the third sample image is obtained by performing data augmentation processing on the first sample image;

The training module 70 also acquires dense point cloud data of a third sample face of the third sample image; perform feature learning on a third sample image with the initial neural network to obtain predicted dense point cloud data of a third sample face of the third sample image;

The neural network is trained using the dense point cloud data of the third sample face and the predicted dense point cloud data.

In one optional embodiment, the data augmentation process includes at least one of random occlusion process, Gaussian noise process, motion blur process and color gamut channel change process.

For the description of the processing process of each module in the device and the interaction process between each module, reference may be made to the relevant description in the foregoing method embodiment, which is not described further herein.

As shown in FIG. 11 , an embodiment of the present invention further provides a computer device including a processor 11 and a memory 12; The processor 11 executes machine-readable instructions stored in the memory 12, and when the machine-readable instructions are executed by the processor 11, the processor 11:

obtaining dense point cloud data of an original face included in a target image; Fitting the dense point cloud data of the original face using the dense point cloud data of the first reference face corresponding to each of the plurality of reference images to obtain a fitting coefficient corresponding to each of the dense point cloud data of the first reference face of the plurality of groups step; determining dense point cloud data of a target facial model based on dense point cloud data of a second reference face having a preset style and fitting coefficients corresponding to each of the plurality of groups of dense point cloud data of the first reference face; the second reference face is generated based on the first reference face in the reference image; A step of generating a target face model corresponding to the original face of the target image based on the dense point cloud data of the target face model is performed.

The memory 12 includes an internal memory 121 and an external memory 122; The internal memory 121 herein is also referred to as an internal storage device, and is for temporarily storing data exchanged with the external memory 122, such as operation data and hardware in the processor 11, and the processor 11 has an internal memory ( 121) to exchange data with the external memory 122.

A specific execution process of the command may refer to steps of a facial reconstruction method according to an embodiment of the present invention, and is not described herein any further.

An embodiment of the present invention further provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the facial reconstruction method according to the method embodiment do the steps Here, the storage medium may be a volatile or non-volatile computer readable storage medium.

Embodiments of the present invention further provide a computer program product, wherein the computer program product is loaded with program code, and instructions included in the program code can perform steps of the facial reconstruction method according to the method embodiment; , Specific details may refer to the foregoing method embodiments, which are not described herein any further.

Here, the computer program product in question may be implemented in hardware, software, or a combination thereof. In one possible embodiment, the computer program product may be specifically embodied in a computer storage medium, and in another possible embodiment, the computer program product may be specifically a software product, for example, a software development kit (Software Development Kit, SDK) and the like.

Those skilled in the art can clearly understand that for convenience and simplification of description, the specific working processes of the system and device may refer to the corresponding processes of the method embodiments, so they are not described herein. In some embodiments provided in this application, it will be understood that the disclosed devices and methods may also be implemented in other ways. For example, the device embodiment is merely illustrative, for example, the division of the unit is only a logical division, and there may be other division methods in actual implementation, for example, a plurality of units or modules may be combined. or may be integrated into other systems, or some features may be ignored or not implemented. and the mutual coupling or direct coupling or communication connection indicated or discussed above may be direct coupling, indirect coupling or communication connection through some communication interface, device or unit, which may be electrical, mechanical or otherwise. there is.

A unit described as a separate component may or may not be physically separate, and a component referred to as a unit may or may not be a physical unit, and may be located in one place or distributed over multiple network units. According to actual needs, some or all units may be selected to realize the purpose of the solutions in the embodiments of the present application.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist alone, or two or more units may be integrated into one unit.

If the integrated unit is implemented in the form of a software functional unit and sold or used as a separate product, it may be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or a part contributing to the prior art, or all or part of the technical solution may be implemented in the form of a software product, and the computer software product is a storage unit. It is stored in a medium, and includes some instructions so that one computer device (which may be a personal computer, server, network device, etc.) performs all or part of the steps of the method according to each embodiment of the present invention. The aforementioned storage medium includes various media capable of storing program codes such as USB memory, external hard drive, read-only memory (ROM), random access memory (RAM), diskette or CD. .

Finally, as should be explained, the above embodiments are used to describe the technical solutions of the present application, but are not limited thereto, and the present application has been described in detail with reference to each of the above embodiments, but the present technology Those skilled in the art may modify the technical solutions described in each of the above embodiments, or make equivalents to some or all of the technical features thereof, on the premise that they do not depart from the scope of the technical solutions of the present application described in the claims below. Replacement may be carried out; It should be understood that such modifications, changes or replacements do not deviate from the spirit and scope of the technical solutions in the embodiments of the present invention, and should all be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (15)

As a facial reconstruction method,
obtaining dense point cloud data of an original face included in a target image;
Fitting coefficients corresponding to each of the dense point cloud data of the first reference face of a plurality of groups by fitting the dense point cloud data of the original face using the dense point cloud data of the first reference face corresponding to each of the plurality of reference images obtaining;
Determining dense point cloud data of a target face model based on dense point cloud data of a plurality of groups of second reference faces having a preset style and fitting coefficients corresponding to each of the plurality of groups of dense point cloud data of the first reference face the step of: the second reference faces of the plurality of groups are each generated based on the first reference faces in the plurality of reference images;
and generating a target facial model corresponding to the original face of the target image based on dense point cloud data of the target facial model. According to claim 1,
Determining dense point cloud data of a target face model based on dense point cloud data of a plurality of groups of second reference faces having a preset style and fitting coefficients corresponding to each of the plurality of groups of dense point cloud data of the first reference face The steps are,
generating average value data of multiple groups of dense point cloud data of the second reference face based on multiple groups of dense point cloud data of the second reference face;
Based on the dense point cloud data of the second reference face of the plurality of groups, the average value data, and the fitting coefficient corresponding to each of the plurality of groups of the dense point cloud data of the first reference face of the plurality of groups, the dense point cloud data of the target face model A facial reconstruction method comprising the step of generating. According to claim 2,
Based on the dense point cloud data of the second reference face of the plurality of groups, the average value data, and the fitting coefficient corresponding to each of the plurality of groups of the dense point cloud data of the first reference face of the plurality of groups, the dense point cloud data of the target face model Steps to create are:
A difference value between the dense point cloud data of the second reference face of each group among the dense point cloud data of the second reference face of a plurality of groups and the dense point cloud data of the second reference face of each group based on the average value data determining data;
Performing interpolation on difference value data corresponding to each of the plurality of groups of dense point cloud data of the second reference face based on fitting coefficients corresponding to each of the plurality of groups of dense point cloud data of the first reference face. ;
and generating dense point cloud data of the target facial model based on a result of the interpolation process and the average value data. According to any one of claims 1 to 3,
The step of obtaining dense point cloud data of the original face included in the target image,
obtaining the target image including the original face;
and processing the target image using a pre-trained neural network to obtain dense point cloud data of the original face in the target image. According to any one of claims 1 to 4,
Dense point cloud data of a first reference face corresponding to each of the plurality of reference images,
a method of acquiring a plurality of reference images including a first reference face;
For each of the reference images in the plurality of reference images, a method of obtaining dense point cloud data of the first reference face in each of the reference images by processing each of the reference images using a pre-trained neural network. A facial reconstruction method, characterized in that obtained. According to any one of claims 1 to 5,
Fitting coefficients corresponding to each of the dense point cloud data of the first reference face of a plurality of groups by fitting the dense point cloud data of the original face using the dense point cloud data of the first reference face corresponding to each of the plurality of reference images The steps to get
obtaining an intermediate coefficient corresponding to each of a plurality of groups of dense point cloud data of the first reference face by performing least square processing on the dense point cloud data of the original face and the dense point cloud data of the first reference face;
Determining a fitting coefficient corresponding to the dense point cloud data of the first reference face of each group based on the intermediate coefficient corresponding to the dense point cloud data of the first reference face of each group. reconstruction method. According to claim 6,
Determining fitting coefficients corresponding to the dense point cloud data of the first reference face of each group based on the intermediate coefficients corresponding to the dense point cloud data of the first reference face of each group,
determining a first type of dense point cloud data corresponding to a part of a target facial model among the first reference faces from the dense point cloud data of the first reference faces of each group;
obtaining first fitting coefficients by adjusting intermediate coefficients corresponding to the first type of dense point cloud data among the dense point cloud data of the first reference face;
Determining an intermediate coefficient corresponding to a second type of dense point cloud data of the first reference face among the dense point cloud data of the first reference face as a second fitting coefficient - the second type of dense point cloud data of the first reference face Dense point cloud data excluding the first type of dense point cloud data from the dense point cloud data -;
and obtaining a fitting coefficient of dense point cloud data of the first reference face of each group based on the first fitting coefficient and the second fitting coefficient. According to any one of claims 1 to 7,
Dense point cloud data of the second reference face having a preset style,
a method of obtaining dense point cloud data of a second reference face having the preset style by adjusting dense point cloud data of a first reference face in the reference image; or,
generating a virtual face image including a second reference face having the preset style based on a first reference face in the reference image; The face reconstruction method, characterized in that obtained through a method of generating dense point cloud data of the second reference face in the virtual face image using a pre-trained neural network. The method of any one of claims 4, 5 and 8,
Training the neural network,
obtaining a sample image set, wherein the sample image set includes a plurality of first sample images including a first sample face; The plurality of first sample images are partitioned into a plurality of subsets of first sample images, and in each subset of first sample images, first sample facial expressions having the same type of expression obtained by collecting each from a plurality of predetermined collection angles are included. image included -;
obtaining dense point cloud data of a first sample face of a first sample image in the sample image set;
obtaining predicted density point cloud data of a first sample face of the first sample image by performing feature learning on a first sample image in the sample image set with the neural network;
and training the neural network using the dense point cloud data of the first sample face and the predicted dense point cloud data. According to claim 9,
The sample image set further includes a plurality of second sample images including a second sample face and a background, and the training of the neural network comprises:
acquiring facial keypoint data of each of the second sample images;
fitting and generating dense point cloud data of a second sample face of the second sample image using facial keypoint data of the second sample image and the second sample image;
obtaining predicted density point cloud data of a second sample face of the second sample image by performing feature learning on a second sample image in the sample image set with the neural network;
The face reconstruction method of claim 1, further comprising training the neural network using the dense point cloud data of the second sample face and the predicted dense point cloud data. The method of claim 9 or 10,
the sample image set further includes a third sample image; the third sample image is obtained by performing data augmentation processing on the first sample image; Training the neural network,
obtaining dense point cloud data of a third sample face of the third sample image;
obtaining predicted density point cloud data of a third sample face of the third sample image by performing feature learning on the third sample image with the neural network;
The facial reconstruction method of claim 1 , further comprising training the neural network using the dense point cloud data of the third sample face and the predicted dense point cloud data. According to claim 11,
The face reconstruction method, characterized in that the data augmentation process includes at least one of random occlusion process, Gaussian noise process, motion blur process, and color gamut channel change process. As a facial reconstruction device,
a first acquiring module for obtaining dense point cloud data of an original face included in a target image;
Fitting coefficients corresponding to each of the dense point cloud data of the first reference face of a plurality of groups by fitting the dense point cloud data of the original face using the dense point cloud data of the first reference face corresponding to each of the plurality of reference images A first processing module to obtain;
Determining dense point cloud data of a target face model based on dense point cloud data of a plurality of groups of second reference faces having a preset style and fitting coefficients corresponding to each of the plurality of groups of dense point cloud data of the first reference face but; a determination module, wherein each of the plurality of groups of second reference faces is generated based on a first reference face in the plurality of reference images;
and a generation module for generating a target facial model corresponding to the original face of the target image based on dense point cloud data of the target facial model. A computer device comprising a processor and memory,
Machine-readable instructions executable by the processor are stored in the memory, the processor executes the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the processor performs a first A computer device for performing the steps of the facial reconstruction method according to any one of claims 1 to 12. A computer-readable storage medium in which a computer program is stored,
When the computer program is executed by a computer device, the computer device performs the steps of the facial reconstruction method according to any one of claims 1 to 12. A computer-readable storage medium.

Images