WO2021027585A1

WO2021027585A1 - Human face image processing method and electronic device

Info

Publication number: WO2021027585A1
Application number: PCT/CN2020/105873
Authority: WO
Inventors: 丁欣; 王利强; 周恒�
Original assignee: 华为技术有限公司
Priority date: 2019-08-09
Filing date: 2020-07-30
Publication date: 2021-02-18
Also published as: CN112348937A

Abstract

A human face image processing method, comprising: an electronic device obtains a two-dimensional image to be processed, constructs a three-dimensional grid model corresponding to said image according to a preset reference grid, obtains a texture map of the three-dimensional grid model according to a photographing parameter of said image, and determines a boundary point and a control point corresponding thereto according to the visible boundary of the face of the reference grid; the electronic device deforms the three-dimensional grid model in combination with the correspondence between the boundary point and the control point according to the preset deformation requirement, renders the texture map to the deformed three-dimensional grid model, and generates a processed image according to the rendered three-dimensional grid model.

Description

Human face image processing method and electronic equipment

This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on August 9, 2019, the application number is 201910735963.9, and the application name is "face image processing method and electronic equipment", the entire content of which is incorporated herein by reference Applying.

Technical field

This application relates to the field of image processing, and in particular to a face image processing method and electronic equipment.

Background technique

With the development of camera technology, the camera function is more and more widely used in smart devices, such as smart phones, tablet computers, and notebook computers. The camera function can facilitate users to collect images, including taking photos or videos, making smart devices more popular with users.

When using the camera function of a smart device, since the camera device of the smart device is generally set in an area outside the screen, when the user looks at the screen, the facial posture in the image collected by the camera device will be tilted. Correct the face posture. The current method of deformation fusion based on 3D face reconstruction after reconstruction takes a long time to generate a corrected 2D face and fusion with the background, especially when the camera function is used for video calls, which cannot achieve real-time processing. Requirements.

Summary of the invention

The present application provides a face image processing method and electronic device to solve the problem that when correcting the face posture in the image in the prior art, it takes a long time and cannot meet the real-time requirements.

In order to achieve the above objectives, this application adopts the following technical solutions:

In the first aspect, an embodiment of the present application provides a face image processing method for a face image. The face image processing method includes: an electronic device acquires a two-dimensional image to be processed, and constructs a face image according to a preset reference grid. The three-dimensional mesh model corresponding to the two-dimensional image to be processed, the texture map of the three-dimensional mesh model is obtained according to the shooting parameters of the two-dimensional image to be processed, and the visible boundary of the face of the reference mesh is determined Boundary points, and control points corresponding to the boundary points; the electronic device performs deformation processing on the three-dimensional mesh model according to preset deformation requirements and in combination with the corresponding relationship between the boundary points and the control points, and The texture image is rendered to the deformed three-dimensional mesh model, and the processed image is generated according to the rendered three-dimensional mesh model.

It can be seen that the electronic device acquires shooting parameters according to the acquired two-dimensional image to be processed, builds a three-dimensional grid model according to the two-dimensional image combined with the reference grid, determines the boundary points and control points according to the reference grid, and according to the preset deformation It is required to combine the boundary points and control points to deform the 3D mesh model, which can effectively improve the efficiency of the deformation processing of the 3D mesh model, and render the processed 3D mesh model to the texture image, thereby improving the electronic equipment The efficiency of image processing can better meet the timeliness requirements of electronic equipment for real-time image processing. For example, it can be applied to real-time processing of video images to reduce the delay of processed images and improve user experience.

Possibly, the electronic device extracts video frames in real time from the video collected by the camera, and uses the extracted video frames as the two-dimensional images to be processed; or, the electronic device uses the photos taken by the camera as the two-dimensional images to be processed. Dimensional image. After acquiring the two-dimensional image to be processed, it can be detected whether the two-dimensional image includes a face image. If the two-dimensional image includes a face image, the face image processing method described in this application is activated and the preset Build a three-dimensional grid model based on a predetermined reference grid. If no face image is detected in the two-dimensional image, the collected image can be directly displayed.

In a possible implementation manner, the reference grid includes a three-dimensional face model reference grid and a background plane reference grid, and the three-dimensional grid model includes a three-dimensional face model modeling grid and a background plane modeling grid. The step of the electronic device constructing a three-dimensional grid model corresponding to the two-dimensional image to be processed according to a preset reference grid includes: the electronic device combines the reference grid of the three-dimensional face model with the two-dimensional image to be processed Fit, and obtain the shooting parameters of the two-dimensional image to be processed according to the fitted reference grid of the three-dimensional face model; according to the shooting parameters, the electronic device poses the reference grid of the three-dimensional face model Through adjustment, a three-dimensional face model modeling grid is obtained, and the three-dimensional face model modeling grid is consistent with the face pose in the two-dimensional image.

Exemplarily, the reference grid of the three-dimensional face model in the reference grid is a general face model or a three-dimensional deformed model. A fitting operation is performed on the reference grid based on the features of the face image included in the two-dimensional image to be processed, so that after the reference grid is transformed, the reference grid including the face features is obtained. Then, in combination with the posture features in the two-dimensional image, the reference grid can be rotated by the rotation vector in the shooting parameters in this application, so that the three-dimensional face model can effectively match the face in the two-dimensional image. Posture matching.

In a possible implementation manner, the shooting parameters include a model view matrix and a projection matrix, and the step of adjusting the posture of the reference grid of the three-dimensional face model by the electronic device according to the shooting parameters includes: In the model view matrix, the electronic device extracts the rotation component; according to the extracted rotation component, the electronic device controls the fitted reference grid of the three-dimensional face model to rotate to the face pose corresponding to the two-dimensional image to be processed.

Of course, when the pose adjustment is performed on the reference grid of the three-dimensional face model, it is not limited to this, and the rotation of the reference grid of the three-dimensional face model can also be determined by means of feature point combination or feature comparison. Component, rotate the reference grid of the fitted 3D face model.

In a possible implementation manner, the step of the electronic device constructing a three-dimensional grid model corresponding to the two-dimensional image to be processed according to a preset reference grid further includes: determining a three-dimensional face model after posture adjustment Boundary points in the modeling grid that have changed positions; search for corresponding control points according to the changed positional boundary points, and perform deformation control on the background plane reference grid according to the searched control points.

It can be seen that by setting the background plane modeling grid and the 3D face model modeling grid, after adjusting the 3D face model modeling grid, the background plane can be quickly constructed according to the changes in the boundary points. The adaptive adjustment of the mold grid is helpful to improve the response speed of adjustment. Wherein, the three-dimensional face model modeling grid is obtained by fitting a reference grid of a three-dimensional face model with a face image in a two-dimensional image, and the background plane modeling grid can be obtained after a reference grid of a three-dimensional face model When the grid is fitted or the attitude is adjusted, the background plane reference grid is adjusted according to the relationship between the boundary points and the control points.

In a possible implementation, the texture map includes a three-dimensional face model grid texture map and a background plane grid texture map, and the electronic device obtains the three-dimensional face model according to the shooting parameters of the two-dimensional image to be processed. The step of the texture map of the grid model includes: according to the model view matrix and the projection matrix, the electronic device obtains the three-dimensional face model grid texture map; according to the projection matrix and the translation vector in the model view matrix And a zoom vector, the electronic device obtains the background plane grid texture map.

Wherein, according to the model view matrix and the projection matrix, the step of obtaining the three-dimensional face model mesh texture map by the electronic device may include: the electronic device obtaining the three-dimensional face model modeling grid The coordinates of the vertices in the space rectangular coordinate system, when the z coordinate in the vertices coordinates is 0, the first plane is rendered; the electronic device is based on the position of the first pixel of the first plane and the model view matrix and projection The product of the matrix determines the second pixel corresponding to the first pixel on the two-dimensional image to be processed, and the color of the first pixel is determined according to the color of the second pixel.

Wherein, according to the projection matrix, and the translation vector and the zoom vector in the model view matrix, the step of obtaining the background plane grid texture map by the electronic device may include: the electronic device modeling the network according to the background plane The grid determines the second plane, and extracts the translation matrix and the zoom matrix in the model view matrix; the electronic device determines the second plane according to the position of each third pixel in the second plane and the product of the translation matrix, the zoom matrix, and the projection matrix. For the fourth pixel point corresponding to the three-pixel point on the two-dimensional image to be processed, the color of the third pixel point is determined according to the color of the fourth pixel point.

It can be seen that through the model view matrix and the projection matrix, the correspondence between the pixels in the two-dimensional image and the texture map (including the three-dimensional face model grid texture map and the background plane grid texture map) can be determined, and the background plane network The lattice texture map does not need to be rotated, you can use the zoom vector and the translation vector in the model view matrix. By generating the texture map of the 3D mesh model, it is convenient to quickly generate the image of the 3D mesh model after the 3D mesh model is transformed. Not limited to this, the texture map of the three-dimensional mesh model can also be generated by means of feature point matching.

In a possible implementation manner, the step of the electronic device deforming the three-dimensional mesh model includes: the electronic device obtains the information of the three-dimensional face model modeling mesh in the constructed three-dimensional mesh model Posture; the electronic device rotates the three-dimensional face model modeling grid according to the angle relationship between the posture of the constructed three-dimensional face model modeling grid and the target posture.

Exemplarily, when the angle of the user image collected by the electronic device is the looking-up angle, and the obtained two-dimensional image presents features such as double chin and upward nose, when the two-dimensional image is adjusted, the determined target posture is The user image corresponding to the horizontal angle. The angle relationship between the posture of the constructed three-dimensional face model modeling grid and the target posture is the angle between the upward angle of the image collected by the electronic device and the horizontal line. The three-dimensional face model modeling grid is rotated downward according to the included angle, so that the posture of the three-dimensional face model modeling grid faces forward horizontally.

In a possible implementation manner, the step of the electronic device performing deformation processing on the three-dimensional face model modeling grid according to preset deformation requirements includes: the electronic device obtains preset face beautification parameters ; According to the face beautification parameters, the electronic device adjusts the three-dimensional face model modeling grid in the three-dimensional grid model.

In a possible implementation manner, the face beautification parameters include one or more of eye size parameters, eye spacing parameters, face fatness and thinness parameters, mouth size parameters, eye bag removal parameters, face shape parameters, and nose wings size parameters.

Exemplarily, when an electronic device receives a user's beautification request, it can select preset eye size parameters, eye spacing parameters, face fatness and thinness parameters, mouth size parameters, eye bag removal parameters, and face shape according to the beautification request. One or more of the parameters and the wing size parameters are used as the face beautification parameters corresponding to the current beautification request. The modeling grid of the three-dimensional face model is adjusted according to the face beautification parameters. When adjusting, the distance between the feature points in the 3D face model modeling grid can be adjusted according to a certain ratio. For example, when the face beautification parameters include face shape parameters, two or more than two sets of feature point pairs can be selected to characterize the face width, and the distance of the feature point pairs conforms to a predetermined ratio relationship. According to the pre-set ratio of the distance between the feature point pairs, adjust the ratio of the distance between the corresponding feature point pairs in the current three-dimensional face model modeling grid to make it consistent with the preset ratio relationship. Realize the beautification of the face, such as adjusting the width of the chin relative to the face, so that the face shape of melon seeds can be obtained after beautification.

In an implementation manner, the step of performing deformation processing on the three-dimensional mesh model by the electronic device in combination with the corresponding relationship between the boundary points and the control points includes: the electronic device acquires that the boundary points are located in the reference grid The first position on the reference grid of the three-dimensional face model, and the boundary point is located at the second position on the three-dimensional face model modeling grid in the three-dimensional grid model; the difference between the second position and the first position When the distance is greater than a predetermined value, the electronic device searches for a control point corresponding to the boundary point; the electronic device performs deformation processing on the background plane modeling grid according to the searched control point.

In a possible implementation manner, the step of the electronic device performing deformation processing on the background plane modeling grid according to the searched control point includes: the electronic device obtains the coordinate position of the boundary point on the background plane According to the coordinate change of the coordinate position of the boundary point in the background plane, the electronic device determines the target position of the control point; according to the target position, the electronic device models the background plane Grid performs Laplace deformation processing.

It can be seen that when the reference grid of the 3D face model is deformed, it includes processing methods such as fitting or posture adjustment. The background plane reference grid can quickly determine the target transformation position according to the corresponding relationship between the boundary points and the control points. , Thereby quickly completing the adjustment to the background plane reference grid, thereby improving the response speed of image processing.

Wherein, when the corresponding control point is determined according to the coordinate position of the boundary point, the vertical projection of the boundary point on the background plane can be obtained, or the two-dimensional coordinates can be obtained by intercepting the three-dimensional coordinates of the boundary point The corresponding control.

In a possible implementation manner, according to the target position, the step of the electronic device performing Laplace deformation processing on the background plane modeling grid includes: the electronic device obtains the background plane modeling grid According to the set control point and the target position of the control point, the electronic device performs Laplace deformation processing on the background plane modeling grid.

It can be seen that by performing Laplacian deformation processing on the background plane modeling grid with the target position of the control point in the background plane modeling grid, the background plane modeling grid can be combined with the three-dimensional face model. The modeling grid is effectively fused to improve the authenticity of the transformed image and avoid image gaps.

In a possible implementation manner, before the step of acquiring the two-dimensional image to be processed by the electronic device, the method further includes: the electronic device constructing a three-dimensional face model reference grid and a background plane reference grid; The electronic device obtains the face area in the reference grid of the three-dimensional face model; according to the visible boundary of the face area, the electronic device determines the positions of the boundary points and the control points.

In a second aspect, the present application provides an electronic device that includes a memory, a processing screen, and a computer program. The display screen is used for processed images. The computer program is stored in the memory. The computer program includes instructions, which when executed by the electronic device, cause the electronic device to execute the face image processing method according to any one of the first aspect.

In a third aspect, the present application provides a computer-readable storage medium that stores a computer program that, when executed by a processor, realizes the face image described in any one of the first aspect Approach.

In a fourth aspect, this application provides a computer program product containing instructions that, when the computer program product runs on an electronic device, causes the electronic device to execute the face image processing method described in any one of the first aspect.

Understandably, the electronic equipment described in the second aspect, the computer storage medium described in the third aspect, and the computer program product described in the fourth aspect provided above are all used to execute the corresponding methods provided above. For the beneficial effects that can be achieved, please refer to the beneficial effects in the corresponding method provided above, which will not be repeated here.

Description of the drawings

FIG. 1 is a schematic structural diagram of a hidden camera provided by an embodiment of the application;

2 is a schematic diagram of using a hidden camera to make a video call according to an embodiment of the application;

FIG. 3 is a schematic diagram of the effect of an image captured by a hidden camera provided by an embodiment of the application;

4 is a schematic diagram of the comparison effect of face image processing based on a target image matching method provided by an embodiment of the application;

FIG. 5 is a schematic flowchart of a face image processing provided by an embodiment of the application;

Fig. 6 is a schematic diagram of a general face model provided by an embodiment of the application;

FIG. 7 is a schematic diagram of a three-dimensional deformation model provided by an embodiment of the application;

FIG. 8 is a schematic diagram of a reference grid provided by an embodiment of this application;

FIG. 9 is a schematic diagram of marking boundary points at the front right perspective of a reference grid according to an embodiment of the application;

FIG. 10 is a schematic diagram of marking boundary points on the front of a reference grid according to an embodiment of the application;

11 is a schematic diagram of a modeling mesh after deformation processing provided by an embodiment of the application;

12 is a schematic diagram of a mesh texture map of a three-dimensional face model provided by an embodiment of the application;

FIG. 13 is a schematic diagram of a background plane grid texture map provided by an embodiment of this application;

14a and 14b are respectively a side view and a front view of a three-dimensional face model modeling grid after deformation processing provided by an embodiment of the application;

15 is a schematic diagram of an image after posture adjustment and rendering provided by an embodiment of the application;

16a and 16b are respectively a front view and a side view of a three-dimensional grid model after adjusting the background plane modeling grid according to an embodiment of the application;

FIG. 17 is a schematic structural diagram of an image processing device provided by an embodiment of this application;

FIG. 18 is a schematic structural diagram of an electronic device provided by an embodiment of this application;

FIG. 19 is a block diagram of the software structure of an electronic device provided by an embodiment of the application.

detailed description

In the following description, for the purpose of illustration rather than limitation, specific details such as a specific system structure and technology are proposed for a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, this application omits detailed descriptions of well-known systems, devices, circuits, and methods, so as to avoid unnecessary details from obstructing the description of this application.

The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions "a", "an", "said", "above", "the" and "this" are intended to also This includes expressions such as "one or more" unless the context clearly indicates to the contrary. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two, or more than two; "and/or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and/or B can mean: A alone exists, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character "/" generally indicates that the associated objects are in an "or" relationship.

The main purpose of the embodiments of the present application is to solve the problems of poor transformation effect and slow response speed when processing face images collected in real time in the existing face image processing. The details are as follows:

The face image processing methods described in this application may include transforming the pose or angle of the face in the image, or performing special effects processing on the face in the image, including collecting facial beauty, fattening, thinning, and mirroring. And other face image processing methods.

In the application scenario of transforming the face posture in the image, in order to increase the screen ratio and solve the problem of anti-candid shooting, a notebook with a hidden camera appears. The camera may be a hidden camera with a keyboard set at the bottom of the screen. As shown in Figure 1, the hidden camera can choose to set a special button in the middle of the keyboard, and install the camera through this button. When the camera is set on the button, the camera can be set on the side of the button through a pressing structure, for example, the camera is set on the front side of the button, or when the button is a push button that can be rotated, it can be set at any side.

When the user needs to use the camera, he can press the button. The rear side of the button can be fixed by a rotating shaft. The front side of the button can be popped up by the elastic force of the elastic member, so that the side of the keyboard where the camera is located is in line with the keyboard. The angle between the front keyboard planes is greater than 90 degrees, such as 120 degrees, 135 degrees, etc., so that the camera can collect a complete face image of the user who normally uses the notebook.

Of course, the camera can also be set to be flexibly adjustable to multiple angles. The rotation angle of the camera can be adjusted according to the integrity of the face image currently detected. For example, when the user is close to the screen, if only the lower part of the user's face image can be detected according to the image collected by the camera, the camera can be controlled to rotate upward until the complete face image is obtained, or the user is far away from the screen If the face of the collected image is located in the lower part of the image, the angle of the camera can be adjusted so that the face image is located in the middle of the image.

When the user uses the notebook to make a video call, as shown in Figure 2, the user’s eye gaze direction is usually in the direction corresponding to the screen area during the video call. The hidden camera captures the face at an elevation angle, so that the user is in When making a video call or taking a photo, there is a certain angle β between the camera's shooting angle and the user's gaze angle, and the captured face image will show a "big chin", "upward nose", etc. as shown in Figure 3 The image effect does not meet the requirements of the shooting angle of view desired by the user. In addition, during a video call, the gaze angle of the face in the captured image usually shows the illusion of not looking at or paying attention to the other party (actually the user is looking at the other party on the screen), which is not conducive to improving users Communication experience.

Or, when a desktop computer acquires a user's face image through an external camera, or a tablet or a laptop acquires a user's face image through a camera set above or below the screen, the difference between the position of the camera and the center of the screen makes the captured image The angle of the image has a certain depression or elevation angle, and it is also necessary to transform the face through the face image processing method, so that the image can display the desired shooting angle of view, and improve the user's call experience during video calls.

In the application scenario of special effects processing on the face in the image, the smart terminal can adjust the facial features in the captured image in order to obtain a better shooting effect when collecting photos or videos that include the user's face image. Or adjust the figure and so on. This is convenient for users to publish adjusted photos or videos, including, for example, users publishing short videos through small video applications (including applications such as Douyin, Kuaishou, etc.), or publishing photos or small videos through platforms such as WeChat, Weibo, and blogs. Video etc.

In order to transform the collected two-dimensional image, transform the collected face image into a positive face image, or transform the collected face image into an image processed by special effects, the following face transformation methods can be included:

Processing the collected face image in a two-dimensional transformation manner may include: first obtaining a face photo of the user as reference data for subsequent image or video processing, and detecting feature points of the face photo. When the photo to be transformed is obtained, the feature points in the photo to be transformed are recognized. The captured image is deformed according to the pre-stored front face photo, and the sight direction and the orientation of the facial features in the deformed face image remain unchanged.

As shown in Figure 4, the picture on the left is the photo to be transformed. The photo to be converted may be the user using an electronic device, for example, when using a laptop with a hidden camera function to make a video call, or using a laptop with a hidden camera function to take a video, or the screen position of other cameras and electronic devices In the case of mismatched electronic devices, since the user generally looks at the screen area, the camera is set in front of and below the face, and the camera's collection angle is the upward viewing angle. The chin and nostrils of the collected images are large, and the user experience is not good. Of course, this figure shows the effect of the camera set at the bottom of the screen. When the camera is set at a position other than the screen, the image obtained is also not the frontal image expected by the user. The middle image in Figure 4 is a deformed photo using the pre-stored front face photo as the target. Although the entire face shape has changed, the sight direction and facial features of the face have not changed, while the target in the right image in Figure 4 It can be seen from the renderings that, in order to be able to effectively process the face image, it is necessary to change the line of sight of the eyes in the face, and it is necessary to change the display perspective of the nose in the face.

Secondly, when transforming the face through the target image, the feature points are usually used to create a grid on the picture, and for each triangle in the grid, the image is mapped according to the transformation matrix or function, and it needs to be traversed during the transformation process. For each pixel position, the pixel color is calculated by interpolation. The larger the deformation processing range, the greater the processing time that needs to be spent. Under normal circumstances, it takes 4-6ms to deform only the facial features and only the face The time spent in deformation is usually 20-40ms, and the time required for full image transformation of the collected image is 130-170ms. Therefore, if in order to obtain a better image effect, a larger processing delay will occur.

The three-dimensional deformation + background two-dimensional fusion method is used to perform three-dimensional reconstruction according to the collected images, deform the reconstructed face in three-dimensional space, and then reproject the deformed face into a two-dimensional face target image, and When integrating a two-dimensional face target image with a background image, if only the face is reconstructed, deformed and projected, it usually takes 30-40ms to complete. However, there will be a gap between the face image and the background image, which will affect The real effect of the image. When fusing the projected two-dimensional face target image with the background image, it takes a long time, and it usually takes 35-55ms to fuse the face image with the background image. The process of generating a corrected two-dimensional face model and fusion requires 65-95ms, which cannot meet the requirements for real-time video processing.

In order to solve the problem in the prior art that when the face image is processed, the perspective of the facial features is not changed correspondingly, or the time required to complete the face image processing is long, the embodiment of the present application constructs a three-dimensional face model benchmark Grid and background plane datum grid, when transforming the 3D face model datum grid, adjust the background plane datum grid accordingly through the boundary points of the 3D face model datum grid and the background plane datum grid control point, Thereby improving the efficiency of face image processing.

Wherein, the reference grid of the three-dimensional face model may be a grid model of the three-dimensional face model formed by polygons (including triangles, quadrilaterals, etc.). The three-dimensional face model selected by the three-dimensional face reference grid may be a general face model, or may be a three-dimensional face model of the user who uses the image processing device.

The background plane reference grid is a plane grid corresponding to the size of the image to be processed. The background plane grid can be composed of a uniform grid or a non-uniform grid.

When processing the image to be adjusted according to the face image processing method described in this application, it can usually be shown in Figure 5, which mainly includes the following three processing steps: S1 reference grid preprocessing, S2 current image modeling, and S3 current image processing . The respective processing procedures are further described below in conjunction with the drawings.

S1. Preprocessing of reference grid:

S11 builds a reference grid

The reference grid to be constructed may include a three-dimensional face model reference grid and a background plane reference grid.

The reference grid of the three-dimensional face model may be a general face model, a three-dimensional deformable model (3DMM, 3D Morphable Models), or a variant model thereof.

Wherein, the general face model is a three-dimensional face model grid established based on a general face, and may include, for example, a CANDIDE-3 model, a CANDIDE model, and the like. As shown in Figure 6, the CANDIDE-3 model includes 113 vertices and 168 faces. Through the operation and adjustment of these points and faces, the feature points in the general face model and the two-dimensional image collected by the camera (requires three-dimensional face The facial features in the reconstructed image) are matched. The advantage of using a general face model is that the amount of calculation is small, and it can quickly respond to the reconstruction of the three-dimensional face image.

The general face model may be obtained by using a three-dimensional scanner to obtain the data of the general face model, or may be created by computer graphics technology, or the general face model may be generated by commercial modeling software.

In an optional implementation manner, the universal face model may acquire a user of the image processing device, and establish a universal face model corresponding to the user.

For example, the user's face model can be scanned as the general face model. When the electronic device has multiple users, the general face models corresponding to the multiple users can be stored, and the general face models corresponding to the users can be selected according to the user currently using the electronic device.

For example, when a mobile phone user logs in to an electronic device with a fingerprint or account, he can learn the user who is currently using the device. According to the pre-stored correspondence between the user and the general face model, search for the general face model corresponding to the current user. If during use, the matching degree between the current user and the general face model is lower than the predetermined matching degree, such as when a new user uses the general face model of a small number of users stored in the image processing device. A general face model corresponding to the user's group may be selected according to the user's facial features. For example, the user's group may be a group determined by middle-aged men, young girls, etc.

If you build their own general face models for users, you can improve the efficiency of fitting the collected images to the reference grid of the three-dimensional face model during subsequent reconstruction of the three-dimensional face model, which is beneficial to improve the response speed of the device. For example, when a user uses a laptop computer with hidden camera function to make a video call, or when shooting a video, the real-time images of the big chin, big nostril, and upward viewing angle taken by the laptop can be fitted for processing, which can improve the response speed after fitting processing , Improve the real-time performance of video display.

Of course, if the current user does not have a preset universal face model, a preset universal face model corresponding to the user group can be used. When the user has used the user for a predetermined number of times, the facial features of the user can be collected, and the corresponding relationship between the user and the general face model can be added to the system.

When the three-dimensional deformed model 3DMM is selected as the reference grid of the three-dimensional face model, the three-dimensional deformed model 3DMM can be established on the basis of the three-dimensional face database, with the face shape and face texture statistics as constraints, and can take into account the human Due to the influence of the posture of the face and illumination factors, the three-dimensional deformation model can be adjusted with higher precision. Figure 7 is a schematic diagram of the 3D deformed model. Compared with the general face model, the reference grid of the 3D face model based on the 3D deformed model has greatly increased the number of triangles and vertices. The 3D deformed model includes More features and details.

Assuming that the three-dimensional deformed model is composed of m face models, each face model includes a corresponding face 3D coordinate shape vector Si, the new 3D face model can be expressed as:

Among them, S _new is a new face shape model,

Represents the average face shape model, _si represents the main component of the new face shape, that is, the feature of the new face that is different from other faces, and α _i is the face shape coefficient.

In an optional implementation manner, on the basis of the three-dimensional deformation model, facial expression data may be further included, so that the three-dimensional deformation model is further expanded to:

Among them, e _i represents the main components of facial expressions, that is, the characteristic parts that are different from other facial expressions, and β _i is the facial expression coefficient. Wherein, the facial shape coefficient α _i and the facial expression coefficient β _i can be solved using the least square method.

Of course, it is also possible to establish a 3D deformed model corresponding to the user based on the collected user image data of the electronic device, and call the corresponding 3D deformed model according to the user currently using the electronic device. If the matching degree between the 3D deformed model called by the current user and the currently acquired image is lower than the predetermined value, the 3D deformed model determined by the user group may be used.

While constructing the reference grid of the three-dimensional face model, the background plane reference grid can be constructed according to the size of the collected image. In the schematic diagram of the reference grid shown in FIG. 8, the reference grid of the three-dimensional face model is overlaid on the background plane reference grid. The constructed background plane reference grid can be constructed according to a grid of a predetermined size and shape. The predetermined shape of the grid may include a triangular grid, a square grid, and the like. The triangular grids in the background plane reference grid can be all the same triangular grids, or can only set the triangular grids of the same shape within a predetermined range around the face, and can be outside the face area, and A denser triangular grid is set in a predetermined range within the head area, so that the face image can be deformed and more accurate and adaptive adjustment can be made to the predetermined range of the head image.

S12 marking boundary points and control points

After constructing the reference grid of the 3D face model and the background plane reference grid, it is necessary to further mark the constructed reference grid, including marking the boundary points and background of the reference grid of the 3D face model. Fixed points and control points of the plane datum grid. The boundary point is a point on the visible boundary of the face of the reference grid of the three-dimensional face model under a predetermined viewing angle. Figure 9 is a schematic diagram of the boundary points marked by the front right perspective of the reference grid. In the front right perspective, there are more visible parts on the right side of the face model, and the distance between the boundary points on the visible boundary and the front center line of the face Far away.

The boundary points may select different numbers of boundary points according to the accuracy of image processing. For example, the higher the accuracy required for image processing, the more boundary points can be selected. The boundary points may be evenly distributed on the intersection line, or the boundary points may be densely distributed in the parts where the contour line of the human face is deformed more frequently. As shown in Figure 9, more boundary points can be set near the mouth of the face.

When the visible angle of view is the front of the reference grid, as shown in FIG. 10, the visible boundary of the face is determined according to the visible area of the face of the reference grid of the three-dimensional face model. According to the visible boundary, Determine the boundary point corresponding to the frontal perspective.

After determining the boundary points of the reference grid of the three-dimensional face model, the control points of the background plane reference grid may also be determined according to the boundary points. As shown in Figure 10, the boundary points determined according to the frontal perspective of the reference grid of the three-dimensional face model and the control points of the background plane reference grid determined according to the boundary points can completely overlap, but of course, it should not be limited to this , The distance between the boundary point and the control point may be set to be less than a predetermined distance.

S13 marking fixed point

In addition, in order to make the background plane reference grid affect the overall size of the image when it is stretched or compressed, this application can also set a fixed point for the background plane reference grid, and the fixed point makes the overall size of the background plane reference grid Will not change, on the other hand, it is also helpful to ensure the overall stability of the picture.

After the control points of the background plane reference grid are determined, the fixed points of the background plane reference grid can be further set. As shown in FIG. 10, the position of the fixed point may be set at the outer boundary of the background plane reference grid, and the fixed point on the outer boundary is used to prevent the image size after the deformation process from changing. Of course, it is also possible to intelligently recognize the target object in the background area of the image, and according to the recognized target object, set a fixed point at the position of the target object in the image corresponding to the background plane reference grid.

The boundary point is located on the reference grid of the three-dimensional face model. When the reference grid of the three-dimensional face model undergoes deformation processing, the position of the boundary point on the reference grid of the three-dimensional face model will also be corresponding The changes include the leftward, rightward, downward or upward movement of the boundary point in the plane direction, and may also include the upper left forward, the upper left rear, and the lower left forward. According to the change of the position of the boundary point, the corresponding control point determines the deformation of the background plane reference grid, and the background plane reference grid may be compressed or stretched according to the Laplace deformation method.

Assuming that the plane where the background plane reference grid is located is the plane determined by the X-axis and Y-axis of the Cartesian coordinate system, after the position of the boundary point changes, the alignment control can be performed according to the changes in the x-coordinate and y-coordinate of the boundary point Changes in the x-coordinate and y-coordinate of the corresponding control point.

For example, the coordinate position of the boundary point before the deformation and the coordinate position after the deformation can be obtained, and the change of the x coordinate and the y coordinate in the coordinate position before and after the change can be extracted, and the boundary can be controlled according to the change of the extracted x coordinate and y coordinate The coordinate change of the control point corresponding to the point.

That is to say, if the control point and the boundary point overlap, the target position of the control point that needs to be deformed can be determined directly according to the x coordinate and the y coordinate in the coordinate position of the boundary point after the change.

When the corresponding relationship between the control point and the boundary point is a predetermined distance apart, the control point corresponding to the boundary point can be determined according to the amount of change in the position of the boundary point in the xoy plane. The amount of change in the position of the point.

For example, the change in the position of the boundary point in the xoy plane can be decomposed into the horizontal change and the vertical change, and the movement of the control point is determined according to the horizontal change and the vertical change, so that It corresponds to the boundary points on the 3D face model modeling grid. Or, the change amount of the boundary point can be decomposed into a moving distance and a moving direction, and the movement of the control point is determined according to the moving distance and the moving direction.

As an optional implementation manner of the present application, the items in the image corresponding to the background plane reference grid can be identified, and the items in the image corresponding to the background plane reference grid can be globally stretched or compressed. That is, through the recognized object in the two-dimensional image corresponding to the background plane reference grid, the equal-proportion adjustment control point of the object is determined. When any part of the object needs to be deformed, the deformation processing method according to the deformed position , Perform the same deformation processing on other control points of the object, so that the shape of the deformed object is still in a normal state, avoiding the defect of background image deformation caused by fusion.

For example, when there is a target object A near the three-dimensional face model, after recognizing the overall outline of the target object A, the control points of the boundary position of the target object A are set. If the control point located at the boundary point is stretched or compressed, so that the part of the target object A is changed, such as the part is stretched or squeezed, it will be deformed according to the scaled control point corresponding to the recognized target object A The processing enables the target object A to achieve overall stretching or compression, so as to avoid affecting the authenticity of the background image when the background plane reference grid is deformed to obtain the background plane modeling grid.

It is worth noting that the construction process of the aforementioned reference grid can be executed at any time before face processing, and the reference grid can be constructed in non-real-time. That is, the construction of the reference grid can be completed before the image is taken, and once constructed, it can be repeatedly applied to the taken photo processing, or repeatedly applied to the image processing of the video frame of the taken video image.

S2, current image modeling:

S21 face model fitting and acquisition of shooting parameters

After the preprocessing of the reference grid is completed, the image can be modeled and used for multiple times according to the pre-built reference grid. For example, after completing a reference grid preprocessing, when the face transformation processing needs to be activated at any time, the pre-built reference grid can be called for image modeling at any time. In addition, the reference grid may be constructed in an offline state, that is, the reference grid may not be constructed in the process of performing face image processing on the image.

When the user uses the camera to take a picture or uses the camera to take a video, if the user receives an instruction to turn on the portrait processing, the captured photo is processed in real time, or the captured video is processed with the face image of the video frame deal with. Combining the pre-built three-dimensional face model reference grid and the background plane reference grid to restore the three-dimensional face image corresponding to the face image in the captured photo or video frame.

Specifically, when modeling the collected two-dimensional image (photographed or video frame in the captured video), a model fitting method can be used to obtain a three-dimensional mesh model corresponding to the two-dimensional image. The face grid may include a three-dimensional face model. That is, the feature points of the face in the image and the feature points in the pre-built reference grid of the three-dimensional face model can be used to fit the model, and the three-dimensional face model modeling can be matched with the feature points of the collected two-dimensional image grid. And the shooting parameters of the two-dimensional image may be determined according to the positions of the feature points of the two-dimensional image, and the shooting parameters may include a model view matrix and a projection matrix.

When fitting the image to the pre-built three-dimensional face model reference grid, the position of the vertex of the three-dimensional face model reference grid can be changed, and the three-dimensional face model reference grid can be deformed, etc. Processing, so that when the feature point positions in the three-dimensional face model modeling grid are mapped to the collected image, they match with the feature point positions in the collected image. That is, the location of the feature points mapped by the 3D face model modeling grid is consistent with the location of the feature points in the collected image.

For example, in the fitting process, the attitude of the reference grid of the three-dimensional face model can be adjusted first, including the adjustment of the yaw angle, pitch angle or roll angle of the reference grid of the three-dimensional face model, and then the attitude The vertices at the detailed image of the adjusted three-dimensional face model reference grid are fitted and deformed, so that the feature points in the collected image are different from those in the image projected and mapped by the adjusted three-dimensional face model modeling grid. The sum of the distances between the corresponding feature points is the smallest, or the feature points in the collected image, and the corresponding feature points in the image projected and mapped by the adjusted three-dimensional face model modeling grid are completely matched.

After fitting the three-dimensional face model modeling grid to the collected image, the corresponding relationship between the two sets of feature points and the three-dimensional face model grid (including the three-dimensional face model reference grid, and According to the changes of the feature points of the three-dimensional face model modeling grid obtained by fitting the three-dimensional face model reference grid before and after fitting, it is determined that the three-dimensional face model reference grid is transformed into the collected image When, the corresponding posture model view (MV, model view) matrix and projection matrix. Among them, the technology of determining the MV matrix and the projection matrix of the collected image from the three-dimensional face model modeling grid according to the corresponding relationship of the feature points and the change of the feature points is a well-known technology, and will not be detailed here. description.

For example, after collecting a two-dimensional image from the upward-looking perspective by a camera, the position of the feature points of the face in the two-dimensional image can be analyzed, combined with the feature points in the preset reference grid of the three-dimensional face model, and the The feature points of the face in the two-dimensional image are fitted to the reference grid of the three-dimensional face model, that is, the positions of the vertices in the reference grid of the three-dimensional face model are adjusted so that the adjusted grid matches The features of the human face in the two-dimensional image.

S22 3D face model deformation processing

Wherein, the MV matrix can extract the rotation component, and rotate the three-dimensional face model modeling grid at the same angle according to the extracted rotation component, so as to transform the pose of the three-dimensional face model modeling grid It is the posture corresponding to the collected image. Wherein, if the MV matrix is:

Then, the rotation component R extracted from the MV matrix is:

Among them, I0, I1, and I2 are the modulus of the vector formed by column 0, column 1, and column 2 of the MV matrix, respectively. And the movement parameters m30, m31 and m32 are not related to the rotation parameters.

According to the MV matrix determined in the fitting process, the posture of the reference grid of the three-dimensional face model can be adjusted. According to the rotation component in the model view matrix, the reference grid of the three-dimensional face model can be rotated and transformed to restore the face pose corresponding to the collected image.

In an optional implementation manner, when the reference grid of the three-dimensional face model is a general face model grid, when the general face model grid is used for reconstruction, it may include the entirety of the face model grid. Sexual adjustment and local adjustment:

The overall adjustment can be adjusted for the contour of the model. The overall layout of the universal face model, including parts such as eyes, ears, nose, mouth, and eyebrows, can be made consistent with the five senses layout of the picture to be restored by including the corresponding feature points.

The local adjustment may be fine-tuned for local details, especially facial features, so that local details are more accurate.

After adjustment, the face can be reconstructed using vertex-based interpolation.

When the three-dimensional face model reference grid is a three-dimensional deformed model grid, when performing face reconstruction on the three-dimensional deformed model grid, the three-dimensional face model reference may be controlled according to the rotation component in the MV matrix The grid is rotated. For example, through the rotation component R in the MV matrix, the reference grid of the three-dimensional face model is adjusted to the pose corresponding to the collected image, which can be further based on the feature points in the five senses in the two-dimensional face model. Position, to determine the changes in facial features of the reference grid of the 3D face model, which can include adjusting facial styles such as eyes, eyebrows, mouth, nose, ears, or face shape, so as to make the adjusted 3D face model grid It more closely matches the face in the captured image.

The modeling mesh of the fitted 3D face model is inconsistent with the pose of the face in the collected 2D image. Including, for example, the human face in the two-dimensional image is from the upward perspective, while the three-dimensional face model modeling grid is the frontal perspective. In order to facilitate rendering to obtain the three-dimensional face model mesh texture map, the three-dimensional face model modeling grid needs to be Rotate so that the rotated 3D face model modeling grid is consistent with the angle of view of the face in the collected 2D image.

S23 background grid deformation

After completing the reconstruction of the three-dimensional face model modeling grid, the position of the boundary points of the rotated three-dimensional face model may change when the three-dimensional face model is transformed, according to the preset Whether the position of the boundary point of the three-dimensional face model is changed to determine whether the position of the control point in the background plane reference grid needs to be adjusted accordingly. When the position of the boundary point of the three-dimensional face model modeling grid changes, the control point corresponding to the boundary point is searched, and the background plane reference network is set according to the change of the x-coordinate and y-coordinate of the boundary point. The control points corresponding to the boundary points in the grid are aligned with the boundary points, thereby completing the deformation processing of the background plane reference grid to obtain a background plane modeling grid.

After determining the control point corresponding to the boundary point after the position change, the deformation of the control point can be determined according to the change of the coordinate of the boundary point before and after the deformation, for example, according to the change of the x coordinate and the y coordinate of the boundary point Amplitude such that the control point is aligned with the boundary point in the x direction and the y direction. According to the corresponding adjustment control points of the alignment operation, the background plane reference grid is deformed, so that the distance between the position of the deformed control point and the position of the corresponding boundary point is less than a predetermined value, or the processed The position of the control point coincides with the position of the corresponding boundary point.

In an alternative embodiment, when the object in the image corresponding to the background plane reference grid is recognized, and the proportional adjustment control point is set according to the recognized object, if any control point in the object needs Adjustment, that is, when any one of the control points moves, the other control points of the object also move correspondingly, and when any one of them is scaled and adjusted, the other proportional control points are correspondingly controlled to scale.

When performing deformation processing on the background plane reference grid through the control points, it can also be combined with the fixed points of the background plane reference grid, so that the image size after the deformation processing does not change. The fixed point may be set around the background plane reference grid, or may be set at the position of some objects.

The deformation processing method of the background plane grid (including the background plane reference grid and the background plane modeling grid) may include Laplace transform. It should not be limited to this, it can also include other mesh deformation processing methods such as bone skin animation algorithm.

Figure 11 is a schematic diagram of the modeling mesh after deformation processing. The modeling grid schematic diagram includes a three-dimensional face model modeling grid and a background plane modeling grid. According to the reference grid shown in Figure 8, after the deformation processing of the model view matrix, and according to the change of the position of the boundary points, the background plane reference grid is deformed by the control points and fixed points to obtain the background plane modeling grid Schematic of the image. It can be seen from Figure 11 that through fitting and combining the model view matrix to deform the reference grid of the three-dimensional face model, including rotating and deforming the reference grid of the three-dimensional face model and/or scaling and deforming, the result is The position of the boundary point of the visible boundary of the three-dimensional face model modeling grid changes. According to the change of the position of the boundary point, the position of the control point of the background plane reference grid is adjusted accordingly, and the background plane reference grid is deformed After processing, the 3D face model modeling grid and the background plane modeling grid obtained after deformation can still be effectively integrated.

S24 texture image rendering

After obtaining the three-dimensional face model modeling grid with the same pose as the collected two-dimensional image, the collected image can be rendered according to the MV matrix and the projection matrix to obtain the three-dimensional face model grid texture map; extract the The translation component and zoom component in the MV matrix form a new matrix, and use the same projection matrix as the three-dimensional face model to render the background plane grid texture map.

Wherein, when acquiring a three-dimensional face model mesh texture map, a rectangular coordinate system OXYZ for marking the vertex positions on the three-dimensional face model modeling grid may be set, and the three-dimensional face model modeling grid The coordinate position of any vertex on can be expressed as (x, y, z). A UV coordinate system is set on the XOY plane in the rectangular coordinate system, and the collected image can be expressed as (u, v) through the UV coordinate system. You can set the z coordinate in the vertex coordinates (x, y, z) of all meshes on the 3D face model modeling grid to 0, set the x, y coordinates to the two-dimensional u, v coordinates of the vertex, and render Obtain the first plane, and then determine the color of each first pixel on the plane: According to the position of the pixel, multiply the MV matrix and the projection matrix to determine that the first pixel corresponds to the second pixel on the collected picture Point, the color of the first pixel on the first plane is taken according to the corresponding second pixel.

Similarly, when acquiring the background plane grid texture map, the background plane modeling grid determines the second plane, and the second plane may correspond to the two-dimensional image to be processed by a scaling ratio. Since the background plane grid is modeled as a plane, there should be no posture changes. Therefore, for the color of each third pixel on the plane, only the translation component and the zoom component in the MV matrix need to be obtained without consideration. The rotation component is multiplied by the projection matrix, and the translation component and zoom component in the MV matrix to determine that the third pixel is in the acquired two-dimensional image to be processed, such as the fourth pixel corresponding to the acquired picture. The corresponding fourth pixel takes a value on the color of the third pixel on the second plane.

For example, when extracting the translation component and the scaling component in the MV matrix, for a given MV matrix:

The translation component T can be extracted as:

The scaling component S is:

Among them, I0, I1, and I2 are the modulus of the vector formed by column 0, column 1, and column 2 of the MV matrix, respectively.

According to the rendered 3D face model mesh texture map and the background plane mesh texture map, it is convenient to render the entire mesh after adjusting the perspective of the 3D face model modeling mesh to obtain a processed image.

For example, for the two-dimensional image of the upward viewing angle shown in Figure 3, after generating the three-dimensional face model modeling grid corresponding to the human face in the two-dimensional image, the projection matrix and the MV matrix are used to determine the rendering plane The corresponding relationship between the pixel value of the pixel and the two-dimensional image results in the three-dimensional face model grid texture map shown in FIG. 12 and the background plane grid texture map shown in FIG. 13.

S3, current image processing

S31 face model deformation

For the restored 3D face model modeling mesh, the 3D face model modeling mesh needs to be deformed according to actual usage.

For example, when the user uses a laptop computer with a hidden camera or a desktop computer with an external camera for video, in order to enhance the video call experience, the posture of the portrait in the video screen is transformed from the posture of looking at the screen to the posture of looking at the camera. It is necessary to transform the posture of the restored 3D face model modeling grid, for example, for a notebook with a hidden camera, adjust the top view angle of the 3D face model, etc., when the camera is set on the left or right side of the desktop computer monitor , Adjusting the yaw angle of the three-dimensional face model modeling grid, that is, rotating the three-dimensional face model to the left or to the right by a certain angle.

Wherein, the required rotation angle of the three-dimensional face model modeling grid can be determined according to the rotation component extracted from the MV matrix obtained during the modeling of the three-dimensional face model reference grid.

For example, the posture angle of a two-dimensional face image collected by a notebook with a hidden camera is 20 degrees upwards horizontally. In order to make the posture of the portrait in the video screen, the posture of looking at the screen is transformed into the posture of looking at the camera. The face model modeling grid is rotated downward by 20 degrees to obtain a front view of the grid as shown in Fig. 14a and a side view of the grid in Fig. 14b. At this time, according to Figure 14b, it can be seen that the position of the boundary points of the three-dimensional face model modeling grid has changed, but the control points corresponding to the boundary points have not undergone corresponding deformation processing, and the background plane modeling grid A gap appears between the deformed three-dimensional face model modeling grid.

When performing deformation processing on the 3D face model modeling grid after restoring the pose corresponding to the human face in the 2D image, a variety of different deformation processing methods can be included according to different application scenarios. For example, it may include face posture deformation processing, face partial deformation processing, and so on.

For facial posture deformation processing, it can be applied to devices where the image capturing camera does not match the center of the screen, including notebooks and desktop computers with hidden camera functions.

For example, when a user uses a laptop with a hidden camera to take photos, make a video call, or record a video, if the user is currently looking at the screen, the captured image will be as shown in Figure 3. The camera is located where the face is looking at. Below the direction, the collected images will show images with a big chin and nose facing upwards that are not what the user expects.

Before the user uses the notebook with the hidden camera, the reference grid can be constructed offline in advance, including the construction of the reference grid of the three-dimensional face model and the background plane reference grid. When marking the constructed reference grid, by identifying the position of the face in the reference grid of the three-dimensional face model in the reference grid, the boundary points of the face are determined, and the boundary points are used to determine the background plane The grid (including the background plane reference grid and the background plane modeling grid) is the control point for transformation, and the fixed point in the background plane grid can be determined according to the magnitude of the image transformation. For example, the fixed point can be set as the edge of the image Place.

When the camera collects the user's face image, it can obtain the feature points of the face image, and fit the collected face image with the reference grid of the 3D face model based on the 3D reconstruction algorithm. The correspondence between the feature points and the changes of the feature points of the 3D face model modeling grid before and after fitting, determine the model view matrix and the projection matrix, and use the fitted 3D face model reference grid according to the model view matrix Rotate the rotation component in the image to obtain the face pose model corresponding to the captured image, and determine the control points that need to be adjusted according to the changes in the position of the boundary points of the three-dimensional face model modeling grid obtained after the rotation. The control points deform the background plane reference grid to fuse the background plane modeling grid with the 3D face model modeling grid, and determine the acquisition according to the MV matrix including the image shooting parameters and the projection matrix including the imaging parameters 3D face model grid texture map and background plane grid texture map corresponding to the image.

According to the user's requirements, for example, according to the size of the shooting angle of view, the face pose is transformed so that the adjusted angle of view is adjusted downward to a certain angle, such as 20 degrees downward, and the adjusted 3D model grid is rendered , The adjusted image effect as shown in Figure 15 is obtained, and the user's front face image is obtained. In this way, it is possible to obtain an image that the user is looking at in a video chat, and to control the background deformation through the corresponding relationship between the control point and the boundary point, which is beneficial to improve the integration efficiency of the background image.

When the user uses a desktop computer with an external camera, or uses an ordinary notebook to take pictures, make a video call, or record a video, if the user is currently looking at the screen, the captured image may capture the user’s profile image or capture The user's overhead image is not conducive to improving the user's call experience.

When the reference grid of the 3D face model is rotated according to the collected images, the reference grid of the 3D face model is rotated in the opposite direction when the camera is set on the left side of the display according to the recognized face posture , To restore the pose of the three-dimensional face model corresponding to the two-dimensional face image. When the camera is set on the right side of the display, the reference grid of the three-dimensional face model is rotated in the opposite direction to restore the posture of the three-dimensional face image corresponding to the two-dimensional face image, and complete the construction of the currently collected image. mold.

When performing deformation processing on the currently modeled face model, according to the change of the position of the boundary point of the three-dimensional face model modeling grid, deform the background plane reference grid to obtain the background plane modeling grid , So that the background plane modeling grid and the background plane modeling grid are effectively integrated. And according to the MV matrix and the projection matrix obtained during fitting, the three-dimensional face model grid texture map and the background plane grid texture map are obtained. According to the perspective in the two-dimensional plane image, the modeled face model is deformed, including rotating left or right, etc., and the rotated background plane reference grid is deformed, and the obtained three-dimensional The face model grid texture map and the background plane grid texture map are used to render the entire deformed grid to obtain a processed two-dimensional image, and the gaze direction and camera position of the face in the processed two-dimensional image Matching enables users to get a better attention experience during video calls. In addition, the time required for the conversion process is relatively short, which helps to improve the real-time performance of video image display and to improve the user experience.

When applied to the local deformation processing of human face, after the user uses any smart device to collect images, including images such as photos or videos, the user may need to send the collected images to other users, or send the collected images to social media In order to improve the satisfaction of users, the service platform needs to make partial adjustments to the collected images, such as thinning the face and thinning the chin.

In this application scenario, after the image collected by the camera is obtained, the face in the collected image can be fitted with the reference grid of the three-dimensional face model based on the preset reference grid model and restored to the image The pose corresponding to the face is determined by the MV matrix and projection matrix according to the fitting process, and the fusion transformation is performed according to the boundary points in the 3D face model modeling grid and the control points in the background plane reference grid. According to the MV matrix and The projection matrix obtains the three-dimensional face model grid texture map and the background plane grid texture map. According to the preset face transformation requirements, for example, the three-dimensional face model can be modeled according to preset face beautification parameters The mesh is deformed. The face beautification parameters may include one or more of eye size parameters, eye spacing parameters, face fatness and thinness parameters, mouth size parameters, eye bag removal parameters, face shape parameters, and nose wing size parameters. The modeling grid of the three-dimensional face model is further deformed and adjusted according to the face beautification parameters. After the three-dimensional face model modeling grid is adjusted, the background plane modeling grid is further transformed and fused, and a transformed image is obtained through rendering, for example, an image with a thin face or a thin chin is obtained after transformation.

S32 background grid deformation

According to the control points corresponding to the boundary points of the three-dimensional face model modeling grid, combined with the fixed points in the background plane modeling grid, transform the background plane modeling grid, such as performing Laplace Deformation, so that the front view of the mesh after the background plane modeling mesh is deformed as shown in FIG. 16a and the side view of the mesh after the deformation is completed as shown in FIG. 16b. Through the rotation vector in the MV matrix, the three-dimensional face model modeling grid that is consistent with the perspective of the face in the acquired two-dimensional image is controlled to rotate in the opposite direction, so that the rotated three-dimensional face model The modeling grid is transferred to a frontal perspective, which is convenient for generating a frontal two-dimensional image based on the three-dimensional face model modeling grid of the frontal perspective.

S33 render to image

According to the 3D face model mesh texture map and background plane mesh texture map generated by the rendering, the transformed 3D face model modeling grid and background plane modeling grid are rendered to obtain the rendered 3D face The image and the background plane image are projected on the three-dimensional face image to obtain a processed two-dimensional image.

When the user is a user of a smart device, in order to obtain a better captured image, it is necessary to perform face deformation processing on the photo or video frame in the video captured by the camera. The method for deforming the face includes deforming the face such as thinning the face or thinning the chin.

The 3D face model restored from the currently collected image can be modeled according to the way of pre-setting the standard grid of the 3D face model (3D face model grid for beautification), or preset beautification parameters The grid is transformed, and control points and fixed points can be set on the 3D face model modeling grid, and the 3D face model modeling grid can be controlled to deform according to the standard 3D face model or beautification parameters After processing, the deformed three-dimensional face model modeling mesh is obtained.

The position of the boundary point on the 3D face model modeling grid obtained after transformation is detected. If the position of the boundary point on the 3D face model modeling grid obtained after transformation is relative to the 3D face before transformation If the position of the boundary point of the model reference grid is changed, the control point of the corresponding background plane modeling grid can be determined according to the transformed boundary point, and the background plane modeling grid can be transformed to obtain the transformation The resulting background plane modeling grid. Or when transforming the background plane modeling grid, the fixed points of the background plane modeling grid can also be set, and the retention properties of the background plane modeling grid can be controlled by the fixed points. For example, the fixed point may be set around the background plane modeling grid.

The 3D face model modeling mesh and background plane modeling mesh obtained by the transformation can be rendered according to the texture characteristics of the 3D mesh model before the transformation, or combined with the current light, to obtain the rendered mesh model , Combined with the current posture angle to project the rendered grid model, the processed image can be obtained.

Since this application sets the boundary points of the face and the control points in the background while constructing the reference grid, the background is quickly deformed through the correspondence between the background points and the control points, and the same configuration is used. Next, while generating images with no gap effect, it can also significantly improve image processing efficiency.

For example, when the device configuration information used is: CPU is i7-8550, memory is 16GB, and the resolution of the image captured by the camera is 1280*720, the face image processing method described in this Set the reference grid, reconstruct the reference grid of the 3D face model, obtain the 3D face model modeling grid after reconstruction, and get the 3D face model mesh texture map and background plane grid texture map. This process It usually takes 25-35ms to deform the reconstructed 3D face model modeling mesh and background plane modeling mesh. This process usually takes 2-3ms. Render the deformed image to get The process of the face deformed image is usually 4-6ms. Therefore, the entire process takes about 31-45ms, while the existing method of using the target image for two-dimensional transformation requires 130-170ms. Through the three-dimensional deformation+ The background fusion mode requires 65-95 ms. The face image processing method described in the embodiment of the present application ensures a good fusion of the face image and the background image while effectively reducing the image processing time.

It should be understood that the steps in the foregoing embodiments do not mean the order of execution, and the execution order of the processes should be determined by their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

Corresponding to the face image processing method described in the above embodiment, FIG. 17 shows a structural block diagram of a face image processing device provided by an embodiment of the present application. For ease of description, only the same as the embodiment of the present application is shown. The relevant part.

FIG. 17 is a schematic structural diagram of a face image processing device provided by an embodiment of the present application, and the face image processing device includes:

The image modeling unit 1701 is configured to obtain a two-dimensional image to be processed by an electronic device, construct a three-dimensional grid model corresponding to the two-dimensional image to be processed according to a preset reference grid, and according to the two-dimensional image to be processed Acquiring the texture map of the three-dimensional grid model by taking the shooting parameters of the three-dimensional image, and determining the boundary point according to the visible boundary of the face of the reference grid, and the control point corresponding to the boundary point;

The model deformation unit 1702 is configured to perform deformation processing on the three-dimensional mesh model by the electronic device according to preset deformation requirements in combination with the corresponding relationship between the boundary points and the control points, and render the texture image to deform The processed 3D mesh model generates a processed image according to the rendered 3D mesh model.

The face image processing device described in FIG. 17 corresponds to the aforementioned face processing method.

Those skilled in the art can clearly understand that for the convenience and conciseness of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned functions can be allocated to different functional units and modules as required. Module completion, that is, divide the internal structure of the device into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist alone physically, or two or more units can be integrated into one unit. The above-mentioned integrated units can be hardware-based Formal realization can also be realized in the form of software functional units. In addition, the specific names of the functional units and modules are only used to facilitate distinguishing each other, and are not used to limit the protection scope of the present application. For the specific working process of the units and modules in the foregoing system, reference may be made to the corresponding process in the foregoing method embodiment, which is not repeated here.

In the foregoing embodiments, the description of each embodiment has its own focus. For parts that are not detailed or recorded in a certain embodiment, reference may be made to related descriptions of other embodiments.

The face image processing method provided in the embodiments of this application can be applied to notebooks, desktop computers, tablet computers, mobile phones, wearable devices, vehicle-mounted devices, augmented reality (AR)/virtual reality (VR) devices, Ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (personal digital assistants, PDAs) and other electronic devices with cameras, the embodiments of this application do not impose any restrictions on the specific types of electronic devices.

FIG. 18 shows a block diagram of a part of the structure of an electronic device 1800 provided in an embodiment of the present application. Referring to FIG. 18, the electronic device 1800 includes: a memory 1810, a camera 1820, a display unit 1830, a power supply 140, a processor 1850 and other components. Those skilled in the art can understand that the structure of the electronic device 1800 shown in FIG. 18 does not constitute a limitation on the electronic device 1800, and may include more or fewer components than shown, or a combination of certain components, or different components Layout.

The components of the electronic device 100 are specifically introduced below in conjunction with FIG. 18:

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2. , Mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber identification module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include pressure sensor 180A, gyroscope sensor 180B, air pressure sensor 180C, magnetic sensor 180D, acceleration sensor 180E, distance sensor 180F, proximity light sensor 180G, fingerprint sensor 180H, temperature sensor 180J, touch sensor 180K, ambient light Sensor 180L, bone conduction sensor 180M, etc.

It can be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange different components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU), etc. Among them, the different processing units may be independent devices or integrated in one or more processors.

The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

A memory may also be provided in the processor 110 to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.

In some embodiments, the processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, and a universal asynchronous transmitter (universal asynchronous transmitter) interface. receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and / Or Universal Serial Bus (USB) interface, etc.

The I2C interface is a two-way synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple sets of I2C buses. The processor 110 may be coupled to the touch sensor 180K, charger, flash, camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may couple the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through an I2C bus interface to implement the touch function of the electronic device 100.

The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Then the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is processed by the baseband processor and then passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

The electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode). AMOLED, flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, and N is a positive integer greater than one.

The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

The ISP is used to process the data fed back from the camera 193. For example, when taking a picture, the shutter is opened, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transfers the electrical signal to the ISP for processing and is converted into an image visible to the naked eye. ISP can also optimize the image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193.

The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. ISP outputs digital image signals to DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats. In some embodiments, the electronic device 100 may include 1 or N cameras 193, and N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects the frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in a variety of encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, and so on.

NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, for example, the transfer mode between human brain neurons, it can quickly process input information and can continuously learn by itself. The NPU can realize applications such as intelligent cognition of the electronic device 100, such as image recognition, face recognition, voice recognition, text understanding, and so on.

The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the present invention takes an Android system with a layered architecture as an example to exemplify the software structure of the electronic device 100.

FIG. 2 is a software structure block diagram of an electronic device 100 according to an embodiment of the present invention.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Communication between layers through software interface. In some embodiments, the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

Android Runtime includes core libraries and virtual machines. Android runtime is responsible for the scheduling and management of the Android system.

The core library consists of two parts: one part is the function functions that the java language needs to call, and the other part is the core library of Android.

The application layer and the application framework layer run in a virtual machine. The virtual machine executes the java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

The system library can include multiple functional modules. For example: surface manager (surface manager), media library (Media Libraries), three-dimensional graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc.

The surface manager is used to manage the display subsystem and provides a combination of 2D and 3D layers for multiple applications.

The media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to realize 3D graphics drawing, image rendering, synthesis, and layer processing.

The 2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

The application layer is used to run installed applications or applications in the system, including, for example, camera, calendar, map, WLAN, music, short message, gallery, call, navigation, Bluetooth, and video.

In the following, the workflow of the software and hardware of the electronic device 100 will be exemplified in conjunction with capturing a photo scene.

When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes touch operations into original input events (including touch coordinates, time stamps of touch operations, etc.). The original input events are stored in the kernel layer. The application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon as an example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer. The camera 193 captures still images or videos.

Finally, it should be noted that the above are only specific implementations of this application, but the scope of protection of this application is not limited to this. Any changes or substitutions within the technical scope disclosed in this application should be covered by this application. Within the scope of protection applied for. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

A face image processing method, characterized in that the face image processing method includes:

The electronic device obtains the two-dimensional image to be processed, constructs a three-dimensional grid model corresponding to the two-dimensional image to be processed according to a preset reference grid, and obtains the three-dimensional image according to the shooting parameters of the two-dimensional image to be processed. The texture map of the grid model, determining the boundary points according to the visible boundary of the face of the reference grid, and the control points corresponding to the boundary points;

The electronic device performs deformation processing on the three-dimensional mesh model according to preset deformation requirements and in combination with the corresponding relationship between the boundary points and the control points, and renders the texture image to the deformed three-dimensional mesh model, Generate a processed image according to the rendered 3D mesh model.
The face image processing method according to claim 1, wherein the reference grid includes a three-dimensional face model reference grid and a background plane reference grid, and the three-dimensional grid model includes a three-dimensional face model modeling Grid and background plane modeling grid. The step of constructing a three-dimensional grid model corresponding to the two-dimensional image to be processed by the electronic device according to a preset reference grid includes:

The electronic device fits the reference grid of the three-dimensional face model with the two-dimensional image to be processed, and obtains the shooting parameters of the two-dimensional image to be processed according to the fitted reference grid of the three-dimensional face model;

According to the shooting parameters, the electronic device adjusts the posture of the reference grid of the three-dimensional face model to obtain a three-dimensional face model modeling grid. The three-dimensional face model modeling grid and the two-dimensional image The faces are consistent.
The face image processing method according to claim 2, wherein the shooting parameters include a model view matrix and a projection matrix, and according to the shooting parameters, the electronic device compares the three-dimensional face model reference net The steps to adjust the posture of the grid include:

According to the model view matrix, the electronic device extracts the rotation component;

According to the extracted rotation component, the electronic device controls the fitted three-dimensional face model reference grid to rotate to the face pose corresponding to the two-dimensional image to be processed.
The face image processing method according to claim 2, wherein the step of the electronic device constructing a three-dimensional grid model corresponding to the two-dimensional image to be processed according to a preset reference grid further comprises:

Determine the boundary point where the position changes in the 3D face model modeling grid after the pose adjustment;

The corresponding control point is searched for according to the boundary point where the position has changed, and the background plane reference grid is deformed and controlled according to the searched control point.
The face image processing method according to claim 2, wherein the texture map comprises a three-dimensional face model grid texture map and a background plane grid texture map, and the electronic device is based on the two-dimensional face model to be processed. The step of obtaining the texture map of the three-dimensional mesh model by the shooting parameters of the image includes:

According to the model view matrix and the projection matrix, the electronic device obtains the three-dimensional face model grid texture map;

According to the projection matrix, and the translation vector and the zoom vector in the model view matrix, the electronic device obtains the background plane grid texture map.
The face image processing method according to claim 5, wherein the step of obtaining the three-dimensional face model grid texture map by the electronic device according to the model view matrix and the projection matrix comprises:

The electronic device obtains the coordinates of the vertices in the three-dimensional face model modeling grid in the spatial rectangular coordinate system, and renders the first plane when the z coordinate in the vertices coordinates is 0;

The electronic device determines the second pixel corresponding to the first pixel on the two-dimensional image to be processed according to the product of the position of the first pixel on the first plane, the model view matrix and the projection matrix, and according to the second pixel The color of the dot determines the color of the first pixel.
The face image processing method according to claim 5, wherein the electronic device obtains the background plane grid texture map according to the projection matrix, and the translation vector and the zoom vector in the model view matrix The steps include:

The electronic device determines the second plane according to the background plane modeling grid, and extracts the translation matrix and the zoom matrix in the model view matrix;

The electronic device determines the fourth pixel corresponding to the third pixel on the two-dimensional image to be processed according to the position of each third pixel in the second plane and the product of the translation matrix, the zoom matrix and the projection matrix, The color of the third pixel is determined according to the color of the fourth pixel.
7. The face image processing method according to any one of claims 1-7, wherein the step of deforming the three-dimensional mesh model by the electronic device comprises:

Acquiring, by the electronic device, the posture of the three-dimensional face model modeling grid in the constructed three-dimensional grid model;

The electronic device rotates the three-dimensional face model modeling grid according to the angle relationship between the posture of the constructed three-dimensional face model modeling grid and the target posture.
7. The face image processing method according to any one of claims 1-7, wherein the step of deforming the three-dimensional face model modeling grid by the electronic device according to preset deformation requirements comprises :

The electronic device acquires preset face beautification parameters;

According to the face beautification parameter, the electronic device adjusts the three-dimensional face model modeling mesh in the three-dimensional mesh model.
The face image processing method according to claim 9, wherein the face beautification parameters include eye size parameters, eye spacing parameters, face fatness and thinness parameters, mouth size parameters, eye bag removal parameters, face shape parameters, and nose wings One or more of the size parameters.
The face image processing method according to any one of claims 1-7, wherein the electronic device combines the corresponding relationship between the boundary points and the control points to deform the three-dimensional mesh model include:

The electronic device acquires the first position where the boundary point is located on the reference grid of the three-dimensional face model in the reference grid, and the boundary point is located on the three-dimensional face model modeling grid in the three-dimensional grid model. Second position

When the distance between the second position and the first position is greater than a predetermined value, the electronic device searches for the control point corresponding to the boundary point;

The electronic device performs deformation processing on the background plane modeling grid according to the searched control points.
The face image processing method according to claim 11, wherein the step of deforming the background plane modeling grid by the electronic device according to the searched control point comprises:

Acquiring, by the electronic device, the coordinate variation of the coordinate position of the boundary point on the background plane;

The electronic device determines the target position of the control point according to the coordinate change amount of the coordinate position of the boundary point on the background plane;

According to the target position, the electronic device performs Laplace deformation processing on the background plane modeling grid.
The face image processing method according to claim 12, wherein the step of performing Laplace deformation processing on the background plane modeling grid by the electronic device according to the target position comprises:

Acquiring, by the electronic device, control points set on the background plane modeling grid;

According to the set control points and the target positions of the control points, the electronic device performs Laplace deformation processing on the background plane modeling grid.
The face image processing method according to claim 1, wherein the three-dimensional face model reference grid in the reference grid is a general face model or a three-dimensional deformed model.
The face image processing method according to claim 1, wherein before the step of obtaining the two-dimensional image to be processed by the electronic device, the method further comprises:

The electronic device constructs a three-dimensional face model reference grid and a background plane reference grid;

Acquiring, by the electronic device, a face area in a reference grid of a three-dimensional face model;

According to the visible boundary of the face area, the electronic device determines the position of the boundary point and the control point.
The face image processing method according to claim 1, wherein the step of obtaining the two-dimensional image to be processed by the electronic device comprises:

The electronic device extracts video frames in real time from the video collected by the camera, and uses the extracted video frames as a two-dimensional image to be processed;

Or, the electronic device uses the photo taken by the camera as the two-dimensional image to be processed.
An electronic device, wherein the electronic device includes a memory, a processing screen, and a computer program, the display screen is used for processed images, the computer program is stored in the memory, and the computer program includes instructions When the instruction is executed by the electronic device, the electronic device is caused to execute the face image processing method according to any one of claims 1 to 16.
A computer storage medium storing a computer program, wherein the computer program is executed by a processor to implement the face image processing method according to any one of claims 1 to 16.
A computer program product containing instructions, wherein when the computer program product runs on an electronic device, the electronic device executes the face image processing method according to any one of claims 1-16.