CN112348937A

CN112348937A - Face image processing method and electronic equipment

Info

Publication number: CN112348937A
Application number: CN201910735963.9A
Authority: CN
Inventors: 丁欣; 王利强; 周恒�
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2019-08-09
Filing date: 2019-08-09
Publication date: 2021-02-09
Also published as: WO2021027585A1

Abstract

The application discloses a face image processing method, which comprises the following steps: the method comprises the steps that electronic equipment obtains a two-dimensional image to be processed, a three-dimensional grid model corresponding to the two-dimensional image to be processed is built according to a preset reference grid, a texture map of the three-dimensional grid model is obtained according to shooting parameters of the two-dimensional image to be processed, boundary points and control points corresponding to the boundary points are determined according to visible boundaries of the face of the reference grid; and the electronic equipment performs deformation processing on the three-dimensional grid model according to a preset deformation requirement by combining the corresponding relation between the boundary point and the control point, renders the texture image to the three-dimensional grid model after the deformation processing, and generates a processed image according to the rendered three-dimensional grid model. The three-dimensional grid model is rapidly adjusted through the control points and the boundary points of the three-dimensional grid model, so that the image processing efficiency is improved, and high-quality real-time image processing is facilitated.

Description

Face image processing method and electronic equipment

Technical Field

The present application belongs to the field of image processing, and in particular, to a face image processing method and an electronic device.

Background

With the development of the camera technology, the camera function is more and more widely applied to the smart device, such as a smart phone, a tablet computer, a notebook computer, and the like. Can convenience of customers carries out image acquisition through the function of making a video recording, including shooting photo or video etc for smart machine more receives user's favor.

When the camera shooting function of the intelligent equipment is used, the camera shooting device of the intelligent equipment is generally arranged in an area outside a screen, so that when a user watches the screen, the face posture in an image collected by the camera shooting device can incline, and the face posture in the image needs to be corrected. The deformation fusion method based on the 3D face reconstruction adopted at present needs to consume a long time when generating a corrected 2D face and fusing the face with a background, and particularly cannot meet the requirement of real-time processing when using a camera shooting function to carry out video call.

Disclosure of Invention

The application provides a face image processing method and electronic equipment, which are used for solving the problems that in the prior art, when the face posture in an image is corrected, long time needs to be consumed, and the real-time requirement cannot be met.

In order to achieve the purpose, the technical scheme is as follows:

in a first aspect, an embodiment of the present application provides a face image processing method, where the face image processing method includes: the method comprises the steps that electronic equipment obtains a two-dimensional image to be processed, a three-dimensional grid model corresponding to the two-dimensional image to be processed is built according to a preset reference grid, a texture map of the three-dimensional grid model is obtained according to shooting parameters of the two-dimensional image to be processed, boundary points and control points corresponding to the boundary points are determined according to visible boundaries of the face of the reference grid; and the electronic equipment performs deformation processing on the three-dimensional grid model according to a preset deformation requirement by combining the corresponding relation between the boundary point and the control point, renders the texture image to the three-dimensional grid model after the deformation processing, and generates a processed image according to the rendered three-dimensional grid model.

The method comprises the steps of acquiring shooting parameters according to an acquired two-dimensional image to be processed by the electronic equipment, constructing a three-dimensional grid model according to the two-dimensional image and a reference grid, determining boundary points and control points according to the reference grid, and performing deformation processing on the three-dimensional grid model according to a preset deformation requirement and in combination with the boundary points and the control points, so that the deformation processing efficiency of the three-dimensional grid model can be effectively improved, and the processed three-dimensional grid model is subjected to texture image rendering, so that the image processing efficiency of the electronic equipment can be improved, and the timeliness requirement of the electronic equipment on real-time image processing can be better met. For example, the method can be applied to real-time processing of video images, delay of the processed images is reduced, and user experience is improved.

Possibly, the electronic equipment extracts a video frame in real time according to the video collected by the camera, and the extracted video frame is used as a two-dimensional image to be processed; or the electronic equipment takes the picture shot by the camera as a two-dimensional image to be processed. After a two-dimensional image to be processed is obtained, whether the two-dimensional image comprises a face image or not can be detected, if the two-dimensional image comprises the face image, the face image processing method is started, and a preset reference grid is called to construct a three-dimensional grid model. If no face image is detected in the two-dimensional image, the acquired image can be directly displayed.

In a possible implementation manner, the reference mesh includes a three-dimensional face model reference mesh and a background plane reference mesh, the three-dimensional mesh model includes a three-dimensional face model modeling mesh and a background plane modeling mesh, and the step of the electronic device constructing the three-dimensional mesh model corresponding to the two-dimensional image to be processed according to the preset reference mesh includes: the electronic equipment fits the three-dimensional face model reference grid with the two-dimensional image to be processed, and acquires shooting parameters of the two-dimensional image to be processed according to the fitted three-dimensional face model reference grid; and according to the shooting parameters, the electronic equipment performs posture adjustment on the three-dimensional human face model reference grid to obtain a three-dimensional human face model modeling grid, wherein the three-dimensional human face model modeling grid is consistent with the human face posture in the two-dimensional image.

Illustratively, the three-dimensional face model reference mesh in the reference mesh is a general face model or a three-dimensional deformation model. And fitting the reference grid through the characteristics of the face image in the two-dimensional image to be processed, so that the reference grid including the face characteristics is obtained after the reference grid is transformed. And then, further combining the posture characteristics in the two-dimensional image, the reference grid can be rotated for the rotation vector in the shooting parameters in the application, so that the three-dimensional face model can be effectively matched with the face posture in the two-dimensional image.

In a possible implementation manner, the shooting parameters include a model view matrix and a projection matrix, and the step of performing, by the electronic device, pose adjustment on the three-dimensional face model reference grid according to the shooting parameters includes: according to the model view matrix, the electronic equipment extracts a rotation component; and according to the extracted rotation component, the electronic equipment controls the fitted three-dimensional human face model reference grid to rotate to the human face posture corresponding to the two-dimensional image to be processed.

Of course, when the pose of the three-dimensional face model reference grid is adjusted, the pose is not limited to this, and the rotation component of the three-dimensional face model reference grid may be determined in a feature point combination manner or a feature comparison manner, and the fitted three-dimensional face model reference grid is rotated.

In a possible implementation manner, the step of the electronic device constructing the three-dimensional mesh model corresponding to the two-dimensional image to be processed according to the preset reference mesh further includes: determining boundary points of position change in the three-dimensional face model modeling grid after posture adjustment; and searching a corresponding control point according to the boundary point with the changed position, and performing deformation control on the background plane reference grid according to the searched control point.

It can be seen that by setting the background plane modeling grid and the three-dimensional face model modeling grid, after the three-dimensional face model modeling grid is adjusted, the adaptive adjustment can be rapidly performed on the background plane modeling grid according to the change of the boundary points, and the improvement of the response speed of the adjustment is facilitated. The three-dimensional face model modeling grid is obtained by fitting a three-dimensional face model reference grid with a face image in a two-dimensional image, and the background plane modeling grid can be obtained by adjusting the background plane reference grid according to the relation between boundary points and control points when fitting or posture adjustment is carried out on the three-dimensional face model reference grid.

In a possible implementation manner, the texture map includes a three-dimensional face model mesh texture map and a background plane mesh texture map, and the step of acquiring, by the electronic device, the texture map of the three-dimensional mesh model according to the shooting parameters of the two-dimensional image to be processed includes: according to the model view matrix and the projection matrix, the electronic equipment acquires a three-dimensional face model grid texture map; and according to the projection matrix and the translation vector and the scaling vector in the model view matrix, the electronic equipment acquires the background plane grid texture map.

The step of acquiring, by the electronic device, the mesh texture map of the three-dimensional face model according to the model view matrix and the projection matrix may include: the electronic equipment acquires coordinates of vertexes in the three-dimensional face model modeling grid in a space rectangular coordinate system, and renders the vertexes to obtain a first plane when a z coordinate in the vertex coordinates is 0; and the electronic equipment determines a second pixel point corresponding to the first pixel point on the two-dimensional image to be processed according to the product of the position of the first pixel point of the first plane and the model view matrix and the projection matrix, and determines the color of the first pixel point according to the color of the second pixel point.

Wherein, the step of acquiring, by the electronic device, the background plane mesh texture map according to the projection matrix and the translation vector and the scaling vector in the model view matrix may include: the electronic equipment determines a second plane according to the background plane modeling grid, and extracts a translation matrix and a scaling matrix in the model view matrix; and the electronic equipment determines a fourth pixel point corresponding to each third pixel point on the two-dimensional image to be processed according to the product of the position of each third pixel point on the second plane and the translation matrix, the scaling matrix and the projection matrix, and determines the color of the third pixel point according to the color of the fourth pixel point.

It can be seen that, through the model view matrix and the projection matrix, the corresponding relationship between the pixel points in the two-dimensional image and the texture map (including the three-dimensional face model mesh texture map and the background plane mesh texture map) can be determined, and the background plane mesh texture map does not need to be rotated, and can be obtained through the scaling vector and the translation vector in the model view matrix. By generating the texture map of the three-dimensional grid model, the image of the three-dimensional grid model can be generated quickly after the three-dimensional grid model is transformed. Without being limited to this, the texture map of the three-dimensional mesh model may be generated by means of feature point matching or the like.

In one possible implementation manner, the step of the electronic device performing deformation processing on the three-dimensional mesh model includes: the electronic equipment acquires the posture of a three-dimensional face model modeling grid in the constructed three-dimensional grid model; and the electronic equipment rotates the three-dimensional face model modeling grid according to the angle relation between the posture of the three-dimensional face model modeling grid and the target posture.

Illustratively, when the angle of the user image acquired by the electronic device is an elevation angle and the obtained two-dimensional image presents features of double chin, skynes and the like, and the two-dimensional image is adjusted, the determined target posture is the user image corresponding to the horizontal angle. The angle relation between the posture of the three-dimensional human face model modeling grid and the target posture is the included angle between the elevation angle and the horizontal line of the image collected by the electronic equipment. And rotating the three-dimensional face model modeling grid downwards according to the included angle, so that the posture horizontal plane of the three-dimensional face model modeling grid is forward.

In a possible implementation manner, the step of, by the electronic device, performing deformation processing on the three-dimensional face model modeling grid according to a preset deformation requirement includes: the electronic equipment acquires preset face beautifying parameters; and according to the face beautification parameters, the electronic equipment adjusts the three-dimensional face model modeling grid in the three-dimensional grid model.

In a possible implementation, the face beautification parameters include one or more of an eye size parameter, an eye distance parameter, a face fat-thin parameter, a mouth size parameter, an eye pocket removal parameter, a face shape parameter, and a wing of nose size parameter.

For example, when the electronic device receives a beautification request of a user, one or more of preset eye size parameters, eye distance parameters, face weight parameters, mouth size parameters, pouch removal parameters, face shape parameters and nose size parameters may be selected as face beautification parameters corresponding to the current beautification request according to the beautification request. And adjusting the three-dimensional face model modeling grid according to the face beautification parameters. During adjustment, the distance of the characteristic points in the three-dimensional face model modeling grid can be adjusted according to a certain proportion. For example, when the face beautification parameters include face parameters, two or more groups of feature point pairs for characterizing the face width may be selected, and the distances of the feature point pairs conform to a predetermined ratio relationship. And adjusting the ratio relation of the distances of the corresponding feature point pairs in the current three-dimensional face model modeling grid according to the preset ratio relation of the distances of the feature point pairs to enable the ratio relation to be consistent with the preset ratio relation, so that the beautification of the face is realized, for example, the width of the chin relative to the face is adjusted, and the melon seed face shape and the like are obtained after the beautification.

In one implementation, the step of performing, by the electronic device, deformation processing on the three-dimensional mesh model in combination with the correspondence between the boundary points and the control points includes: the electronic equipment acquires a first position of a boundary point on a three-dimensional face model reference grid in the reference grid and a second position of the boundary point on a three-dimensional face model modeling grid in the three-dimensional grid model; when the distance between the second position and the first position is larger than a preset value, the electronic equipment searches for a control point corresponding to the boundary point; and the electronic equipment carries out deformation processing on the background plane modeling grid according to the searched control point.

In a possible implementation manner, the step of the electronic device performing deformation processing on the background plane modeling grid according to the searched control point includes: the electronic equipment acquires the coordinate variation of the coordinate position of the boundary point on a background plane; according to the coordinate variation of the coordinate position of the boundary point on a background plane, the electronic equipment determines the target position of the control point; and according to the target position, the electronic equipment carries out Laplace deformation processing on the background plane modeling grid.

It can be seen that when the three-dimensional face model reference grid is subjected to deformation processing, the processing modes comprise fitting or posture adjustment and the like, the background plane reference grid can rapidly determine the target transformation position according to the corresponding relation between the boundary point and the control point, so that the adjustment of the background plane reference grid is rapidly completed, and the response speed of image processing is improved.

When the corresponding control point is determined according to the coordinate position of the boundary point, the vertical projection of the boundary point on the background plane can be obtained, or the corresponding control can be obtained by intercepting the two-dimensional coordinate of the three-dimensional coordinate of the boundary point.

In a possible implementation manner, according to the target position, the step of performing, by the electronic device, laplacian deformation processing on the background plane modeling grid includes: the electronic equipment acquires control points arranged on the background plane modeling grid; and according to the set control points and the target positions of the control points, the electronic equipment performs Laplace deformation processing on the background plane modeling grid.

The method has the advantages that the target positions of the control points in the background plane modeling grid are subjected to Laplace deformation processing, so that the background plane modeling grid and the three-dimensional face model modeling grid can be effectively fused, the authenticity of the transformed image is improved, and the image gap is avoided.

In one possible implementation, before the step of acquiring, by the electronic device, a two-dimensional image to be processed, the method further includes: the electronic equipment constructs a three-dimensional face model reference grid and a background plane reference grid; the electronic equipment acquires a face area in a three-dimensional face model reference grid; according to the visible boundary of the face area, the electronic equipment determines the positions of boundary points and control points.

In a second aspect, the present application provides an electronic device comprising a memory, a processing screen and a computer program, the display screen being for a processed image, the computer program being stored in the memory, the computer program comprising instructions that, when executed by the electronic device, cause the electronic device to perform the method of processing a face image according to any one of the first aspect.

In a third aspect, the present application provides a computer-readable storage medium storing a computer program, which when executed by a processor implements the face image processing method according to any one of the first aspect.

In a fourth aspect, the present application provides a computer program product containing instructions, which when run on an electronic device, causes the electronic device to execute the face image processing method according to any one of the first aspect.

It is to be understood that the electronic device according to the second aspect, the computer storage medium according to the third aspect, and the computer program product according to the fourth aspect are all configured to execute the corresponding method provided above, and therefore, the beneficial effects achieved by the electronic device can refer to the beneficial effects in the corresponding method provided above, and are not described herein again.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.

Fig. 1 is a schematic structural diagram of a hidden camera provided in an embodiment of the present application;

fig. 2 is a schematic diagram illustrating a video call using a hidden camera according to an embodiment of the present disclosure;

fig. 3 is a schematic diagram illustrating an effect of an image captured by a hidden camera according to an embodiment of the present disclosure;

fig. 4 is a schematic diagram of a comparison effect of processing a face image based on a target image matching method according to an embodiment of the present application;

fig. 5 is a schematic flowchart of a face image processing method according to an embodiment of the present application;

fig. 6 is a schematic diagram of a general face model according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a three-dimensional deformation model provided in an embodiment of the present application;

FIG. 8 is a schematic diagram of a reference grid provided in an embodiment of the present application;

fig. 9 is a schematic diagram of a reference grid right front view marker boundary point according to an embodiment of the present application;

fig. 10 is a schematic diagram illustrating a method for marking boundary points on a front surface of a reference grid according to an embodiment of the present application;

FIG. 11 is a schematic diagram of a modeling grid after a deformation process according to an embodiment of the present application;

fig. 12 is a schematic diagram of a mesh texture map of a three-dimensional face model according to an embodiment of the present application;

FIG. 13 is a schematic diagram of a background planar grid texture map according to an embodiment of the present application;

fig. 14a and fig. 14b are a side view and a front view of a three-dimensional face model modeling mesh provided by an embodiment of the present application after a mesh deformation process, respectively;

FIG. 15 is a schematic diagram of an image after pose adjustment and rendering according to an embodiment of the present application;

FIGS. 16a and 16b are a front view and a side view of an adjusted three-dimensional mesh model of a background planar modeling mesh according to an embodiment of the present application;

fig. 17 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application;

fig. 18 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

fig. 19 is a block diagram of a software structure of an electronic device according to an embodiment of the present application.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

The terminology used in the following examples is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification of this application and the appended claims, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, such as "one or more", unless the context clearly indicates otherwise. It should also be understood that in the embodiments of the present application, "one or more" means one, two, or more than two; "and/or" describes the association relationship of the associated objects, indicating that three relationships may exist; for example, a and/or B, may represent: a alone, both A and B, and B alone, where A, B may be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

The embodiment of the application mainly aims to solve the problems of poor transformation effect and slow response speed when the face image acquired in real time is processed in the existing face image processing, and specifically introduces the following steps:

the face image processing mode can comprise the steps of changing the face posture or the angle in an image or carrying out special effect processing on the face in the image, and comprises the steps of collecting face images, and processing the face images, such as face beautifying, fat changing, thin changing, distorting mirror and the like.

In an application scenario in which a face pose in an image is changed, in order to improve screen occupation and solve the problem of anti-candid shooting, a notebook with a hidden camera appears, wherein the camera may be a hidden camera arranged on a keyboard below the screen. As shown in fig. 1, the hidden camera can select a special key set in the middle of the keyboard, and the camera is installed through the key set. When the camera is arranged on the key, the camera can be arranged on the side surface of the key through a push type structure, for example, the camera is arranged on the front side of the key, or when the key is a push type key capable of rotating, the camera can be arranged on any side surface.

When the user needs to use the camera, the button can be pressed, the rear side of the button can be fixed through the rotating shaft, the front side of the button can be upwards popped up through the elastic force of the elastic component, so that the included angle between the side face of the keyboard where the camera is located and the keyboard plane in front of the keyboard is larger than 90 degrees, such as 120 degrees, 135 degrees and the like, and the camera can collect normal use of the complete human face image of the user of the notebook.

Of course, the camera can also be set to be flexibly adjusted to a plurality of angles. The rotation angle of the camera can be adjusted according to the integrity of the currently detected face image. For example, when the user is close to the screen, if only the lower part of the face image of the user can be detected according to the image collected by the camera, the camera can be controlled to rotate upwards until the complete face image is obtained, or when the user is far away from the screen, if the face in the collected image is located at the lower part of the image, the angle of the camera can be adjusted, so that the face image is located in the middle of the image.

When a user uses the notebook to carry out video call, as shown in fig. 2, in the video call process, the eye watching direction of the user is usually in the direction corresponding to the screen area, and the angle of the hidden camera shooting the face of the user is the elevation angle, so that when the user carries out video call or shoots the user, a certain included angle β is formed between the shooting angle of the camera and the watching angle of the user, the shot face image can present the image effects of 'big chin', 'towards the nose' and the like as shown in fig. 3, and the requirement of the shooting visual angle expected by the user cannot be met. Moreover, in the video call process, the angle watched by the face in the shot image usually presents an illusion that the user is not watching or paying attention to the call partner (actually, the user is watching the call partner on the screen), which is not beneficial to improving the communication and use experience of the user.

Or, when the desktop acquires the face image of the user by externally connecting a camera, or the tablet computer or the notebook computer acquires the face image of the user by the camera arranged above or below the screen, because the position of the camera is different from the central position of the screen, the angle of the acquired image has a certain depression angle or elevation angle, and the face needs to be changed by a face image processing method, so that the image can display the expected pattern of the shooting visual angle, and the user conversation use experience is improved during video conversation.

In an application scene of performing special effect processing on a face in an image, when an intelligent terminal collects a photo or a video including a face image of a user, in order to obtain a better shooting effect, five sense organs of the face in the shot image can be adjusted, or the figure can be adjusted. Therefore, the user can conveniently release the adjusted photo or video, and the method comprises the step that the user releases a short video through a small video application program (including application programs such as trembling and fast hands), or releases the photo or the small video through platforms such as WeChat, microblog and blog.

In order to transform the acquired two-dimensional image, so as to transform the acquired face image into a front face image, or transform the acquired face image into an image after special effect processing, the following face transformation methods may be included:

the processing of the collected face image in a two-dimensional transformation mode may include: firstly, a face-up photo of a user is obtained and used as reference data for subsequent picture or video processing, and feature points of the face-up photo are detected. And when the photo to be transformed is acquired, identifying the characteristic points in the photo to be transformed. And (3) deforming the acquired image according to a pre-stored face photo serving as a target, wherein the sight direction and the orientation of five sense organs in the deformed face image are not changed.

As shown in fig. 4, the left image is a photograph to be transformed. The photo to be changed may be a photo that a user uses an electronic device, for example, when using a notebook with a hidden camera function to perform a video call, or using a notebook with a hidden camera function to shoot a video, or using an electronic device with a camera and an electronic device whose screen position is not matched, because the user generally gazes at a screen area, the camera is disposed at the front lower part of a human face, the collection angle of the camera is an upward angle, the chin and nostrils of the collected image are large, and the user experience is not good. Of course, the illustration is intended to show the effect that the camera is arranged below the screen, and when the camera is arranged at other positions outside the screen, the obtained image is not the front face image desired by the user. The middle image in fig. 4 is a photograph obtained by deforming a pre-stored front face photograph as a target, although the whole face shape is changed, the sight line direction and the five sense organs of the face are not changed, and the target effect image in the right image in fig. 4 shows that in order to effectively process the face image, the sight line of the eyes in the face needs to be changed, and the display view angle of the nose in the face needs to be changed.

Secondly, when a face is transformed through a target image, a grid is usually established on the image by using feature points, the image is mapped to each triangular patch in the grid according to a transformation matrix or a function, each pixel position needs to be traversed in the transformation process, the pixel color is calculated through interpolation, the larger the transformation processing range is, the larger the processing time needs to be consumed is, usually, the time consumed when only the part of five sense organs is transformed is 4-6ms, the time consumed when only the part of the face is transformed is usually 20-40ms, and the time required when the acquired image is transformed in a full-map mode is 130-170 ms. Therefore, if a good image effect is to be obtained, a large processing delay occurs.

And performing three-dimensional reconstruction according to an acquired image by adopting a three-dimensional deformation and background two-dimensional fusion mode, deforming the reconstructed human face in a three-dimensional space, then re-projecting the deformed human face into a two-dimensional human face target image, and integrating the two-dimensional human face target image with a background image, wherein if the reconstruction, the deformation and the projection are only performed on the human face, the reconstruction, the deformation and the projection can be generally completed within 30-40ms, but a gap can exist between the human face image and the background image, so that the real effect of the image is influenced. When the projected two-dimensional human face target image is fused with the background image, a long time is consumed, and the fusion of the human face image and the background image usually needs 35-55 ms. The flow of generating the corrected two-dimensional face model and fusing needs 65-95ms, and the requirement on real-time video processing cannot be met.

In order to solve the problem that the visual angle of five sense organs of a human face is not correspondingly changed or the time required for finishing the processing of the human face image is long in the prior art, the embodiment of the application constructs the three-dimensional human face model reference grid and the background plane reference grid, and correspondingly adjusts the background plane reference grid through the boundary point of the three-dimensional human face model reference grid and the control point of the background plane reference grid when the three-dimensional human face model reference grid is changed, so that the processing efficiency of the human face image is improved.

The three-dimensional face model reference mesh may be a mesh model of a three-dimensional face model formed by polygons (including triangles, quadrilaterals, and the like). The three-dimensional face model selected by the three-dimensional face reference grid can be a general face model and can also be a three-dimensional face model of a user using image processing equipment.

The background plane reference grid is a plane grid corresponding to the size of the image to be processed. The background plane grid can be composed of a uniform grid or a non-uniform grid.

When processing an image to be adjusted according to the face image processing method of the present application, as shown in fig. 5, the method mainly includes the following three processing steps: s1 reference grid pre-processing, S2 current image modeling, and S3 current image processing. The respective processes will be further described with reference to the drawings.

S1, reference grid preprocessing:

s11 construction of a reference grid

The reference grids to be constructed may include a three-dimensional face model reference grid and a background plane reference grid.

The three-dimensional face model reference grid can be a general face model, a three-dimensional deformation model (3DMM, 3D deformable Models) or a variant model thereof, and the like.

The general face model is a three-dimensional face model grid established according to a general face, and can include a CANDIDE-3 model, a CANDIDE model and the like. As shown in fig. 6, the CANDIDE-3 model includes 113 vertices and 168 faces, and the feature points in the generic face model are matched with the face features in the two-dimensional image (image requiring three-dimensional face model reconstruction) acquired by the camera by adjusting the operation of the points and the faces. The universal human face model has the advantages of small calculation amount and quick response to the reconstruction of the three-dimensional human face image.

The general face model can acquire data of the general face model through a three-dimensional scanner, or can be created through a computer graphics technology, or can be generated through commercial modeling software.

In an optional implementation manner, the general face model may obtain a user of the image processing device, and establish a general face model corresponding to the user.

For example, the face model of the user may be scanned as the generic face model. When the electronic device has a plurality of users, the universal face models corresponding to the plurality of users can be stored, and the universal face model corresponding to the user can be selected according to the user currently using the electronic device.

For example, when a user of a mobile phone logs in an electronic device through a fingerprint or an account, the user currently using the device can be known. And searching the universal face model corresponding to the current user according to the corresponding relation between the pre-stored user and the universal face model. If the matching degree of the current user and the general face model is lower than the preset matching degree in the using process, such as when a new user uses the general face models of a small number of users stored in the image processing device. The general face model corresponding to the group where the user is located can be selected according to the face features of the user, for example, the group where the user is located can be a group determined by middle-aged men, young girls and other groups.

If respective general face models are respectively established for the users, the fitting efficiency of the acquired images and the reference grids of the three-dimensional face models can be improved during the subsequent reconstruction of the three-dimensional face models, and the improvement of the response speed of equipment is facilitated. For example, when a user uses a notebook computer with a camera hiding function to carry out video call or shoots a video, the user can carry out fitting processing on real-time images of a big chin, a big nostril and an upward viewing angle shot by the notebook computer, so that the response speed after the fitting processing can be improved, and the real-time performance of video display can be improved.

Of course, if the current user does not have a preset general face model, a preset general face model corresponding to the user group may be adopted. When the use times of the user reach the preset times, the facial features of the user can be collected, and the corresponding relation between the user and the general face model is increased in the system.

When the three-dimensional deformation model 3DMM is selected as the three-dimensional face model reference grid, the three-dimensional deformation model 3DMM can be established on the basis of a three-dimensional face database, the statistics of the face shape and the face texture is taken as constraints, and the influence of the face posture and the illumination factor can be considered, so that the adjustable precision of the three-dimensional deformation model is higher. As shown in fig. 7, which is a schematic diagram of a three-dimensional deformation model, compared with a general face model, the number of triangle surfaces and vertices of a three-dimensional face model reference mesh based on the three-dimensional deformation model is greatly increased, and the three-dimensional deformation model includes more features and details.

Assuming that the three-dimensional deformation model consists of m face models, where each face model contains a corresponding face 3D coordinate shape vector Si, the new 3D face model can be represented as:

wherein S is_newIn order to create a new face shape model,

representing the mean shape model of the face, s_iThe main component representing the shape of the new face, i.e. the features of the new face that distinguish it from other faces, alpha_iIs the face shape coefficient.

In an optional embodiment, on the basis of the three-dimensional deformation model, facial expression data may be further included, so that the three-dimensional deformation model is further expanded to:

wherein e is_iRepresenting the main component of a facial expression, i.e. the characteristic part, beta, which is distinguished from other facial expressions_iIs a facial expression coefficient. Wherein the face shape coefficient α_iFacial expression coefficient beta_iThe solution can be performed using a least squares method.

Of course, the 3D deformation model corresponding to the user may also be established according to the acquired user image data of the electronic device, and the corresponding 3D deformation model may be invoked according to the user currently using the electronic device. If the matching degree of the 3D deformation model called by the current user and the currently acquired image is lower than a preset value, the 3D deformation model determined by the user group can be adopted.

When the three-dimensional face model reference grid is constructed, the background plane reference grid can be constructed according to the size of the collected image. In the reference mesh diagram shown in fig. 8, the three-dimensional face model reference mesh is overlaid on the background plane reference mesh. The constructed background plane reference grid may be constructed as a grid of predetermined size and shape. The predetermined shaped mesh may include a triangular mesh, a square mesh, and the like. The triangular meshes in the background plane reference mesh can be all the same triangular meshes, or only triangular meshes with the same shape are set in a preset range around the human face, and the triangular meshes which are relatively dense can be set in the preset range in the head region outside the human face region, so that the human face image can be subjected to deformation processing, and the head image in the preset range can be subjected to more accurate adaptive adjustment.

S12 marks boundary points and control points

After the three-dimensional face model reference grid and the background plane reference grid are constructed, further marking processing needs to be carried out on the constructed reference grid, wherein the marking processing comprises marking boundary points of the three-dimensional face model reference grid, fixing points and control points of the background plane reference grid. The boundary points are points on the visible boundary of the face of the three-dimensional face model reference grid under a preset visual angle. As shown in fig. 9, which is a schematic diagram of boundary points marked by a right front view angle of the reference grid, under the right front view angle, more visible parts on the right side of the face model exist, and the distance between the boundary points on the visible boundary and the center line of the front face of the face is longer.

The boundary points may be selected in different numbers depending on the accuracy of the image processing. For example, the higher the accuracy required for image processing, the greater the number of boundary points that can be selected. The boundary points can be uniformly distributed on the intersecting lines, and also can be distributed densely according to the parts of the human face where deformation processing is frequent. As shown in fig. 9, more boundary points may be set near the mouth of the face.

When the visible view angle is the front face of the reference grid, as shown in fig. 10, the visible boundary of the face is determined according to the visible region of the face of the three-dimensional face model reference grid, and the boundary point corresponding to the front view angle can be determined according to the visible boundary.

After the boundary point of the three-dimensional face model reference grid is determined, the control point of the background plane reference grid can be determined according to the boundary point. As shown in fig. 10, the boundary points determined from the frontal view of the three-dimensional face model reference mesh and the control points of the background plane reference mesh determined from the boundary points may completely coincide, but of course, not limited thereto, the distance between the boundary points and the control points may be set to be less than a predetermined distance.

S13 marking fixed points

In addition, in order to enable the background plane reference grid to influence the overall size of the image when stretching or compressing is carried out, a fixed point can be set for the background plane reference grid, the overall size of the background plane reference grid cannot be changed through the fixed point, and on the other hand, the stability of the overall screen is favorably ensured.

After the control points of the background plane reference grid are determined, the fixed points of the background plane reference grid can be further set. As shown in fig. 10, the position of the fixed point may be set at the outer boundary of the reference grid of the background plane, and the size of the image after the deformation processing is not changed by the fixed point on the outer boundary. Of course, by intelligently identifying the target object in the background area of the image, a fixed point may be set at the position of the target object in the image corresponding to the background plane reference grid according to the identified target object.

The boundary points are located in the three-dimensional face model reference grid, and when the three-dimensional face model reference grid is subjected to deformation processing, the positions of the boundary points located on the three-dimensional face model reference grid are also subjected to corresponding changes, including leftward movement, rightward movement, downward movement or upward movement of the boundary points in the plane direction, and also including leftward upper front, leftward upper rear, leftward lower front and the like. According to the change of the position of the boundary point, the deformation of the background plane reference grid is correspondingly determined by the control point, and the background plane reference grid can be compressed or stretched according to a Laplace deformation method.

Assuming that the plane of the background plane reference grid is a plane determined by the X axis and the Y axis of the rectangular coordinate system, after the position of the boundary point is changed, the X coordinate and the Y coordinate of the control point corresponding to the alignment control can be changed according to the change of the X coordinate and the Y coordinate of the boundary point.

For example, the coordinate position of the boundary point before the deformation and the coordinate position after the deformation may be acquired, the changes of the x coordinate and the y coordinate in the coordinate positions before and after the change are extracted, and the change of the coordinate of the control point corresponding to the boundary point may be controlled according to the extracted changes of the x coordinate and the y coordinate.

That is, if the control point and the boundary point coincide with each other, the target position to be deformed by the control point may be determined directly from the x-coordinate and the y-coordinate in the changed coordinate position of the boundary point.

When the corresponding relationship between the control point and the boundary point is a predetermined distance, the variation of the position of the control point corresponding to the boundary point can be correspondingly determined according to the variation of the position of the boundary point in the xoy plane.

For example, the variation of the position of the boundary point in the xoy plane can be decomposed into a horizontal variation and a vertical variation, and the movement of the control point is determined according to the horizontal variation and the vertical variation, so that the control point corresponds to the boundary point on the three-dimensional face model modeling grid. Or, the variation of the boundary point may be decomposed into a movement distance and a movement direction, and the movement of the control point is determined according to the movement distance and the movement direction.

In an optional embodiment of the present application, the article in the image corresponding to the background plane reference grid may be identified, and the article in the image corresponding to the background plane reference grid may be globally stretched or compressed. The method comprises the steps of identifying an article in a two-dimensional image corresponding to a background plane reference grid, determining an equal-proportion adjustment control point of the article, and when any part of the article needs to be deformed, implementing the same deformation processing on other control points of the article according to the deformation processing mode of a deformation position, so that the shape of the article after deformation processing is still in a normal state, and avoiding the defect problem of deformation of the background image caused by fusion.

For example, when there is a target object a near the three-dimensional face model, after the overall contour of the target object a is recognized, the control point of the equal scaling of the boundary position of the target object a is set. If the control points located at the boundary points are stretched or compressed, so that the local part of the target object A is changed, for example, the local part is stretched or extruded, deformation processing is performed according to the control points which are scaled in equal proportion and correspond to the identified target object A, so that the target object A is stretched or compressed integrally, and the influence on the reality of a background image when a background plane modeling grid is obtained due to the deformation of a background plane reference grid is avoided.

It should be noted that the above-mentioned construction process of the reference grid may be performed at any time before the face processing, and the reference grid may be constructed in non-real time. I.e., the construction of the reference grid may be completed before the image is captured, and once constructed, may be repeatedly applied to the captured photographic process, or to the image process of the video frames of the captured video image.

S2, current image modeling:

s21 human face model fitting and shooting parameter obtaining

After the preprocessing of the reference grid is completed, the image can be modeled and used for multiple times according to the pre-constructed reference grid. For example, after one-time reference grid preprocessing is completed, when face transformation processing is required to be started at any time, a pre-constructed reference grid can be called at any time to perform image modeling. Moreover, the reference grid can be constructed in an off-line state, that is, the reference grid does not need to be constructed in the process of processing the face image of the image.

When a user uses the camera to shoot or uses the camera to shoot a video, if an instruction of starting human face processing by the user is received, the human face image processing is carried out on the shot picture in real time, or the human face image processing of a video frame is carried out on the shot video. And restoring the three-dimensional face image corresponding to the face image in the shot picture or video frame by combining the pre-constructed three-dimensional face model reference grid and the background plane reference grid.

Specifically, when modeling the acquired two-dimensional image (a shot photo or a video frame in a shot video), a model fitting method may be used to obtain a three-dimensional mesh model corresponding to the two-dimensional image, where the three-dimensional face mesh may include a three-dimensional face model. The feature points of the face in the image and the feature points in the pre-constructed three-dimensional face model reference grid are used for model fitting, and the three-dimensional face model modeling grid matched with the feature points of the collected two-dimensional image is obtained. And determining shooting parameters of the two-dimensional image according to the positions of the characteristic points of the two-dimensional image, wherein the shooting parameters can comprise a model view matrix and a projection matrix.

When the image is fitted with the pre-constructed three-dimensional face model reference grid, the positions of the vertexes of the three-dimensional face model reference grid can be changed, and the three-dimensional face model reference grid is deformed and the like, so that the positions of the feature points in the three-dimensional face model modeling grid are matched with the positions of the feature points in the acquired image when being mapped to the acquired image. Namely, the positions of the characteristic points mapped by the three-dimensional face model modeling grid are consistent with the positions of the characteristic points in the acquired image.

For example, in the fitting process, the pose of the three-dimensional face model reference grid may be adjusted, including adjusting the yaw angle, pitch angle, or roll angle of the three-dimensional face model reference grid, and then the vertex at the detail image of the pose-adjusted three-dimensional face model reference grid is subjected to fitting deformation, so that the sum of the distances between the feature points in the acquired image and the corresponding feature points in the image projected and mapped by the adjusted three-dimensional face model modeling grid is minimum, or the feature points in the acquired image are completely matched with the corresponding feature points in the image projected and mapped by the adjusted three-dimensional face model modeling grid.

After the three-dimensional face model modeling grid is fitted with the acquired image, a Model View (MV) matrix and a projection matrix of a corresponding posture when the three-dimensional face model reference grid is transformed into the acquired image can be determined according to the corresponding relation of two groups of feature points and the change of the feature points of the three-dimensional face model grid (including the three-dimensional face model reference grid and the three-dimensional face model modeling grid obtained by fitting according to the three-dimensional face model reference grid) before and after fitting. The technology of determining the MV matrix and the projection matrix transformed by the three-dimensional face model modeling grid into the acquired image according to the corresponding relationship of the feature points and the change of the feature points is a known technology, and is not described in detail herein.

For example, after a two-dimensional image of an upward viewing angle is acquired by a camera, the feature points of the face in the two-dimensional image may be fitted to the three-dimensional face model reference grid by analyzing the positions of the feature points of the face in the two-dimensional image and combining the feature points in the preset three-dimensional face model reference grid, that is, the positions of the vertices in the three-dimensional face model reference grid are adjusted, so that the adjusted grid matches the features of the face in the two-dimensional image.

S22 three-dimensional human face model deformation processing

The MV matrix can extract a rotation component therein, and the three-dimensional face model modeling grid is rotated by the same angle according to the extracted rotation component, so that the posture of the three-dimensional face model modeling grid can be transformed into the posture corresponding to the acquired image. Wherein, if the MV matrix is:

then, the rotation component R is extracted from the MV matrix as:

wherein, I0, I1 and I2 are the moduli of vectors formed by the 0 th column, the 1 st column and the 2 nd column of the MV matrix, respectively. And the movement parameters m30, m31 and m32 are not related to the rotation parameters.

And performing attitude adjustment on the three-dimensional face model reference grid according to the MV matrix determined in the fitting process. And according to the rotation component in the model view matrix, performing rotation transformation on the three-dimensional human face model reference grid, and restoring the human face posture corresponding to the acquired image.

In an optional embodiment, when the three-dimensional face model reference mesh is a general face model mesh and is reconstructed by using the general face model mesh, the overall adjustment and the local adjustment of the face model mesh may be included:

the global adjustment may be adjusted to the contour of the model. The overall layout of the general face model, including the positions such as eyes, ears, nose, mouth, eyebrows and the like, can be consistent with the layout of the five sense organs of the picture to be restored as much as possible by means of corresponding characteristic points.

The local adjustment can be carried out on the local details, particularly the facial features, in a micro-adjustment mode, so that the local details are more accurate.

After the adjustment, the face can be reconstructed using a vertex-based interpolation operation.

When the three-dimensional face model reference grid is a three-dimensional deformation model grid and face reconstruction is performed on the three-dimensional deformation model grid, the three-dimensional face model reference grid can be controlled to rotate according to the rotation component in the MV matrix. For example, the three-dimensional human face model reference grid is adjusted to the posture corresponding to the acquired image through the rotation component R in the MV matrix, and the change of the facial features of the three-dimensional human face model reference grid can be further determined according to the positions of the feature points in the five sense organs in the two-dimensional human face model, and the change may include adjusting the facial patterns such as eyes, eyebrows, mouth, nose, ears, or face, so that the adjusted three-dimensional human face model grid is more matched with the human face in the acquired image.

And for the fitted three-dimensional face model modeling grid, the posture of the face in the acquired two-dimensional image is inconsistent. If the face in the two-dimensional image is in an upward view angle and the three-dimensional face model modeling grid is in a forward view angle, the three-dimensional face model modeling grid needs to be rotated to obtain a three-dimensional face model grid texture map for rendering, so that the rotated three-dimensional face model modeling grid is consistent with the view angle of the face in the acquired two-dimensional image.

S23 background grid deformation

After the reconstruction of the three-dimensional face model modeling grid is completed, because the position of the boundary point of the rotated three-dimensional face model may be changed when the three-dimensional face model is transformed, whether the position of the control point in the background plane reference grid needs to be correspondingly adjusted is determined according to whether the position of the boundary point of the preset three-dimensional face model is changed. When the position of a boundary point of the three-dimensional face model modeling grid is changed, a control point corresponding to the boundary point is searched, and the control point corresponding to the boundary point in the background plane reference grid is aligned with the boundary point according to the change of the x coordinate and the y coordinate of the boundary point, so that the background plane reference grid is deformed to obtain the background plane modeling grid.

After determining the control point corresponding to the boundary point after the position change, the deformation magnitude of the control point may be determined according to the change of the coordinates of the boundary point before and after the deformation, for example, according to the change of the x coordinate and the y coordinate of the boundary point, so that the control point is aligned with the boundary point in the x direction and the y direction. And according to the corresponding adjustment control point of the alignment operation, carrying out deformation processing on the background plane reference grid, so that the distance between the position of the control point after the deformation processing and the position of the corresponding boundary point is smaller than a preset value, or the position of the control point after the deformation processing is superposed with the position of the corresponding boundary point.

In an optional embodiment, when an object in an image corresponding to a background plane reference grid is identified, and an equal-scale control point is set according to the identified object, if any one control point in the object needs to be adjusted, that is, when any one control point moves, other control points of the object also move correspondingly, and when any one scaling adjustment is performed, other equal-scale control points are correspondingly controlled to be scaled.

When the background plane reference grid is deformed through the control point, the control point can be combined with the fixed point of the background plane reference grid, so that the size of the image after deformation processing is not changed. The fixed points can be arranged around the background plane reference grid, and can also be arranged at the positions of some objects.

The deformation processing mode of the background plane mesh (including the background plane reference mesh and the background plane modeling mesh) may include laplacian transform. And should not be limited to this, other mesh deformation processing methods such as a skeleton skin animation algorithm can be included.

FIG. 11 is a schematic diagram of a modeled grid after a deformation process. The modeling grid schematic diagram comprises a three-dimensional face model modeling grid and a background plane modeling grid. According to the reference grid shown in fig. 8, after the deformation processing of the model view matrix is performed, and according to the change of the position of the boundary point, the deformation processing is performed on the background plane reference grid by the control point and the fixed point, the schematic diagram of the image of the background plane modeling grid is obtained. As can be seen from fig. 11, the three-dimensional face model reference mesh is subjected to deformation processing by fitting and combining with the model view matrix, including that after the three-dimensional face model reference mesh is subjected to rotational deformation and/or scaling deformation, the positions of the boundary points of the visible boundary of the obtained three-dimensional face model modeling mesh are changed, according to the change of the positions of the boundary points, the positions of the control points of the background plane reference mesh are correspondingly adjusted, the background plane reference mesh is subjected to deformation processing, and the three-dimensional face model modeling mesh and the background plane modeling mesh obtained after deformation can still be effectively fused.

S24 texture image rendering

After a three-dimensional face model modeling grid with the same posture as the acquired two-dimensional image is obtained, the acquired image can be rendered according to the MV matrix and the projection matrix to obtain a three-dimensional face model grid texture map; and extracting a translation component and a scaling component in the MV matrix to form a new matrix, and rendering to obtain a background plane grid texture map by using a projection matrix which is the same as the three-dimensional face model.

When obtaining a three-dimensional face model mesh texture map, a rectangular coordinate system oyx for labeling vertex positions on the three-dimensional face model modeling mesh can be set, and the coordinate position of any vertex on the three-dimensional face model modeling mesh can be expressed as (x, y, z). The XOY plane in the rectangular coordinate system sets a UV coordinate system by which the acquired image can be represented as (u, v). The z coordinate in the vertex coordinates (x, y, z) of all the grids on the three-dimensional face model modeling grid can be set as 0, the x and y coordinates are set as the two-dimensional u and v coordinates of the vertex, a first plane is obtained through rendering, and then the color of each first pixel point on the plane is determined: the position of the pixel can be multiplied by the MV matrix and the projection matrix to determine that the first pixel corresponds to the second pixel on the collected picture, and the value of the color of the first pixel on the first plane is taken according to the corresponding second pixel.

Similarly, when obtaining the background plane mesh texture map, determining a second plane by the background plane modeling mesh, wherein the second plane can correspond to the two-dimensional image to be processed through scaling. Because the background plane grid is modeled into a plane, and there should be no change in posture, for the color of each third pixel point on the plane, only the translation component and the zoom component in the MV matrix need to be obtained, the rotation component does not need to be considered, the rotation component is multiplied by the projection matrix, and the translation component and the zoom component in the MV matrix are multiplied, the third pixel point is determined to correspond to a fourth pixel point on the obtained two-dimensional image to be processed, such as the collected image, and the value is taken according to the color of the third pixel point on the second plane by the corresponding fourth pixel point.

For example, in extracting the translation component and the scaling component in the MV matrix, for a given MV matrix:

its translation component T can be extracted as:

its scaling component S is:

wherein, I0, I1 and I2 are the moduli of vectors formed by the 0 th column, the 1 st column and the 2 nd column of the MV matrix, respectively.

According to the three-dimensional face model mesh texture image and the background plane mesh texture image obtained by rendering, the whole mesh is rendered after the visual angle of the three-dimensional face model modeling mesh is adjusted, and a processed image is obtained.

For example, for the two-dimensional image of the bottom view angle shown in fig. 3, after generating the three-dimensional face model modeling grid corresponding to the face in the two-dimensional image, the corresponding relationship between the pixel values of the pixel points on the rendering plane and the two-dimensional image is determined according to the projection matrix and the MV matrix, so as to obtain the three-dimensional face model grid texture map shown in fig. 12 and the background plane grid texture map shown in fig. 13.

S3, current image processing

S31 face model deformation

And for the restored three-dimensional face model modeling grid, deformation processing needs to be carried out on the three-dimensional face model modeling grid according to the actual use condition.

For example, when a user uses a notebook computer with a hidden camera or uses a desktop computer with an external camera to perform video, in order to improve the video call experience, the posture of a portrait in a video picture is changed from the posture of a watching screen to the posture of the watching camera, and the posture of the restored three-dimensional face model modeling grid needs to be changed, for example, for the notebook with the hidden camera, the overlooking angle of the three-dimensional face model is adjusted, and when the camera is arranged on the left side or the right side of a display of the desktop computer, the yaw angle of the three-dimensional face model modeling grid is adjusted, that is, the three-dimensional face model is rotated to the left or to the right by a certain angle.

The angle of rotation required by the three-dimensional face model modeling grid can be determined according to a rotation component extracted from an MV matrix obtained during modeling of the three-dimensional face model reference grid.

For example, the pose angle of the two-dimensional face image collected by the notebook with hidden camera is 20 degrees upwards horizontally, in order to make the pose of the portrait in the video picture transformed from the pose of the gaze screen to the pose of the gaze camera, the three-dimensional face model modeling grid is rotated downwards by 20 degrees, and the grid front view shown in fig. 14a and the grid side view shown in fig. 14b are obtained. At this time, as can be seen from fig. 14b, the positions of the boundary points of the three-dimensional face model modeling grid are changed, the control points corresponding to the boundary points are not subjected to the corresponding deformation processing, and a gap occurs between the background plane modeling grid and the three-dimensional face model modeling grid after the deformation processing.

When the three-dimensional face model modeling grid after restoring the pose corresponding to the face in the two-dimensional image is subjected to deformation processing, various deformation processing modes can be included according to different practical application scenes. For example, the method may include human face pose deformation processing, human face local deformation processing, and the like.

When the human face posture is deformed, the method can be applied to equipment with unmatched positions of the camera for collecting images and the center of the screen, such as a notebook computer with a hidden camera function, a desktop computer and the like.

For example, when a user takes a picture, makes a video call, or records a video using a notebook computer with a hidden camera, if the user is watching a screen at present, the captured image will be as shown in fig. 3, the camera is located below the direction watched by the face of the user, and the captured image will show an image that is not expected by the user and is big at the chin and towards the nose and the sky.

Before the user uses the notebook of the hidden camera, the reference grid can be constructed offline in advance, including the construction of the three-dimensional face model reference grid and the background plane reference grid. When the constructed reference grid is marked, the position of the face in the three-dimensional face model reference grid in the reference grid is identified, the boundary point of the face is determined, the control point for transforming the background plane grid (including the background plane reference grid and the background plane modeling grid) is determined according to the boundary point, and the fixed point in the background plane grid can be determined according to the amplitude of image transformation, for example, the fixed point can be set to be the edge of the image.

When a camera acquires a face image of a user, characteristic points of the face image can be acquired, the acquired face image is fitted with a three-dimensional face model reference grid based on a three-dimensional reconstruction algorithm, a model view matrix and a projection matrix are determined according to the corresponding relation of the characteristic points and the change of the characteristic points of the three-dimensional face model modeling grid before and after fitting, the fitted three-dimensional face model reference grid is rotated according to the rotation component in the model view matrix to obtain a face posture model corresponding to a shot image, control points needing to be adjusted are determined according to the change of the positions of boundary points of the three-dimensional face model modeling grid obtained after rotation, and the background plane reference grid is subjected to deformation processing according to the control points to ensure that the background plane modeling grid is fused with the three-dimensional face model modeling grid, and determining a three-dimensional face model mesh texture map and a background plane mesh texture map corresponding to the acquired image according to the MV matrix comprising the image shooting parameters and the projection matrix comprising the imaging parameters.

According to the use requirement of the user, for example, according to the size of the shooting angle of view, the face pose is transformed, so that the adjusted angle of view is adjusted downward by a certain angle, for example, downward by 20 degrees, and the adjusted three-dimensional model mesh is rendered, so that the adjusted image effect shown in fig. 15 is obtained, and the frontal face image of the user is obtained. Therefore, the image which is in sight with the user during video chat can be obtained, the background deformation is controlled through the corresponding relation between the control points and the boundary points, and the integration efficiency of the background image is improved.

When a user uses a desktop computer with an external camera or uses a common notebook computer to take pictures, carry out video call or record videos, if the user is watching a screen picture at present, the collected images may collect the images of the side faces of the user or the overlook images of the user, which is not beneficial to improving the call experience of the user.

When the three-dimensional face model reference grid is rotated according to the acquired image, the three-dimensional face model reference grid is rotated in the opposite direction according to the recognized face gesture when the camera is arranged on the left side of the display, so that the gesture of the three-dimensional face model corresponding to the two-dimensional face image is restored. And when the camera is arranged on the right side of the display, the three-dimensional face model reference grid is rotated in the opposite direction to restore the posture of the three-dimensional face image corresponding to the two-dimensional face image, so that the modeling of the currently acquired image is completed.

And when the face model after current modeling is subjected to deformation processing, according to the change of the position of the boundary point of the face three-dimensional model modeling grid, the background plane reference grid is subjected to deformation processing to obtain a background plane modeling grid, so that the background plane modeling grid and the background plane modeling grid are effectively integrated. And obtaining a three-dimensional face model grid texture image and a background plane grid texture image according to the MV matrix and the projection matrix obtained in the fitting process. According to the visual angle in the two-dimensional plane image, deformation processing is carried out on the modeled human face model, wherein the deformation processing comprises left rotation or right rotation and the like, deformation processing is carried out on the rotated background plane reference grid, the deformed whole grid is rendered through the acquired three-dimensional human face model grid texture image and the acquired background plane grid texture image, the processed two-dimensional image is obtained, the watching direction of the human face in the processed two-dimensional image is matched with the position of a camera, and therefore a user can obtain better attention experience when in video call. And the time required by the conversion process is relatively short, thereby being beneficial to improving the real-time performance of video image display and improving the use experience of users.

When the method is applied to human face local deformation processing, after a user acquires an image by using any intelligent device, the image comprises a photo or a video and the like, the user may need to send the acquired image to other users, or send the acquired image to a social service platform, and in order to improve the satisfaction degree of the user, the acquired image needs to be locally adjusted, such as face thinning, chin thinning and the like.

In the application scenario, after the image acquired by the camera is obtained, the face in the acquired image and the three-dimensional face model reference grid are fitted and the posture corresponding to the face in the image is restored based on the preset reference grid model, the MV matrix and the projection matrix are determined according to the fitting process, the boundary points in the three-dimensional face model modeling grid and the control points in the background plane reference grid are subjected to fusion transformation, the three-dimensional face model grid texture map and the background plane grid texture map are obtained according to the MV matrix and the projection matrix, and the three-dimensional face model modeling grid can be subjected to deformation processing according to the preset face transformation requirements, such as preset face beautification parameters. The face beautification parameters may include one or more of an eye size parameter, an eye separation parameter, a face slimming parameter, a mouth size parameter, a pouch removal parameter, a face shape parameter, and a wing of nose size parameter. And further deforming and adjusting the three-dimensional face model modeling grid according to the face beautification parameters. And after the three-dimensional face model modeling grid is adjusted, further performing transformation fusion on the background plane modeling grid, and rendering to obtain a transformed image, for example, an image with a thin face or a thin chin is obtained after transformation.

S32 background grid deformation

According to the control points corresponding to the boundary points of the three-dimensional face model modeling grid, the background plane modeling grid is transformed, for example, laplacian transformation is performed on the background plane modeling grid in combination with the fixed points in the background plane modeling grid, so that a grid front view after the deformation of the background plane modeling grid is completed and a grid side view after the deformation processing is completed are shown in fig. 16a and fig. 16 b. And controlling the three-dimensional face model modeling grid consistent with the visual angle of the face in the acquired two-dimensional image to rotate in the opposite direction through the rotation vector in the MV matrix, so that the rotated three-dimensional face model modeling grid is transferred to a front visual angle, and the front two-dimensional image is generated according to the three-dimensional face model modeling grid of the front visual angle.

S33 rendering to image

Rendering the three-dimensional face model modeling grid and the background plane modeling grid obtained by transformation according to the three-dimensional face model grid texture map generated by rendering and the background plane grid texture map to obtain a rendered three-dimensional face image and a rendered background plane image, and projecting the three-dimensional face image to obtain a processed two-dimensional image.

When the user is a user of the intelligent device, in order to obtain a better shot image, the face deformation processing needs to be performed on the picture collected by the camera or the video frame in the collected video. And the human face deformation processing mode comprises face thinning processing or chin thinning and other deformation processing on the human face.

The three-dimensional face model modeling grid restored by the current collected image can be transformed according to a preset mode of a three-dimensional face model standard grid (a three-dimensional face model grid for beautifying) or preset beautifying parameters, and the three-dimensional face model modeling grid is controlled to be deformed according to a standard three-dimensional face model or beautifying parameters in a mode of setting control points and fixed points on the three-dimensional face model modeling grid, so that the deformed three-dimensional face model modeling grid is obtained.

And detecting the positions of the boundary points on the three-dimensional face model modeling grid obtained after transformation, if the positions of the boundary points on the three-dimensional face model modeling grid obtained after transformation are changed relative to the positions of the boundary points of the three-dimensional face model reference grid before transformation, determining the control points of the corresponding background plane modeling grid according to the boundary points after transformation, and transforming the background plane modeling grid to obtain the background plane modeling grid obtained after transformation. Or when the background plane modeling grid is transformed, a fixed point of the background plane modeling grid can be set, and the reserved attribute of the background plane modeling grid is controlled through the fixed point. For example, the fixed points may be disposed around the background plane modeling grid.

The three-dimensional face model modeling grid and the background plane modeling grid obtained through transformation can be rendered according to the texture features of the three-dimensional grid model before transformation or can be combined with the current light to obtain a rendered grid model, and the rendered grid model is projected according to the current attitude angle to obtain a processed image.

According to the method and the device, the boundary points located on the face and the control points located on the background are set while the reference grid is constructed, the background is rapidly subjected to deformation processing through the corresponding relation between the background points and the control points, and on the premise of adopting the same configuration, the image without a gap effect can be generated, and meanwhile, the processing efficiency of the image can be remarkably improved.

For example, the adopted device configuration information is: when the CPU is i7-8550, the memory is 16GB, and the resolution of an image acquired by a camera is 1280 × 720, by using the face image processing method described in the embodiment of the present application, reconstructing a three-dimensional face model reference grid according to a preset reference grid, obtaining a face three-dimensional model modeling grid after reconstruction, and rendering to obtain a three-dimensional face model grid texture map and a background plane grid texture map, the process usually requires 25-35ms, performing deformation processing on the reconstructed three-dimensional face model modeling grid and the background plane modeling grid, the process usually takes 2-3ms, rendering the image after deformation processing, and obtaining the image after face deformation processing usually takes 4-6ms, so the time consumed in the whole process is about 31-45ms, whereas the existing method of performing two-dimensional transformation using a target image requires 130-170ms, the method for processing the face image needs 65-95ms by the three-dimensional deformation and background fusion mode, and the image processing time is effectively reduced while the face image and the background image are well fused.

It should be understood that, in the foregoing embodiments, each step does not mean an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

Fig. 17 shows a block diagram of a face image processing apparatus according to an embodiment of the present application, which corresponds to the face image processing method described in the foregoing embodiment, and only shows portions related to the embodiment of the present application for convenience of description.

Fig. 17 is a schematic structural diagram of a face image processing apparatus according to an embodiment of the present application, where the face image processing apparatus includes:

an image modeling unit 1701, configured to obtain, by an electronic device, a two-dimensional image to be processed, construct a three-dimensional mesh model corresponding to the two-dimensional image to be processed according to a preset reference mesh, obtain a texture map of the three-dimensional mesh model according to shooting parameters of the two-dimensional image to be processed, and determine boundary points and control points corresponding to the boundary points according to a visible boundary of a face of the reference mesh;

a model deformation unit 1702, configured to perform deformation processing on the three-dimensional mesh model by the electronic device according to a preset deformation requirement in combination with the corresponding relationship between the boundary point and the control point, render the texture image to the three-dimensional mesh model after the deformation processing, and generate a processed image according to the rendered three-dimensional mesh model.

The face image processing apparatus shown in fig. 17 corresponds to the face processing method.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.

The face image processing method provided by the embodiment of the application can be applied to electronic devices with cameras, such as a notebook, a desktop computer, a tablet computer, a mobile phone, a wearable device, a vehicle-mounted device, an Augmented Reality (AR)/Virtual Reality (VR) device, an ultra-mobile personal computer (UMPC), a netbook, a Personal Digital Assistant (PDA), and the like, and the embodiment of the application does not limit the specific types of the electronic devices at all.

Fig. 18 is a block diagram illustrating a partial structure of an electronic device 1800 according to an embodiment of the present disclosure. Referring to fig. 18, the electronic device 1800 includes: memory 1810, camera 1820, display unit 1830, power source 140, and processor 1850. Those skilled in the art will appreciate that the configuration of the electronic device 1800 shown in FIG. 18 is not intended to be limiting of the electronic device 1800 and may include more or fewer components than shown, or some components in combination, or a different arrangement of components.

The following describes each component of the electronic device 100 in detail with reference to fig. 18:

the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a Subscriber Identification Module (SIM) card interface 195, and the like. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

It is to be understood that the illustrated structure of the embodiment of the present invention does not specifically limit the electronic device 100. In other embodiments of the present application, electronic device 100 may include more or fewer components than shown, or some components may be combined, some components may be split, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

Processor 110 may include one or more processing units, such as: the processor 110 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural-Network Processing Unit (NPU), etc. The different processing units may be separate devices or may be integrated into one or more processors.

The controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of instruction fetching and instruction execution.

A memory may also be provided in processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to reuse the instruction or data, it can be called directly from the memory. Avoiding repeated accesses reduces the latency of the processor 110, thereby increasing the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (I2C) interface, an integrated circuit built-in audio (I2S) interface, a Pulse Code Modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a general-purpose input/output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and/or a Universal Serial Bus (USB) interface, etc.

The I2C interface is a bi-directional synchronous serial bus that includes a serial data line (SDA) and a Serial Clock Line (SCL). In some embodiments, processor 110 may include multiple sets of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces, respectively. For example: the processor 110 may be coupled to the touch sensor 180K via an I2C interface, such that the processor 110 and the touch sensor 180K communicate via an I2C bus interface to implement the touch functionality of the electronic device 100.

The modem processor may include a modulator and a demodulator. The modulator is used for modulating a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used for demodulating the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then passes the demodulated low frequency baseband signal to a baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transferred to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be provided in the same device as the mobile communication module 150 or other functional modules, independent of the processor 110.

The electronic device 100 implements display functions via the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and an application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 194 is used to display images, video, and the like. The display screen 194 includes a display panel. The display panel may adopt a Liquid Crystal Display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (active-matrix organic light-emitting diode, AMOLED), a flexible light-emitting diode (FLED), a miniature, a Micro-oeld, a quantum dot light-emitting diode (QLED), and the like. In some embodiments, the electronic device 100 may include 1 or N display screens 194, with N being a positive integer greater than 1.

The electronic device 100 may implement a shooting function through the ISP, the camera 193, the video codec, the GPU, the display 194, the application processor, and the like.

The ISP is used to process the data fed back by the camera 193. For example, when a photo is taken, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converting into an image visible to naked eyes. The ISP can also carry out algorithm optimization on the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in camera 193.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light sensing element converts the optical signal into an electrical signal, which is then passed to the ISP where it is converted into a digital image signal. And the ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signal in standard RGB, YUV and other formats. In some embodiments, the electronic device 100 may include 1 or N cameras 193, N being a positive integer greater than 1.

The digital signal processor is used for processing digital signals, and can process digital image signals and other digital signals. For example, when the electronic device 100 selects a frequency bin, the digital signal processor is used to perform fourier transform or the like on the frequency bin energy.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 may play or record video in a variety of encoding formats, such as: moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, and the like.

The NPU is a neural-network (NN) computing processor that processes input information quickly by using a biological neural network structure, for example, by using a transfer mode between neurons of a human brain, and can also learn by itself continuously. Applications such as intelligent recognition of the electronic device 100 can be realized through the NPU, for example: image recognition, face recognition, speech recognition, text understanding, and the like.

The software system of the electronic device 100 may employ a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention uses an Android system with a layered architecture as an example to exemplarily illustrate a software structure of the electronic device 100.

Fig. 2 is a block diagram of a software configuration of the electronic apparatus 100 according to the embodiment of the present invention.

The layered architecture divides the software into several layers, each layer having a clear role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system is divided into four layers, an application layer, an application framework layer, an Android runtime (Android runtime) and system library, and a kernel layer from top to bottom.

The Android Runtime comprises a core library and a virtual machine. The Android runtime is responsible for scheduling and managing an Android system.

The core library comprises two parts: one part is a function which needs to be called by java language, and the other part is a core library of android.

The application layer and the application framework layer run in a virtual machine. And executing java files of the application program layer and the application program framework layer into a binary file by the virtual machine. The virtual machine is used for performing the functions of object life cycle management, stack management, thread management, safety and exception management, garbage collection and the like.

The system library may include a plurality of functional modules. For example: surface managers (surface managers), Media Libraries (Media Libraries), three-dimensional graphics processing Libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), and the like.

The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

The media library supports a variety of commonly used audio, video format playback and recording, and still image files, among others. The media library may support a variety of audio-video encoding formats, such as MPEG4, h.264, MP3, AAC, AMR, JPG, PNG, and the like.

The three-dimensional graphic processing library is used for realizing three-dimensional graphic drawing, image rendering, synthesis, layer processing and the like.

The 2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is a layer between hardware and software. The inner core layer at least comprises a display driver, a camera driver, an audio driver and a sensor driver.

The application layer is used for running installed applications or applications in the system, including cameras, calendars, maps, WLANs, music, short messages, galleries, calls, navigation, bluetooth, video, etc.

The following describes exemplary workflow of the software and hardware of the electronic device 100 in connection with capturing a photo scene.

When the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is issued to the kernel layer. The kernel layer processes the touch operation into an original input event (including touch coordinates, a time stamp of the touch operation, and other information). The raw input events are stored at the kernel layer. And the application program framework layer acquires the original input event from the kernel layer and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation, and taking a control corresponding to the click operation as a control of a camera application icon as an example, the camera application calls an interface of an application framework layer, starts the camera application, further starts a camera drive by calling a kernel layer, and captures a still image or a video through the camera 193.

The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.

Claims

1. A face image processing method is characterized by comprising the following steps:

the method comprises the steps that electronic equipment obtains a two-dimensional image to be processed, a three-dimensional grid model corresponding to the two-dimensional image to be processed is built according to a preset reference grid, a texture map of the three-dimensional grid model is obtained according to shooting parameters of the two-dimensional image to be processed, boundary points and control points corresponding to the boundary points are determined according to visible boundaries of the face of the reference grid;

and the electronic equipment performs deformation processing on the three-dimensional grid model according to a preset deformation requirement by combining the corresponding relation between the boundary point and the control point, renders the texture image to the three-dimensional grid model after the deformation processing, and generates a processed image according to the rendered three-dimensional grid model.

2. The method according to claim 1, wherein the reference mesh comprises a three-dimensional face model reference mesh and a background plane reference mesh, the three-dimensional mesh model comprises a three-dimensional face model modeling mesh and a background plane modeling mesh, and the step of the electronic device constructing the three-dimensional mesh model corresponding to the two-dimensional image to be processed according to the preset reference mesh comprises:

the electronic equipment fits the three-dimensional face model reference grid with the two-dimensional image to be processed, and acquires shooting parameters of the two-dimensional image to be processed according to the fitted three-dimensional face model reference grid;

and according to the shooting parameters, the electronic equipment performs posture adjustment on the three-dimensional human face model reference grid to obtain a three-dimensional human face model modeling grid, wherein the three-dimensional human face model modeling grid is consistent with the human face posture in the two-dimensional image.

3. The method of claim 2, wherein the shooting parameters include a model view matrix and a projection matrix, and wherein the step of the electronic device performing pose adjustment on the three-dimensional face model reference grid according to the shooting parameters includes:

according to the model view matrix, the electronic equipment extracts a rotation component;

and according to the extracted rotation component, the electronic equipment controls the fitted three-dimensional human face model reference grid to rotate to the human face posture corresponding to the two-dimensional image to be processed.

4. The method for processing the human face image according to claim 2, wherein the step of the electronic device constructing the three-dimensional mesh model corresponding to the two-dimensional image to be processed according to the preset reference mesh further comprises:

determining boundary points of position change in the three-dimensional face model modeling grid after posture adjustment;

and searching a corresponding control point according to the boundary point with the changed position, and performing deformation control on the background plane reference grid according to the searched control point.

5. The method according to claim 2, wherein the texture map comprises a three-dimensional face model mesh texture map and a background plane mesh texture map, and the step of acquiring, by the electronic device, the texture map of the three-dimensional mesh model according to the shooting parameters of the two-dimensional image to be processed comprises:

according to the model view matrix and the projection matrix, the electronic equipment acquires a three-dimensional face model grid texture map;

and according to the projection matrix and the translation vector and the scaling vector in the model view matrix, the electronic equipment acquires the background plane grid texture map.

6. The method for processing the human face image according to claim 5, wherein the step of the electronic device obtaining the mesh texture map of the three-dimensional human face model according to the model view matrix and the projection matrix comprises:

the electronic equipment acquires coordinates of vertexes in the three-dimensional face model modeling grid in a space rectangular coordinate system, and renders the vertexes to obtain a first plane when a z coordinate in the vertex coordinates is 0;

and the electronic equipment determines a second pixel point corresponding to the first pixel point on the two-dimensional image to be processed according to the product of the position of the first pixel point of the first plane and the model view matrix and the projection matrix, and determines the color of the first pixel point according to the color of the second pixel point.

7. The method of claim 5, wherein the step of the electronic device obtaining the background planar mesh texture map according to the projection matrix and the translation vector and the scaling vector in the model view matrix comprises:

the electronic equipment determines a second plane according to the background plane modeling grid, and extracts a translation matrix and a scaling matrix in the model view matrix;

and the electronic equipment determines a fourth pixel point corresponding to each third pixel point on the two-dimensional image to be processed according to the product of the position of each third pixel point on the second plane and the translation matrix, the scaling matrix and the projection matrix, and determines the color of the third pixel point according to the color of the fourth pixel point.

8. The facial image processing method as claimed in any one of claims 1-7, wherein said step of deforming said three-dimensional mesh model by said electronic device comprises:

the electronic equipment acquires the posture of a three-dimensional face model modeling grid in the constructed three-dimensional grid model;

and the electronic equipment rotates the three-dimensional face model modeling grid according to the angle relation between the posture of the three-dimensional face model modeling grid and the target posture.

9. The method for processing the human face image according to any one of claims 1 to 7, wherein the step of the electronic device performing deformation processing on the three-dimensional human face model modeling grid according to a preset deformation requirement comprises:

the electronic equipment acquires preset face beautifying parameters;

and according to the face beautification parameters, the electronic equipment adjusts the three-dimensional face model modeling grid in the three-dimensional grid model.

10. The method of claim 9, wherein the face beautification parameters comprise one or more of an eye size parameter, an eye separation parameter, a face fatness parameter, a mouth size parameter, a pouch removal parameter, a face shape parameter, and a wing of nose size parameter.

11. The method for processing the human face image according to any one of claims 1 to 7, wherein the step of performing deformation processing on the three-dimensional mesh model by the electronic device in combination with the corresponding relationship between the boundary points and the control points comprises:

the electronic equipment acquires a first position of a boundary point on a three-dimensional face model reference grid in the reference grid and a second position of the boundary point on a three-dimensional face model modeling grid in the three-dimensional grid model;

when the distance between the second position and the first position is larger than a preset value, the electronic equipment searches for a control point corresponding to the boundary point;

and the electronic equipment carries out deformation processing on the background plane modeling grid according to the searched control point.

12. The method of claim 11, wherein the step of the electronic device performing deformation processing on the background plane modeling grid according to the searched control point comprises:

the electronic equipment acquires the coordinate variation of the coordinate position of the boundary point on a background plane;

according to the coordinate variation of the coordinate position of the boundary point on a background plane, the electronic equipment determines the target position of the control point;

and according to the target position, the electronic equipment carries out Laplace deformation processing on the background plane modeling grid.

13. The method of claim 12, wherein the step of performing laplacian warping on the background plane modeling mesh by the electronic device according to the target position comprises:

the electronic equipment acquires control points arranged on the background plane modeling grid;

and according to the set control points and the target positions of the control points, the electronic equipment performs Laplace deformation processing on the background plane modeling grid.

14. The method according to claim 1, wherein the three-dimensional face model reference mesh in the reference mesh is a general face model or a three-dimensional deformation model.

15. The method of claim 1, wherein before the step of acquiring the two-dimensional image to be processed by the electronic device, the method further comprises:

the electronic equipment constructs a three-dimensional face model reference grid and a background plane reference grid;

the electronic equipment acquires a face area in a three-dimensional face model reference grid;

according to the visible boundary of the face area, the electronic equipment determines the positions of boundary points and control points.

16. The method for processing the human face image according to claim 1, wherein the step of acquiring the two-dimensional image to be processed by the electronic device comprises the following steps:

the electronic equipment extracts video frames in real time according to videos collected by the camera, and the extracted video frames are used as two-dimensional images to be processed;

or the electronic equipment takes the picture shot by the camera as a two-dimensional image to be processed.

17. An electronic device, characterized in that the electronic device comprises a memory, a processing screen and a computer program, the display screen being for a processed image, the computer program being stored in the memory, the computer program comprising instructions which, when executed by the electronic device, cause the electronic device to carry out the face image processing method of any one of claims 1 to 16.

18. A computer storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the face image processing method according to any one of claims 1 to 16.

19. A computer program product comprising instructions which, when run on an electronic device, cause the electronic device to perform the method of processing face images according to any one of claims 1 to 16.