CN109147012B - Image processing method and device - Google Patents
Image processing method and device Download PDFInfo
- Publication number
- CN109147012B CN109147012B CN201811100506.4A CN201811100506A CN109147012B CN 109147012 B CN109147012 B CN 109147012B CN 201811100506 A CN201811100506 A CN 201811100506A CN 109147012 B CN109147012 B CN 109147012B
- Authority
- CN
- China
- Prior art keywords
- key point
- image
- face image
- user
- user face
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/60—Editing figures and text; Combining figures or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/40—Filling a planar surface by adding surface attributes, e.g. colour or texture
Abstract
The embodiment of the application provides an image processing method and device, wherein the method comprises the following steps: acquiring a pre-established standard face image and acquiring a first key point of the standard face image; acquiring a user face image in a real-time video frame, and acquiring a second key point of the user face image; determining a position transformation relation between the first key point and the second key point, and adjusting a standard face model corresponding to the standard face image based on the position transformation relation to obtain a user face model corresponding to the user face image; and mapping the user face image in the real-time video frame based on the user face model. By the embodiment, the problems of large calculation amount, long calculation time consumption and low real-time performance of video transmission during image processing in a real-time video scene can be solved.
Description
Technical Field
The present application relates to the field of image processing, and in particular, to an image processing method and apparatus.
Background
Currently, a user can receive real-time video images of other users by using a mobile terminal, for example, a viewer receives a live video image of a main broadcast during live broadcasting, and both parties of a chat receive real-time video images of the other party during video chat.
In order to increase the interest of the real-time video, after the mobile terminal collects the real-time video images of the user, the mobile terminal can firstly establish a face model of the user for each frame of image according to a modeling algorithm, then process the face images of the user based on the model, and send the processed face images to terminals of other users. For example, after the mobile terminal collects a real-time video image of a user, a texture, an ornament, and the like are added to the facial image of the user.
In the prior art, the face model of the user needs to be established for each frame of image according to a modeling algorithm, so that the problems of large image processing calculation amount, long calculation time consumption and low real-time performance of video transmission are solved.
Disclosure of Invention
The embodiment of the application aims to provide an image processing method and an image processing device, so as to solve the problems of large computation amount, long computation time consumption and low real-time performance of video transmission when image processing is performed in a real-time video scene.
In order to solve the above technical problem, the embodiment of the present application is implemented as follows:
in a first aspect, an embodiment of the present application provides an image processing method, including:
acquiring a pre-established standard face image and acquiring a first key point of the standard face image;
acquiring a user face image in a real-time video frame, and acquiring a second key point of the user face image;
determining a position transformation relation between the first key point and the second key point, and adjusting a standard face model corresponding to the standard face image based on the position transformation relation to obtain a user face model corresponding to the user face image;
and mapping the user face image in the real-time video frame based on the user face model.
In a second aspect, an embodiment of the present application provides an image processing apparatus, including:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a pre-established standard face image and acquiring a first key point of the standard face image;
the second acquisition module is used for acquiring a user face image in a real-time video frame and acquiring a second key point of the user face image;
the model generation module is used for determining a position transformation relation between the first key point and the second key point, and adjusting a standard face model corresponding to the standard face image based on the position transformation relation to obtain a user face model corresponding to the user face image;
and the mapping processing module is used for mapping the user face image in the real-time video frame based on the user face model.
In a third aspect, an embodiment of the present application provides an image processing apparatus, including: a memory, a processor and computer executable instructions stored on the memory and executable on the processor, which when executed by the processor implement the steps of the image processing method as described in the first aspect above.
In a fourth aspect, the present application provides a computer-readable storage medium for storing computer-executable instructions, which when executed by a processor implement the steps of the image processing method according to the first aspect.
According to the embodiment of the application, in a real-time video scene, after a first key point of a standard face image and a second key point of a user face image are obtained, the position conversion relation between the first key point and the second key point can be determined, the standard face model is adjusted based on the position conversion relation to obtain the user face model, the user face image is subjected to mapping processing in a real-time video frame based on the user face model, so that the user face model does not need to be established respectively for each frame of image, the user face model can be obtained simply and quickly by adopting a mode of adjusting the standard face model, the calculation amount and time consumption required by image processing are reduced, and the real-time performance of video transmission is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without any creative effort.
Fig. 1 is a schematic view of an application scenario of an image processing method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of an image processing method according to an embodiment of the present application;
fig. 3 is a schematic distribution diagram of a first key point according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a distribution of initial keypoints provided by an embodiment of the present application;
FIG. 5 is a schematic diagram of a standard face model provided in an embodiment of the present application;
FIG. 6 is a schematic diagram of a user face model provided in an embodiment of the present application;
FIG. 7 is a diagram illustrating a standard mapping image corresponding to a standard facial image according to an embodiment of the present application;
FIG. 8 is a schematic diagram of a user map image corresponding to a user face image according to an embodiment of the present application
Fig. 9 is a schematic flowchart of an image processing method according to another embodiment of the present application;
fig. 10 is a schematic block diagram of an image processing apparatus according to an embodiment of the present disclosure;
fig. 11 is a schematic block diagram of an image processing apparatus according to another embodiment of the present disclosure;
fig. 12 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making creative efforts shall fall within the protection scope of the present application.
In view of the problems of large image processing computation amount, long computation time consumption and low video transmission real-time performance in image processing in a real-time video scene in the prior art, the embodiments of the present application provide an image processing method and an image processing device to solve the problems.
Fig. 1 is a schematic view of an application scenario of an image processing method according to an embodiment of the present application, as shown in fig. 1, the scenario includes a first mobile terminal 100, a server 200, and a second mobile terminal 300, where the first mobile terminal 100 and the second mobile terminal 300 are respectively in communication connection with the server 200, both the first mobile terminal 100 and the second mobile terminal 300 have a video function, after the video function is turned on, the first mobile terminal 100 can obtain a real-time video frame and send the real-time video frame to the server 200, and the server 200 can perform image processing on the real-time video frame by using the image processing method in the embodiment of the present application and send the processed real-time video frame to the second mobile terminal 300, thereby achieving an effect of performing image processing on a video frame and a real-time video between the first mobile terminal 100 and the second mobile terminal 300. Of course, it can be understood that the image processing method in the present embodiment may also be executed by the first mobile terminal 100 or the second mobile terminal 300 when the first mobile terminal 100 or the second mobile terminal 300 has the image processing function.
Fig. 2 is a schematic flowchart of an image processing method according to an embodiment of the present application, where the image processing method may be executed by a server or a mobile terminal capable of performing image processing, and the method is not limited in this embodiment, and as shown in fig. 2, the method includes the following steps:
step S202, a pre-established standard face image is obtained, and a first key point of the standard face image is obtained;
step S204, acquiring a user face image in the real-time video frame, and acquiring a second key point of the user face image;
step S206, determining a position conversion relation between the first key point and the second key point, and adjusting a standard face model corresponding to the standard face image based on the position conversion relation to obtain a user face model corresponding to the user face image;
and step S208, based on the user face model, mapping processing is carried out on the user face image in the real-time video frame.
Therefore, according to the embodiment of the application, in a real-time video scene, after the first key point of the standard face image and the second key point of the user face image are obtained, the position transformation relation between the first key point and the second key point can be determined, the standard face model is adjusted based on the position transformation relation to obtain the user face model, the user face image is subjected to mapping processing in a real-time video frame based on the user face model, the user face model does not need to be established respectively for each frame of image, the user face model can be obtained simply and quickly by adjusting the standard face model, the calculation amount and time consumption required by image processing are reduced, and the real-time performance of video transmission is improved.
In the above step S202, a standard face image established in advance is acquired, and a first key point of the standard face image is acquired. Wherein the standard facial image may be a facial image meeting the image processing requirements, which is pre-established according to the requirements before the method in fig. 2 is performed, and the standard facial image may be matched with the facial features of most users. In the embodiment of the present application, after a standard facial image is established in advance, the positions of the five sense organs and the cheek may be determined in the standard facial image, and a certain number of feature points are collected at the positions of the five sense organs and the cheek as first key points.
In step S204, a user face image in the real-time video frame is obtained, and a second key point of the user face image is obtained. When the scheme is executed by the server, the server can receive the real-time video frame sent by the mobile terminal and acquire the face image of the user from the real-time video frame. When the scheme is executed by the mobile terminal, the mobile terminal can acquire the face image of the user from the real-time video frame after acquiring the real-time video frame.
In step S204, the second key point of the user face image may be obtained by:
(a1) Acquiring key points in the face image of the user according to the acquisition part of the first key point in the standard face image to obtain an initial key point;
(a2) If the number of the initial key points is less than that of the first key points, generating corresponding key points for the excessive first key points in the face image of the user according to the corresponding relation between the first key points and the initial key points and the relative position relation between the first key points;
(a3) The initial keypoints and the generated keypoints are collectively used as second keypoints of the user face image.
In the above action (a 1), first, the collection location of the first key point in the standard facial image is determined, where the collection location of the first key point includes, but is not limited to, the location of the five sense organs and the set location in the cheek, and then according to the collection location of the first key point, the key point is collected at the same location in the facial image of the user, so as to obtain the initial key point.
Since the initial keypoints are acquired according to the acquisition parts of the first keypoints, the initial keypoints and the first keypoints have a corresponding relationship, for example, the first keypoints with the same acquisition parts correspond to the initial keypoints. When the accuracy of the acquisition algorithm for acquiring the initial key points is poor, the number of the initial key points may be less than the number of the first key points, and therefore in the above action (a 2), when the number of the initial key points is less than the number of the first key points, corresponding key points are generated for the excessive first key points in the user facial image according to the corresponding relationship between the first key points and the initial key points and the relative position relationship between the first key points. The first key points that are added are the first key points that are added compared with the initial key points.
Specifically, an interpolation algorithm may be adopted to generate corresponding key points for the excessive first key points in the user face image according to the correspondence between the first key points and the initial key points and the relative positional relationship between the first key points. The interpolation algorithm includes, but is not limited to, a linear interpolation algorithm, a bezier curve interpolation algorithm, a polygon offset interpolation algorithm, and the like.
Fig. 3 is a schematic distribution diagram of first keypoints provided in an embodiment of the present application, and fig. 4 is a schematic distribution diagram of initial keypoints provided in the embodiment of the present application. As shown in fig. 3 and 4, the first keypoint and the initial keypoint are represented by means of a serial number, respectively. The number 74 first keypoint and the number 83 first keypoint in fig. 3 correspond to the number 1 initial keypoint and the number 66 initial keypoint in fig. 4, respectively, and the number 102 first keypoint in fig. 3 is an extra keypoint located between the number 74 first keypoint and the number 83 first keypoint. According to the relative position relationship among the numbers 102, 74 and 83 and the corresponding relationship among the number 1 key point, the number 66 key point, the number 74 key point and the number 83 key point, the key points which are lacked relative to the number 102 first key point can be marked in the face image of the user through a linear difference algorithm, so that the corresponding key points are generated for the excessive first key points.
Similarly, the interpolation algorithm may also include a bezier curve interpolation algorithm. If the number of the first key points of the lip part in fig. 3 is greater than the number of the initial key points of the lip part in fig. 4, a curve may be constructed at the lip part in fig. 4 by using a bezier curve difference algorithm, and corresponding key points are generated in the curve for the excessive first key points by using the correspondence between the first key points of the lip part in fig. 3 and the initial key points of the lip part in fig. 4 and the relative positional relationship between the first key points of the lip part.
Likewise, interpolation algorithms also include polygon offset interpolation algorithms. In fig. 3, the number of the first key points at the pupil position is greater than that of the initial key points at the pupil position in fig. 4, in this embodiment, the initial key points at the pupil position in fig. 4 may be connected to obtain a first polygon, the first polygon is expanded outward by a certain distance according to the corresponding relationship between the first key points at the pupil position in fig. 3 and the initial key points at the pupil position in fig. 4 and the relative positional relationship between the first key points at the pupil position in fig. 3 to obtain a second polygon, and corresponding key points are generated for the excess first key points on the second polygon.
It can be understood that the interpolation algorithm is not limited to the three algorithms illustrated above, and will not be described herein.
Finally, by the above action (a 3), the initial key point and the generated key point are collectively used as the second key point of the user face image.
Therefore, according to the embodiment, when the number of the first key points is more than that of the initial key points, corresponding key points can be generated for the excessive first key points in the user face image, and the initial key points and the generated key points are jointly used as the second key points of the user face image, so that the number of the second key points of the user face image is enough, and the accuracy of the subsequent generation of the user face model is improved.
In this embodiment, if the number of the initial key points is equal to the number of the first key points, the initial key points may be directly used as the second key points, and step S206 is executed.
In step S206, determining the position transformation relationship between the first key point and the second key point includes:
(b1) Determining the corresponding relation between the first key point and the second key point according to the acquisition part of the first key point in the standard facial image and the acquisition part of the second key point in the user facial image;
(b2) And determining the position conversion relationship between the first key point and the second key point according to the determined corresponding relationship, and the position coordinates of the first key point in the standard face image and the position coordinates of the second key point in the user face image.
As can be seen from the above description of step S204, in the present embodiment, the capturing position of the first key point in the standard facial image is the same as the capturing position of the second key point in the user facial image, so in the above action (b 1), the corresponding relationship between the first key point and the second key point can be determined according to the capturing position of the first key point in the standard facial image and the capturing position of the second key point in the user facial image. Table 1 is a schematic table of a correspondence relationship between a first key point and a second key point provided in an embodiment of the present application, and in table 1, the first key point and the second key point are represented by using serial numbers.
TABLE 1
As shown in table 1, the first key point 01 corresponds to the second key point 100, both of which are key points collected at the forehead position; the first key point 02 corresponds to the second key point 200, and both are key points collected at the nose position; the first key point 03 corresponds to the second key point 300, and both are key points collected at the left mouth corner position; first keypoint 04 corresponds to second keypoint 400, both keypoints being collected at the right corners of the mouth.
In the above-mentioned act (b 2), after the correspondence is determined, a positional transformation relationship between the first key point and the second key point may be determined based on the correspondence, and the positional coordinates of the first key point in the standard face image and the positional coordinates of the second key point in the user face image.
For example, the coordinates of the first key point 01 in the standard facial image are (X1, Y1), and the coordinates of the corresponding second key point 100 in the user facial image are (X2, Y2), where X1, Y1, X2, Y2 are positive numbers, and thus, the positional transformation relationship between the first key point 01 and the second key point 100 can be represented as X1+ c = X2, and Y1+ d = Y2, and according to the positional transformation relationship, the abscissa point X1 of the first key point 01 is translated by c coordinate positions in the horizontal direction, that is, the abscissa X2 of the second key point 100 can be obtained, and the ordinate point Y2 of the first key point 01 is likewise translated by d coordinate positions in the vertical direction, that is, the ordinate Y2 of the second key point 100 can be obtained. In this way, a positional transformation relationship between each first keypoint and its corresponding second keypoint can be obtained, which can be expressed in the form of the above-mentioned positional transformation formula (X1 + c = X2, Y1+ d = Y2).
After obtaining a position transformation relationship between each first key point and its corresponding second key point, the method may adjust, based on the position transformation relationship, a standard face model corresponding to the standard face image to obtain a user face model corresponding to the user face image, and the process specifically includes:
(c1) Marking a first key point on the standard face model;
(c1) Moving the first key point on the standard face model based on a position transformation relation between the first key point and the second key point;
(c3) In the process of moving the first key point, adjusting the standard face model based on the position of the moved first key point;
(c4) And when the position coordinates of the moved first key point in the standard face image are the same as the position coordinates of the second key point corresponding to the first key point in the user face image, determining that the model adjustment is finished, and taking the adjusted standard face model as the user face model corresponding to the user face image.
Specifically, first keypoints are first labeled on a standard face model. Fig. 5 is a schematic diagram of a standard face model according to an embodiment of the present application, where the standard face model is a model corresponding to a standard face image obtained based on the first key point shown in fig. 3, and in fig. 5, a connection point between every two connection lines is the first key point.
In this embodiment, when the standard face model is generated based on the first keypoints, all the first keypoints may be connected to obtain a plurality of triangular topological structures, and the standard face model is obtained by combining the plurality of triangular topological structures. In order to ensure the accuracy of the standard facial model, when determining the first key points, a sufficient number of first key points may be determined, for example, as many first key points as possible are collected around the eyes and in the cheek of the standard facial image, so as to improve the accuracy of the standard facial model.
After the first key points are marked, since the position transformation relationship between the first key points and the second key points can be known through the above-mentioned actions (b 1) and (b 2), the first key points can be moved on the standard face model based on the position transformation relationship, and in the process of moving the first key points, the positions of the connecting lines between every two first key points are synchronously moved, so that the standard face model is adjusted.
And in the process of moving the first key point, comparing the position coordinates between the moved first key point and the corresponding second key point, determining that the model adjustment is finished when the position coordinates of the moved first key point in the standard face image are the same as the position coordinates of the corresponding second key point in the user face image, and taking the adjusted standard face model as the user face model corresponding to the user face image.
Taking table 1 as an example, the first keypoint 01 is moved in the standard face model, when the position coordinate of the moved first keypoint 01 in the standard face image is the same as the position coordinate of the second keypoint 100 corresponding to the moved first keypoint in the user face image, it is determined that the model adjustment is completed, and the adjusted standard face model is used as the user face model corresponding to the user face image.
Fig. 6 is a schematic diagram of a user face model obtained by adjusting the standard face model shown in fig. 5, where the user face model corresponds to the second key point shown in fig. 4 according to an embodiment of the present application.
After obtaining the user face model, the step S208 may be executed, and the mapping process is performed on the user face image in the real-time video frame based on the user face model, where the action specifically includes:
(d1) Drawing a chartlet image for the user face image based on the user face model;
(d2) Determining a fusion mode of the map image and the face image of the user according to the type of the map image;
(d3) According to the fusion mode, the mapping image and the user face image are fused in the real-time video frame so as to perform mapping processing on the user face image.
In this embodiment, a chartlet image drawing rule is preset, where the drawing rule may be a UV drawing rule, and the drawing rule is used to represent different drawing modes corresponding to different positions in the chartlet image. In act (d 1), the chartlet image drawing rule is obtained, and a chartlet image is drawn for the user face image based on the user face model according to the rule. In this embodiment, when a mapping image is drawn for a user face image, color sampling may be performed from a standard mapping image corresponding to the standard face image. Fig. 7 is a schematic diagram of a standard map image corresponding to a standard facial image according to an embodiment of the present application, where the map image is a full-face texture image. Fig. 8 is a schematic diagram of a user map image corresponding to a user face image according to an embodiment of the present application, where the map image is a full-face texture image. In drawing fig. 8, color sampling may be performed from fig. 7.
A corresponding relationship between the type of the chartlet image and the fusion mode of the facial image of the user is also preset in the embodiment, and table 2 is a schematic table of the corresponding relationship between the type of the chartlet image and the fusion mode of the facial image of the user provided in the embodiment of the present application, as shown in table 2, when the chartlet image is a full-face texture image, the corresponding fusion mode is a positive bottom-folded mode, and when the chartlet image is a partial makeup image (such as eyelashes and lip gloss), the corresponding fusion mode is a soft light superposition mode.
TABLE 2
| Type of image to be pasted | Fusion mode |
| Full face texture image | Positive plate bottom-folding mode |
| Local makeup image | Soft light superposition mode |
In this embodiment, based on the correspondence, a fusion mode corresponding to the drawn mapping image is determined according to the type of the drawn mapping image, and based on the fusion mode, the mapping image and the user face image are fused in the real-time video frame, so as to perform mapping processing on the user face image.
In an embodiment, through the steps S202 to S208, a full-face texture map can be applied to the face of the user in the real-time video process, for example, an effect that the user becomes old is achieved, or a partial map can be applied to the face of the user, for example, an effect of partial makeup is achieved, so that interestingness of the real-time video is enriched, and video experience of the user is improved.
Fig. 9 is a flowchart illustrating an image processing method according to another embodiment of the present application. The image processing method may be executed by a server or a mobile terminal capable of performing image processing, and is not limited in this embodiment, as shown in fig. 9, the flow of the method includes the following steps in addition to all the flows of S202-S208 in fig. 2:
step S210, after the mapping processing is carried out on the face image of the user, a face deformation area preset in the face image of the user and a face deformation mode corresponding to the face deformation area preset in the face image of the user are obtained;
step S212, selecting a target deformer for a preset face deformation area from a plurality of preset deformers according to the corresponding relation between the face deformation area and the deformer type;
in step S214, the image in the face deformation region is subjected to deformation processing in a face deformation manner using the target deformer.
In this embodiment, a face deformation region and a face deformation manner corresponding to the preset face deformation region are preset, for example, the face deformation region is preset to be an eye region, and the face deformation manner corresponding to the eye region is preset to be eye enlargement or eye reduction. Or, the preset face deformation area is a chin area, and the face deformation mode corresponding to the chin area is the lengthening or shortening of the chin. The face deformation area and the face deformation mode corresponding to the face deformation area can be set by a user before real-time video is carried out.
Further, in this embodiment, a correspondence relationship between a face deformation region and a deformer type is preset, and table 3 is a schematic table of a correspondence relationship between a face deformation region and a deformer type provided in an embodiment of the present application, where as shown in table 3, an eye region corresponds to a circular cluster deformer, and a chin region corresponds to a cylindrical cluster deformer. In table 3, the circular cluster deformer and the cylindrical cluster deformer are a plurality of preset deformers.
TABLE 3
| Area of facial deformation | Deformer type |
| Eye region | Round shape deforming device for cocooning frame |
| Chin area | Cylindrical shape deforming device for cocooning frame |
In the above step S212, a target deformer, for example, a circular cluster deformer, may be selected for a predetermined face deformation region among a plurality of preset deformers in correspondence as shown in table 3.
Finally, the image in the face deformation region is subjected to deformation processing in a face deformation manner by the target deformer, through step S214.
When an image is subjected to deformation processing, for example, the target deformer is a circular cluster deformer, the face deformation region is an eye region, and the face deformation mode is eye enlargement, the deformation weight of the circular cluster deformer is determined first, and then the image in the eye region is subjected to enlargement processing based on the deformation weight of the circular cluster deformer.
In this embodiment, after the mapping process is performed on the facial image of the user, through the steps S210 to S214, the facial deformation process with different effects can be performed on the mapped image, for example, an amplification effect on the eyes of the user is realized, so that the interest of the real-time video is enriched, and the video experience of the user is improved.
Further, an embodiment of the present application further provides an image processing apparatus, fig. 10 is a schematic diagram of module compositions of the image processing apparatus according to an embodiment of the present application, and as shown in fig. 10, the apparatus includes:
a first obtaining module 101, configured to obtain a pre-established standard face image, and obtain a first key point of the standard face image;
a second obtaining module 102, configured to obtain a user face image in a real-time video frame, and obtain a second key point of the user face image;
the model generating module 103 is configured to determine a position transformation relationship between the first key point and the second key point, and adjust a standard face model corresponding to the standard face image based on the position transformation relationship to obtain a user face model corresponding to the user face image;
a mapping processing module 104, configured to perform mapping processing on the user face image in the real-time video frame based on the user face model.
Optionally, the second obtaining module 102 is specifically configured to:
acquiring key points in the user face image according to the acquisition part of the first key point in the standard face image to obtain an initial key point;
if the number of the first key points is more than that of the initial key points, generating corresponding key points for the excessive first key points in the user facial image according to the corresponding relation between the first key points and the initial key points and the relative position relation between the first key points;
the initial key points and the generated key points are taken as second key points of the face image of the user together.
Optionally, the model generation module 103 is specifically configured to:
determining the corresponding relation between the first key point and the second key point according to the acquisition part of the first key point in the standard facial image and the acquisition part of the second key point in the user facial image;
and determining the position transformation relation between the first key point and the second key point according to the determined corresponding relation, and the position coordinates of the first key point in the standard face image and the position coordinates of the second key point in the user face image.
Optionally, the model generation module 103 is specifically configured to:
marking the first keypoint on the standard face model;
moving the first keypoint on the standard face model based on a positional transformation relationship between the first keypoint and the second keypoint;
in the process of moving the first key point, adjusting the standard face model based on the position of the moved first key point;
and when the position coordinates of the first key point in the standard face image after the movement are the same as the position coordinates of the second key point corresponding to the first key point in the user face image, determining that the model adjustment is finished, and taking the adjusted standard face model as a user face model corresponding to the user face image.
Optionally, the map processing module 104 is specifically configured to:
drawing a chartlet image for the user face image based on the user face model;
determining a fusion mode of the map image and the user face image according to the type of the map image;
and according to the fusion mode, fusing the mapping image and the user face image in the real-time video frame so as to map the user face image.
Fig. 11 is a schematic block diagram of an image processing apparatus according to another embodiment of the present disclosure, and as shown in fig. 11, the another embodiment of the present disclosure further includes the following modules based on fig. 10:
a third obtaining module 111, configured to obtain a preset face deformation region in the user face image and a face deformation mode corresponding to the preset face deformation region after performing mapping processing on the user face image;
a deformer selecting module 112, configured to select a target deformer for a preset face deformation area from a plurality of preset deformers according to a correspondence between the face deformation area and a deformer type;
and a deformation processing module 113, configured to perform deformation processing on the image in the face deformation region according to the face deformation manner by using the target deformer.
According to the method and the device, in a real-time video scene, after the first key point of the standard face image and the second key point of the user face image are obtained, the position conversion relation between the first key point and the second key point can be determined, the standard face model is adjusted based on the position conversion relation to obtain the user face model, and the user face image is subjected to mapping processing in the real-time video frame based on the user face model, so that the user face model can be simply and quickly obtained by adjusting the standard face model without respectively establishing the user face model for each frame of image, the calculation amount and time consumption required by image processing are reduced, and the real-time performance of video transmission is improved.
The image processing apparatus according to the embodiment of the present application may execute the image processing method and implement the processes and effects of the embodiment of the image processing method, which are not described herein again.
Further, an embodiment of the present application further provides an image processing apparatus, and fig. 12 is a schematic structural diagram of the image processing apparatus provided in an embodiment of the present application, as shown in fig. 12, the apparatus includes: memory 1201, processor 1202, bus 1203 and communication interface 1204. The memory 1201, processor 1202, and communication interface 1204 communicate via the bus 1203, and the communication interface 1204 may include input and output interfaces including, but not limited to, a keyboard, a mouse, a display, a microphone, and the like.
In fig. 12, the memory 1201 has stored thereon computer-executable instructions executable on the processor 1202, which when executed by the processor 1202 implement the following process:
acquiring a pre-established standard face image and acquiring a first key point of the standard face image;
acquiring a user face image in a real-time video frame, and acquiring a second key point of the user face image;
determining a position transformation relation between the first key point and the second key point, and adjusting a standard face model corresponding to the standard face image based on the position transformation relation to obtain a user face model corresponding to the user face image;
and mapping the user face image in the real-time video frame based on the user face model.
Optionally, the computer executable instructions, when executed by the processor, the obtaining a second keypoint of the user face image comprises:
acquiring key points in the user face image according to the acquisition part of the first key point in the standard face image to obtain an initial key point;
if the number of the first key points is more than that of the initial key points, generating corresponding key points for the excessive first key points in the user facial image according to the corresponding relation between the first key points and the initial key points and the relative position relation between the first key points;
and using the initial key point and the generated key point together as a second key point of the face image of the user.
Optionally, the computer executable instructions, when executed by the processor, the determining the location transformation relationship between the first keypoint and the second keypoint, comprise:
determining the corresponding relation between the first key point and the second key point according to the acquisition part of the first key point in the standard facial image and the acquisition part of the second key point in the user facial image;
and determining the position transformation relation between the first key point and the second key point according to the determined corresponding relation, and the position coordinates of the first key point in the standard facial image and the position coordinates of the second key point in the user facial image.
Optionally, when the computer-executable instructions are executed by the processor, the adjusting a standard face model corresponding to the standard face image based on the position transformation relation to obtain a user face model corresponding to the user face image includes:
marking the first key point on the standard face model;
moving the first keypoint on the standard face model based on a positional transformation relationship between the first keypoint and the second keypoint;
in the process of moving the first key point, adjusting the standard face model based on the position of the moved first key point;
and when the position coordinates of the first key point in the standard face image after the movement are the same as the position coordinates of the second key point corresponding to the first key point in the user face image, determining that the model adjustment is finished, and taking the adjusted standard face model as a user face model corresponding to the user face image.
Optionally, when executed by the processor, the computer-executable instructions perform mapping processing on the user face image in the real-time video frame based on the user face model, including:
drawing a chartlet image for the user face image based on the user face model;
determining a fusion mode of the map image and the user face image according to the type of the map image;
and according to the fusion mode, fusing the mapping image and the user face image in the real-time video frame so as to map the user face image.
Optionally, the computer executable instructions, when executed by the processor, further comprise:
after the user face image is subjected to mapping processing, acquiring a preset face deformation area in the user face image and a face deformation mode corresponding to the preset face deformation area;
selecting a target deformer for the preset face deformation area from a plurality of preset deformers according to the corresponding relation between the face deformation area and the deformer type;
and carrying out deformation processing on the image in the face deformation area according to the face deformation mode by utilizing the target deformer.
According to the embodiment of the application, in a real-time video scene, after a first key point of a standard face image and a second key point of a user face image are obtained, the position conversion relation between the first key point and the second key point can be determined, the standard face model is adjusted based on the position conversion relation to obtain the user face model, the user face image is subjected to mapping processing in a real-time video frame based on the user face model, so that the user face model does not need to be established respectively for each frame of image, the user face model can be obtained simply and quickly by adopting a mode of adjusting the standard face model, the calculation amount and time consumption required by image processing are reduced, and the real-time performance of video transmission is improved.
The image processing device of the embodiment of the present application may execute the image processing method and implement the processes and effects of the embodiment of the image processing method, which are not described herein again.
Further, an embodiment of the present application also provides a computer-readable storage medium for storing computer-executable instructions, which when executed by a processor implement the following process:
acquiring a pre-established standard face image and acquiring a first key point of the standard face image;
acquiring a user face image in a real-time video frame, and acquiring a second key point of the user face image;
determining a position transformation relation between the first key point and the second key point, and adjusting a standard face model corresponding to the standard face image based on the position transformation relation to obtain a user face model corresponding to the user face image;
and mapping the user face image in the real-time video frame based on the user face model.
Optionally, the computer executable instructions, when executed by a processor, the obtaining a second keypoint of the image of the user's face comprises:
acquiring key points in the user face image according to the acquisition part of the first key point in the standard face image to obtain an initial key point;
if the number of the first key points is more than that of the initial key points, generating corresponding key points for the excessive first key points in the user facial image according to the corresponding relation between the first key points and the initial key points and the relative position relation between the first key points;
the initial key points and the generated key points are taken as second key points of the face image of the user together.
Optionally, the computer-executable instructions, when executed by a processor, determine a position transformation relationship between the first keypoint and the second keypoint, comprising:
determining the corresponding relation between the first key point and the second key point according to the acquisition part of the first key point in the standard facial image and the acquisition part of the second key point in the user facial image;
and determining the position transformation relation between the first key point and the second key point according to the determined corresponding relation, and the position coordinates of the first key point in the standard face image and the position coordinates of the second key point in the user face image.
Optionally, when executed by a processor, the adjusting a standard face model corresponding to the standard face image based on the position transformation relation to obtain a user face model corresponding to the user face image includes:
marking the first keypoint on the standard face model;
moving the first keypoint on the standard face model based on a positional transformation relationship between the first keypoint and the second keypoint;
in the process of moving the first key point, adjusting the standard face model based on the position of the moved first key point;
and when the position coordinates of the first key point in the standard face image after the movement are the same as the position coordinates of the second key point corresponding to the first key point in the user face image, determining that the model adjustment is finished, and taking the adjusted standard face model as a user face model corresponding to the user face image.
Optionally, when executed by a processor, the mapping the user face image in the real-time video frame based on the user face model comprises:
drawing a chartlet image for the user face image based on the user face model;
determining a fusion mode of the map image and the user face image according to the type of the map image;
and according to the fusion mode, fusing the mapping image and the user face image in the real-time video frame so as to map the user face image.
Optionally, the computer executable instructions, when executed by the processor, further comprise:
after the user face image is subjected to mapping processing, a preset face deformation area in the user face image and a face deformation mode corresponding to the preset face deformation area are obtained;
selecting a target deformer for the preset face deformation area from a plurality of preset deformers according to the corresponding relation between the face deformation area and the deformer type;
and carrying out deformation processing on the image in the face deformation area according to the face deformation mode by utilizing the target deformer.
According to the method and the device, in a real-time video scene, after the first key point of the standard face image and the second key point of the user face image are obtained, the position conversion relation between the first key point and the second key point can be determined, the standard face model is adjusted based on the position conversion relation to obtain the user face model, and the user face image is subjected to mapping processing in the real-time video frame based on the user face model, so that the user face model can be simply and quickly obtained by adjusting the standard face model without respectively establishing the user face model for each frame of image, the calculation amount and time consumption required by image processing are reduced, and the real-time performance of video transmission is improved.
The computer-executable instructions in the storage medium of the embodiment of the present application, when executed by the processor, may perform the image processing method described above, and implement the processes and effects of the embodiment of the image processing method described above, which are not described herein again.
The computer-readable storage medium includes a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.
All the embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art to which the present application pertains. Any modification, equivalent replacement, improvement or the like made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.
Claims (10)
1. An image processing method, comprising:
acquiring a pre-established standard face image and acquiring a first key point of the standard face image;
acquiring a user face image in a real-time video frame, and acquiring a second key point of the user face image;
determining a position transformation relation between the first key point and the second key point, and adjusting a standard face model corresponding to the standard face image based on the position transformation relation to obtain a user face model corresponding to the user face image;
based on the user face model, performing mapping processing on the user face image in the real-time video frame;
the adjusting the standard face model corresponding to the standard face image based on the position transformation relation to obtain the user face model corresponding to the user face image includes:
marking the first keypoint on the standard face model;
moving the first keypoint on the standard face model based on a positional transformation relationship between the first keypoint and the second keypoint;
in the process of moving the first key point, adjusting the standard face model based on the position of the moved first key point;
and when the position coordinates of the first key point in the standard face image after the movement are the same as the position coordinates of the second key point corresponding to the first key point in the user face image, determining that the model adjustment is finished, and taking the adjusted standard face model as a user face model corresponding to the user face image.
2. The method of claim 1, wherein the obtaining a second keypoint of the image of the user's face comprises:
acquiring key points in the user face image according to the acquisition part of the first key point in the standard face image to obtain an initial key point;
if the number of the first key points is more than that of the initial key points, generating corresponding key points for the excessive first key points in the user facial image according to the corresponding relation between the first key points and the initial key points and the relative position relation between the first key points;
and using the initial key point and the generated key point together as a second key point of the face image of the user.
3. The method of claim 1, wherein determining the positional transformation relationship between the first keypoint and the second keypoint comprises:
determining the corresponding relation between the first key point and the second key point according to the acquisition part of the first key point in the standard facial image and the acquisition part of the second key point in the user facial image;
and determining the position transformation relation between the first key point and the second key point according to the determined corresponding relation, and the position coordinates of the first key point in the standard face image and the position coordinates of the second key point in the user face image.
4. The method of claim 1, wherein said mapping said user face image in said real-time video frame based on said user face model comprises:
drawing a chartlet image for the user face image based on the user face model;
determining a fusion mode of the map image and the user face image according to the type of the map image;
and according to the fusion mode, fusing the mapping image and the user face image in the real-time video frame so as to map the user face image.
5. The method according to any one of claims 1-4, further comprising:
after the user face image is subjected to mapping processing, a preset face deformation area in the user face image and a face deformation mode corresponding to the preset face deformation area are obtained;
selecting a target deformer for the preset face deformation area from a plurality of preset deformers according to the corresponding relation between the face deformation area and the deformer type;
and carrying out deformation processing on the image in the face deformation area according to the face deformation mode by utilizing the target deformer.
6. An image processing apparatus characterized by comprising:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring a pre-established standard face image and acquiring a first key point of the standard face image;
the second acquisition module is used for acquiring a user face image in a real-time video frame and acquiring a second key point of the user face image;
the model generating module is used for determining a position transformation relation between the first key point and the second key point, and adjusting a standard face model corresponding to the standard face image based on the position transformation relation to obtain a user face model corresponding to the user face image;
the mapping processing module is used for mapping the user face image in the real-time video frame based on the user face model;
the model generation module is specifically configured to:
marking the first keypoint on the standard face model;
moving the first keypoint on the standard face model based on a positional transformation relationship between the first keypoint and the second keypoint;
in the process of moving the first key point, adjusting the standard face model based on the position of the moved first key point;
and when the position coordinates of the first key point in the standard face image after the movement are the same as the position coordinates of the second key point corresponding to the first key point in the user face image, determining that the model adjustment is finished, and taking the adjusted standard face model as a user face model corresponding to the user face image.
7. The apparatus of claim 6, wherein the second obtaining module is specifically configured to:
acquiring key points in the user face image according to the acquisition part of the first key point in the standard face image to obtain an initial key point;
if the number of the first key points is more than that of the initial key points, generating corresponding key points for the excessive first key points in the user face image according to the corresponding relation between the first key points and the initial key points and the relative position relation between the first key points;
and using the initial key point and the generated key point together as a second key point of the face image of the user.
8. The apparatus of claim 6, wherein the model generation module is specifically configured to:
determining the corresponding relation between the first key point and the second key point according to the acquisition part of the first key point in the standard facial image and the acquisition part of the second key point in the user facial image;
and determining the position transformation relation between the first key point and the second key point according to the determined corresponding relation, and the position coordinates of the first key point in the standard face image and the position coordinates of the second key point in the user face image.
9. The apparatus of claim 6, wherein the map processing module is specifically configured to:
drawing a chartlet image for the user face image based on the user face model;
determining a fusion mode of the map image and the user face image according to the type of the map image;
and according to the fusion mode, fusing the mapping image and the user face image in the real-time video frame so as to map the user face image.
10. The apparatus of any one of claims 6-9, further comprising:
the third acquisition module is used for acquiring a preset face deformation area in the user face image and a face deformation mode corresponding to the preset face deformation area after the mapping processing is carried out on the user face image;
the deformer selection module is used for selecting a target deformer for the preset face deformation area from a plurality of preset deformers according to the corresponding relation between the face deformation area and the type of the deformer;
and the deformation processing module is used for carrying out deformation processing on the image in the face deformation area by utilizing the target deformer according to the face deformation mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811100506.4A CN109147012B (en) | 2018-09-20 | 2018-09-20 | Image processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811100506.4A CN109147012B (en) | 2018-09-20 | 2018-09-20 | Image processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109147012A CN109147012A (en) | 2019-01-04 |
| CN109147012B true CN109147012B (en) | 2023-04-14 |
Family
ID=64815361
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811100506.4A Active CN109147012B (en) | 2018-09-20 | 2018-09-20 | Image processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109147012B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111488759A (en) * | 2019-01-25 | 2020-08-04 | 北京字节跳动网络技术有限公司 | Image processing method and device for animal face |
| CN109934766B (en) * | 2019-03-06 | 2021-11-30 | 北京市商汤科技开发有限公司 | Image processing method and device |
| CN110490897A (en) * | 2019-07-30 | 2019-11-22 | 维沃移动通信有限公司 | Imitate the method and electronic equipment that video generates |
| CN110728620A (en) * | 2019-09-30 | 2020-01-24 | 北京市商汤科技开发有限公司 | Image processing method and device and electronic equipment |
| EP3971820A4 (en) | 2019-09-30 | 2022-08-10 | Beijing Sensetime Technology Development Co., Ltd. | Image processing method, apparatus and electronic device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104102409A (en) * | 2013-04-12 | 2014-10-15 | 三星电子（中国）研发中心 | Scenario adaptation device and method for user interface |
| CN106447604A (en) * | 2016-09-30 | 2017-02-22 | 北京奇虎科技有限公司 | Method and device for transforming facial frames in videos |
| CN107820027A (en) * | 2017-11-02 | 2018-03-20 | 北京奇虎科技有限公司 | Video personage dresss up method, apparatus, computing device and computer-readable storage medium |
| CN107911643A (en) * | 2017-11-30 | 2018-04-13 | 维沃移动通信有限公司 | Show the method and apparatus of scene special effect in a kind of video communication |
| CN107959798A (en) * | 2017-12-18 | 2018-04-24 | 北京奇虎科技有限公司 | Video data real-time processing method and device, computing device |
| CN107977644A (en) * | 2017-12-18 | 2018-05-01 | 北京奇虎科技有限公司 | Image processing method and device, computing device based on image capture device |
| CN108335345A (en) * | 2018-02-12 | 2018-07-27 | 北京奇虎科技有限公司 | The control method and device of FA Facial Animation model, computing device |
-
2018
- 2018-09-20 CN CN201811100506.4A patent/CN109147012B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104102409A (en) * | 2013-04-12 | 2014-10-15 | 三星电子（中国）研发中心 | Scenario adaptation device and method for user interface |
| CN106447604A (en) * | 2016-09-30 | 2017-02-22 | 北京奇虎科技有限公司 | Method and device for transforming facial frames in videos |
| CN107820027A (en) * | 2017-11-02 | 2018-03-20 | 北京奇虎科技有限公司 | Video personage dresss up method, apparatus, computing device and computer-readable storage medium |
| CN107911643A (en) * | 2017-11-30 | 2018-04-13 | 维沃移动通信有限公司 | Show the method and apparatus of scene special effect in a kind of video communication |
| CN107959798A (en) * | 2017-12-18 | 2018-04-24 | 北京奇虎科技有限公司 | Video data real-time processing method and device, computing device |
| CN107977644A (en) * | 2017-12-18 | 2018-05-01 | 北京奇虎科技有限公司 | Image processing method and device, computing device based on image capture device |
| CN108335345A (en) * | 2018-02-12 | 2018-07-27 | 北京奇虎科技有限公司 | The control method and device of FA Facial Animation model, computing device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109147012A (en) | 2019-01-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109147012B (en) | Image processing method and device | |
| CN108846793B (en) | Image processing method and terminal equipment based on image style conversion model | |
| US9639914B2 (en) | Portrait deformation method and apparatus | |
| CN111754415B (en) | Face image processing method and device, image equipment and storage medium | |
| WO2021169325A1 (en) | Gaze adjustment method and apparatus, and storage medium | |
| CN112989904A (en) | Method for generating style image, method, device, equipment and medium for training model | |
| CN107452049B (en) | Three-dimensional head modeling method and device | |
| JP2023514289A (en) | 3D face model construction method, 3D face model construction device, computer equipment, and computer program | |
| CN114219878B (en) | Animation generation method and device for virtual character, storage medium and terminal | |
| CN101968892A (en) | Method for automatically adjusting three-dimensional face model according to one face picture | |
| CN110838084A (en) | Image style transfer method and device, electronic equipment and storage medium | |
| CN111008927A (en) | Face replacement method, storage medium and terminal equipment | |
| CN111311733A (en) | Three-dimensional model processing method and device, processor, electronic device and storage medium | |
| CN110536095A (en) | Call method, device, terminal and storage medium | |
| CN113658035B (en) | Face transformation method, device, equipment, storage medium and product | |
| CN111652795A (en) | Face shape adjusting method, face shape adjusting device, live broadcast method, live broadcast device, electronic equipment and storage medium | |
| CN110910512A (en) | Virtual object self-adaptive adjusting method and device, computer equipment and storage medium | |
| CN108062742B (en) | Eyebrow replacing method by digital image processing and deformation | |
| CN111614925B (en) | Figure image processing method and device, corresponding terminal and storage medium | |
| US20220375258A1 (en) | Image processing method and apparatus, device and storage medium | |
| CN112581518A (en) | Eyeball registration method, device, server and medium based on three-dimensional cartoon model | |
| JP2007102478A (en) | Image processor, image processing method, and semiconductor integrated circuit | |
| CN115908712A (en) | Three-dimensional reconstruction and model training method and equipment based on image and storage medium | |
| CN113079383B (en) | Video processing method, device, electronic equipment and storage medium | |
| WO2021197230A1 (en) | Three-dimensional head model constructing method, device, system, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |