WO2023066120A1

WO2023066120A1 - Image processing method and apparatus, electronic device, and storage medium

Info

Publication number: WO2023066120A1
Application number: PCT/CN2022/125036
Authority: WO
Inventors: 杨瑞健; 陶然; 赵代平; 许亲亲
Original assignee: 上海商汤智能科技有限公司
Priority date: 2021-10-20
Filing date: 2022-10-13
Publication date: 2023-04-27
Also published as: CN113657357B; CN114581986A; CN114581987A; CN113657357A

Abstract

The present invention relates to an image processing method and apparatus, an electronic device, and a storage medium. The image processing method comprises: obtaining an image to be processed, wherein the image to be processed comprises a target object; extracting three-dimensional key points of the target object, and determining first correspondences between the three-dimensional key points and vertexes in a three-dimensional adjustment model; determining adjustment results of the three-dimensional key points according to an adjustment instruction, preset deformation data of the vertexes in the three-dimensional adjustment model, and the first correspondences; and determining a target image according to the adjustment results of the three-dimensional key points and the image to be processed.

Description

Image processing method, device, electronic device and storage medium

Related Application Cross Reference

This application claims the priority of the Chinese patent application with the application number 202111222848.5 and the filing date of October 20, 2021. The entire content of the Chinese patent application is hereby incorporated by reference into this application.

technical field

The present disclosure relates to the technical field of image processing, and in particular, to an image processing method, device, electronic equipment, and storage medium.

Background technique

With the continuous development of artificial intelligence technology, there are more and more types of image processing, and the effect is getting better and better. When people take pictures or record videos, they often can perform beauty treatment on the obtained facial images to make their faces more attractive. However, based on the identification and adjustment of the two-dimensional information of the facial image, the beauty treatment is completed, the sense of reality and naturalness is poor, and the effect is difficult to satisfy the user.

Contents of the invention

According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, including: acquiring an image to be processed, wherein the image to be processed includes a target object; extracting 3D key points of the target object, and determining the 3D A first correspondence between key points and vertices in the three-dimensional adjustment model; determining an adjustment result of the three-dimensional key points according to an adjustment instruction, preset deformation data of vertices in the three-dimensional adjustment model, and the first correspondence; A target image is determined according to the adjustment result of the three-dimensional key points and the image to be processed.

According to the second aspect of the embodiments of the present disclosure, there is provided an image processing device, including: an acquisition module, configured to acquire an image to be processed, wherein the image to be processed includes a target object; a first corresponding module, configured to extract the The three-dimensional key points of the target object, and determine the first corresponding relationship between the three-dimensional key points and the vertices in the three-dimensional adjustment model; the adjustment module is used to adjust according to the adjustment instruction, the preset deformation data of the vertices in the three-dimensional adjustment model and The first correspondence is used to determine an adjustment result of the 3D key point; a target module is configured to determine a target image according to the adjustment result of the 3D key point and the image to be processed.

According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, the device includes a memory and a processor, the memory is used to store computer instructions that can be run on the processor, and the processor is used to execute the The computer instructions implement the method described in the first aspect.

According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method described in the first aspect is implemented.

According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, the product includes a computer program, and when the program is executed by a processor, the method described in the first aspect is implemented.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure.

FIG. 1 is a flowchart of an image processing method according to an embodiment of the present disclosure;

Fig. 2 is a schematic diagram of an interactive interface shown according to an embodiment of the present disclosure;

Fig. 3 is a schematic diagram of a face standard model shown according to an embodiment of the present disclosure;

Fig. 4 is a flow chart of determining a first correspondence according to an embodiment of the present disclosure;

Fig. 5 is a flowchart showing a method of determining a target image according to an embodiment of the present disclosure;

FIG. 6 is a flowchart of an image processing method according to another embodiment of the present disclosure;

Fig. 7 is a schematic diagram of an integrated model shown according to an embodiment of the present disclosure;

Fig. 8 is a schematic structural diagram of an image processing device according to an embodiment of the present disclosure;

Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure.

Detailed ways

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present disclosure as recited in the appended claims.

The terminology used in the present disclosure is for the purpose of describing particular embodiments only, and is not intended to limit the present disclosure. As used in this disclosure and the appended claims, the singular forms "a", "the", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms "first", "second", "third", etc. may be used in the present disclosure to describe various information, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of the present disclosure, first information may also be called second information, and similarly, second information may also be called first information. Depending on the context, the word "if" as used herein may be interpreted as "at" or "when" or "in response to a determination."

If the disclosed technical solution involves user information, the product applying the disclosed technical solution has clearly informed the user of the information processing rules and obtained the user's independent consent before processing the user information. If the disclosed technical solution involves sensitive user information, the product applying the disclosed technical solution has obtained the separate consent of the user before processing the sensitive user information, and at the same time meets the requirement of "express consent". For example, at the user information collection device such as a camera, set a clear and prominent sign to inform that the user information has entered the collection range, and the user information will be collected. If the user voluntarily enters the collection range, it is deemed to agree to the collection of his user information; or On the user information processing device, when the user information processing rules are notified with obvious signs/information, the user authorization is obtained through pop-up information or asking the user to upload their user information; among them, the user information processing rules may include user Information processor, user information processing purpose, processing method, type of user information processed and other information.

In a first aspect, at least one embodiment of the present disclosure provides an image processing method. Please refer to FIG. 1 , which shows the flow of the method, including steps S101 to S104.

Wherein, the method can be used to process the image to be processed, for example, the target object in the image to be processed can be adjusted, and the target object can be, for example, at least one of the following objects in the image to be processed: face, hand, limb, adjusted The items can be shape, size, position, etc. For example, when there is a face in the image to be processed, the facial information of the face, that is, the shape, size, proportion, etc. of the facial features, can be adjusted, so as to complete the enhancement processing such as face beautification, micro-shaping, and body shaping.

The image to be processed may be an image captured by the image acquisition device, or a frame of image in a video recorded by the image acquisition device. It can be understood that, by using the method provided by the embodiment of the present disclosure to process each frame of image in the video recorded by the image acquisition device, the processing of the video can be completed.

In addition, the method can be executed by electronic devices such as terminal equipment or servers, and the terminal equipment can be user equipment (User Equipment, UE), mobile equipment, terminal, cellular phone, cordless phone, personal digital assistant (Personal Digital Assistant, PDA) handheld device, computing device, vehicle-mounted device, wearable device, etc., the method can be realized by calling a computer-readable instruction stored in a memory by a processor. Alternatively, the method may be executed by a server, and the server may be a local server, a cloud server, or the like.

In step S101, an image to be processed is acquired, wherein the image to be processed includes a target object.

Wherein, the image to be processed may be an image taken by an image acquisition device, or a frame image in a video recorded by the image acquisition device, wherein the image acquisition device may be an electronic device with an image acquisition function such as a mobile phone or a camera. The target object is the image of a specific object in the real scene in the image, for example, it can be the face in the image to be processed (that is, the image of the face in the image), the hand (that is, the hand in the image The resulting image), etc.

The image captured by the image acquisition device can be identified. If the image contains the target object, the image is acquired as the image to be processed. If the image does not contain the target object, the image is not regarded as the image to be processed; correspondingly, the The video recorded by the image acquisition device is identified. If a frame of image contains the target object, the frame of image is acquired as the image to be processed. If a frame of image does not contain the target object, the frame of image is not regarded as the image to be processed.

In a possible scenario, the user uses a mobile phone to record a video. During the recording process, a face sometimes appears in the screen, and sometimes there is no face. Image frames with no faces in the video frame are not treated as images to be processed for beautification processing, thereby ensuring real-time beautification processing of faces in the video screen, and reducing processor load when there are no faces in the video screen , save energy consumption, and avoid mishandling of video images.

In step S102, the 3D key points of the target object are extracted, and a first corresponding relationship between the 3D key points and vertices in the 3D adjustment model is determined.

There are many ways to extract the 3D key points of the target object. In one example, the depth image corresponding to the image to be processed can be obtained, and the image after the image to be processed is aligned with the depth image can be input into the pre-trained deep neural network. Perform 3D keypoint detection. While the image acquisition device acquires the image to be processed, it can start the structured light sensor of the camera of the image acquisition device to acquire the depth image, or use Time of Flight (ToF) sensing technology to acquire the depth image.

There are multiple detected 3D key points, and each detected 3D key point has an identifier. For example, if the detected 3D key points have a sequence, the identifier of each 3D key point may be a sequence identifier. For example, when detecting the 3D key points of the target object, the detection can be performed according to the characteristics of the target object or detection requirements, that is, whether to detect a certain position of the target object, and to detect several 3D key points can be preset in advance. For example, when detecting a face image, three-dimensional key point detection can be performed on facial features and other key positions. These key positions have signs such as eyes, mouth, eyebrows, nose, etc., so the detected three-dimensional key points is identified.

The three-dimensional adjustment model is used to represent the adjustment method of each adjustment item of the target object. For example, the three-dimensional adjustment model of the face may include a beauty model, which is used to represent various beauty items (such as thin face, big eyes, small mouth, etc.) way of beauty. The 3D adjustment model can be composed of multiple vertices and the mesh structure between the vertices. The vertices of the 3D adjustment model can also correspond to the 3D key points of the target object, and the multiple vertices of the 3D adjustment model are also identified. For example, the 3D adjustment Multiple vertices of the model have order, and the identification of each vertex can be order identification. The order of the multiple vertices may be the same as or different from the order of the three-dimensional key points detected above.

Since the detected 3D key points of the target object correspond to each key position of the target object, and the 3D adjustment model also corresponds to each key position of the target object, the relationship between the 3D key point and the vertices in the 3D adjustment model is established. The first correspondence is to correspond the 3D key points at the same key position with the vertices in the 3D adjustment model.

In step S103, the adjustment result of the three-dimensional key points is determined according to the adjustment instruction, the preset deformation data of vertices in the three-dimensional adjustment model, and the first corresponding relationship.

Wherein, the three-dimensional adjustment model may also store preset deformation data of each vertex, such as displacement data of the vertex when each adjustment item is performed. In each adjustment item, at least one vertex needs to be displaced, so the displacement data of each vertex under different adjustment items can be recorded (note: if the vertex does not need to be displaced, the displacement data is 0). For example, the preset deformation data can be recorded in the form of an array, and an array is created for each vertex, and the three-dimensional coordinates of the vertex are first recorded in the array, and then the displacement data of the vertex under each adjustment item is sequentially recorded. Based on the recording method of the above-mentioned array, after the first corresponding relationship is determined, the coordinate information of the 3D key points corresponding to the vertices in the 3D adjustment model can be converted into the coordinate system of the 3D adjustment model, and the converted 3D key points The coordinates of the vertex are added to the array of the vertex, so as to facilitate the subsequent calculation when adjusting the 3D key points.

Wherein, the adjustment instruction can be generated according to the user's operation, for example, if the user selects at least one adjustment item, an adjustment instruction for the at least one adjustment item is generated; the adjustment instruction can also be automatically generated by the terminal executing the method. When the target object is a face, the user may select at least one adjustment item for the face, so as to generate an adjustment instruction corresponding to the at least one adjustment item. For another example, when the target object is a face, at least one adjustment item may be set in advance, and then an adjustment instruction corresponding to the at least one adjustment item is automatically generated each time an image to be processed is acquired.

The adjustment process of the three-dimensional key points includes the position adjustment process of the three-dimensional key points, so the adjustment result of the three-dimensional key points includes the adjusted three-dimensional key points.

In an example, when determining the adjustment result of the three-dimensional key point in this step, the adjustment item of the target object may be determined according to the adjustment instruction, and the preset deformation data of the vertices in the three-dimensional adjustment model corresponding to the adjustment item may be obtained. The first deformation data. According to the first deformation data, the three-dimensional key point corresponding to at least one vertex in the three-dimensional adjustment model is displaced, so as to obtain the adjustment result of the three-dimensional key point.

The adjustment items include adjustment position and adjustment type. The adjustment position refers to the local position of the target object, such as a facial feature in the face, and the adjustment type refers to the expected adjustment result, such as reducing, increasing, and rounding. For example, when the target object is a face, when the adjustment item is big eyes, the adjustment position is eyes, and the adjustment type is enlarged; when the adjustment item is small mouth, the adjustment position is mouth, and the adjustment type is reduced, etc. Please refer to accompanying drawing 2, it has shown the interactive interface of a face adjustment, has adjustment items such as thin face, big eyes, small head, rhinoplasty and small mouth etc. in this interface, after the user selects at least one adjustment item among them, then generate Contains the adjustment command for this adjustment item.

If there is one adjustment item in the adjustment instruction, one adjustment item is determined, and if there are multiple adjustment items in the adjustment instruction, multiple adjustment items are determined.

The first deformation data includes displacement data of each vertex of the three-dimensional adjustment model under the adjustment item. Each vertex can be traversed to obtain the deformation data of each vertex corresponding to the adjustment item in the adjustment instruction, that is, the displacement data of all vertices under the adjustment item. For example, when the adjustment instruction includes, for example, the big eye item 202 in FIG. 2 , traverse each vertex corresponding to the adjustment item, and obtain the deformation data of the eye vertices (for example, the number of eye vertices of E1, E2, E3, and E4) in the three-dimensional adjustment model of the big eye project, and the three-dimensional adjustment Other vertices in the model, such as mouth vertices and nose vertices, do not have deformation data. When each vertex stores the displacement data under each adjustment item in the form of an array, the displacement data can be obtained according to the corresponding position of the adjustment item to the array. Take the eye vertex E1 in Figure 2 as an example, for example, the array of E1 is [0 ,1,0.2,0.1,0], the first to fifth data positions in the array are the displacement data 0 of E1 under the thin face item 201, the displacement data 1 of E1 under the big eye item 202, and the small head The displacement data of E1 under item 203 is 0.2, the displacement data of E1 under item 204 of rhinoplasty is 0.1, and the displacement data of E1 under item 205 of small mouth is 0, then when the adjustment item is big eyes, it can be directly obtained from the second part of the E1 array. The displacement data 0.2 corresponding to the big-eye item 202 is obtained from the position. Similarly, the displacement data corresponding to the big-eye item can be obtained from the second positions of the E2, E3, and E4 arrays respectively.

When there is only one adjustment item in the adjustment command, the deformation data corresponding to the adjustment item is obtained directly. For example, when the adjustment item is a big eye, the deformation data (displacement data) of vertices E1-E4 corresponding to the big eye item are directly obtained. . When there are multiple adjustment items in the adjustment instruction, the deformation data corresponding to each adjustment item may be acquired. Or, in addition to storing the displacement data of each adjustment item, the three-dimensional adjustment model also stores displacement data corresponding to various combinations of adjustment items, so when there are multiple adjustment items in the adjustment instruction, multiple adjustment items can be directly obtained. The displacement data corresponding to the combination of adjustment items. For example, when the beauty item in the beauty command is only face-lifting, the deformation data corresponding to the face-lifting item 201 is obtained directly; The deformation data corresponding to the deformation data of the large-eye item 202 and the deformation data corresponding to the combination of the thin-face item 201 and the large-eye item 202 can also be obtained.

In addition, while the adjustment command includes the adjustment items, it can also include the adjustment parameter values of each adjustment item. Please continue to refer to Figure 2. There is a scale for adjusting parameters in this interface. Toggle the position of the anchor point of the scale to Adjust the adjustment parameter value of the adjustment item in the adjustment instruction. The adjustment parameter value can be the adjustment degree of the adjustment item. Different adjustment degrees can correspond to different deformation data. For example, when the adjustment degree is 100%, the deformation data is the maximum deformation data corresponding to the adjustment item, and the adjustment degree is 60%. , the deformation data is 60% of the maximum deformation data corresponding to the adjustment item. Therefore, before the 3D key points corresponding to the vertices in the 3D adjustment model are displaced according to the first deformation data to obtain the adjustment result of the 3D key points, the adjustment of the adjustment items can be further determined according to the adjustment instruction. parameter value, and then adjust the first deformation data according to the adjusted parameter value. The degree of the adjustment item can be adjusted by adjusting the parameter value, which increases the flexibility and diversity of the adjustment.

The coordinates of the three-dimensional key points corresponding to the vertices of the three-dimensional adjustment model and the displacement data corresponding to the vertices of the three-dimensional adjustment model can be calculated to obtain the adjustment result of the three-dimensional key points. Under one or more adjustment items, if the displacement data of at least one vertex is not 0, the 3D key points corresponding to these vertices will be displaced after the operation. The preset deformation data is recorded in the form of an array, and after the first corresponding relationship is determined, the coordinate information of the 3D key point corresponding to the vertex is converted into the coordinate system of the 3D adjustment model, and the converted 3D When the coordinate information of the key point is added to the array of the vertex, the coordinate and displacement data of the three-dimensional key point can be directly extracted from the array of each vertex.

In a possible implementation, when a plurality of non-zero displacement data for a certain vertex is obtained according to the adjustment instruction, according to the preset sequence, each displacement data can be used in sequence according to the three-dimensional key point corresponding to the vertex. Coordinates are calculated. For example, if the adjustment command includes small head item, nose augmentation item and big eye item, then the displacement data of the eye vertex E1 is [0.2,0.1,1], then you can follow the order in the adjustment command such as small head item, nose augmentation The displacement data of E1 corresponding to the project and the big-eye project can be used to calculate the coordinates of the three-dimensional key points corresponding to E1; or the displacement data corresponding to the big-eye project, small-head project and nose augmentation project can also be calculated according to the order in the E1 array. The coordinates of the 3D key points corresponding to E1 are calculated.

After the three-dimensional key points are displaced according to at least one of the above methods, the positions of the three-dimensional key points are adjusted, and the three-dimensional key points at the new positions are the adjusted three-dimensional key points in the adjustment result of the three-dimensional key points.

In step S104, a target image is determined according to the adjustment result of the 3D key points and the image to be processed.

In this step, according to the adjustment result of the three-dimensional key points, the same adjustment is performed on the corresponding position of the image to be processed, that is, the adjustment result is mapped to the image to be processed, thereby completing the adjustment of the target object in the image to be processed. For example, when the target object is a facial image, the beautification process on the facial image can be completed in this step.

According to the above-mentioned embodiment, by acquiring the image to be processed, and extracting the three-dimensional key points of the target object in the image to be processed, determining the first corresponding relationship between the three-dimensional key points and the vertices in the three-dimensional adjustment model, according to the adjustment instruction, the vertices in the three-dimensional adjustment model The preset deformation data and the first corresponding relationship determine the adjustment result of the three-dimensional key point, and determine the target image according to the adjustment result of the three-dimensional key point and the image to be processed. Since the 3D adjustment model is 3D, and the vertices of the 3D adjustment model have preset deformation data, the adjustment of the target object is to adjust the 3D key points according to the adjustment instruction and the preset deformation data, so the adjustment process is more precise and controllable , and the three-dimensional key points are extracted from the image to be processed, and finally the adjustment result obtained by adjusting the three-dimensional key points is returned to the image to be processed, so the adjustment effect of the target object in the obtained target image is natural and real, which improves the user satisfaction.

In some embodiments of the present disclosure, the image processing method further includes: acquiring a second corresponding relationship between vertices in the three-dimensional adjustment model and vertices in the three-dimensional standard model.

Wherein, the three-dimensional standard model is the standard model of the target object, which is composed of multiple vertices and meshes between the vertices. For example, when the target object is a facial image, the three-dimensional standard model may be a standard facial model as shown in FIG. 3 . If the 3D key points are extracted from the 3D standard model, all vertices of the 3D standard model can be obtained. The three-dimensional adjustment model is used to characterize the specific adjustment manner of at least one adjustment item of the three-dimensional standard model, that is, the three-dimensional adjustment model includes deformation data of each vertex of the three-dimensional standard model in at least one adjustment item. The 3D adjustment model has a dedicated model coordinate system. The coordinates of each vertex in the 3D adjustment model are the coordinates in the coordinate system of the 3D adjustment model. The 3D adjustment model can also use the coordinate transformation matrix to coordinate the coordinates of the 3D adjustment model in the coordinate system Transform to the world coordinate system. In some embodiments of the present disclosure, the three-dimensional adjustment model may be obtained by performing preset deformations respectively corresponding to at least one adjustment item on the three-dimensional standard model.

Therefore, the coordinates of the vertices in the three-dimensional adjustment model and the coordinates of the vertices in the three-dimensional standard model can be converted into the same coordinate system, for example, the coordinates of the vertices in the three-dimensional standard model are coordinates in the world coordinate system, then The coordinate transformation matrix of the three-dimensional adjustment model can be multiplied by the coordinates of the vertices in the three-dimensional adjustment model, so as to convert the coordinates of the vertices of the three-dimensional adjustment model to the world coordinate system. In this way, the coordinates of the vertices in the converted three-dimensional adjustment model and the three-dimensional standard model The coordinates of vertices are all in the world coordinate system. According to the Euclidean distance between each vertex in the converted three-dimensional adjustment model and each vertex in the three-dimensional standard model, determine the second corresponding relationship, which can be based on the Euclidean distance between two vertices and the preset The distance threshold is to determine whether the two vertices are corresponding, that is, if the Euclidean distance is less than the distance threshold, then the two points are corresponding, otherwise they are not.

Based on this, the first corresponding relationship between the 3D key points and the vertices in the 3D adjustment model may be determined as shown in FIG. 4 , including steps S401 to S402.

In step S401, according to the identification of the 3D key point and the identification of the vertex in the 3D standard model, a third corresponding relationship between the 3D key point and the vertex in the 3D standard model is determined.

Since the 3D standard model is a standard model generated based on a standard object (such as a standard face image), the setting logic of the key points of the standard object and the target object in the 3D standard model is the same, for example, they all include eyebrows, eyes, nose, mouth , therefore, when the position of the vertex in the target object in the 3D standard model generated based on the standard face image is the same as the position of the 3D key point in the target object, then the vertex in the 3D standard model and the identity of the 3D key point are the same , so the same three-dimensional key point and the vertices of the three-dimensional standard model are determined as pairs of vertices corresponding to each other, and the third corresponding relationship is obtained. For example, the order of the vertices of the three-dimensional standard model is consistent with the order of the three-dimensional key points of the target object, and the order of the same three-dimensional key points and the vertices of the three-dimensional standard model can be determined as a pair of vertices corresponding to each other, and the described third correspondence.

In step S402, the first correspondence is determined according to the second correspondence and the third correspondence.

The vertices of the 3D adjustment model corresponding to the same vertex of the 3D standard model and the 3D key points are determined as corresponding to each other.

In this embodiment, by presetting the second corresponding relationship between each vertex of the 3D standard model and each vertex of the 3D adjustment model, the 3D key is determined according to the third corresponding relationship between the 3D key point and each vertex of the 3D standard model. The first corresponding relationship between the point and each vertex of the three-dimensional adjustment model. In other embodiments, after obtaining the 3D key points of the target object of the image to be processed, the identification can also be directly used to determine the first corresponding relationship between the 3D key points and the vertices of the 3D adjustment model, which is simple, convenient, accurate and reliable, and avoids the For the 3D key points of the target object of each image to be processed, complex operations such as Euclidean distance calculation are performed.

In some embodiments of the present disclosure, the target image may be determined according to the adjustment result of the three-dimensional key points and the image to be processed as shown in FIG. 5 , including steps S501 to S502.

In step S501, a first grid is constructed between the adjustment results of the three-dimensional key points according to a pre-established first topology structure.

Wherein, the pre-established first topology refers to the connection relationship of the vertices in the grid among the vertices of the 3D standard model, for example, which other vertices need to be connected to each vertex of the 3D standard model. The first topological structure is recorded with the identification of vertices. And because the identification of the three-dimensional key point corresponds to the identification of the vertices of the three-dimensional standard model one by one, and the third corresponding relationship between the three-dimensional key point and the vertices of the three-dimensional standard model is established in step S401, when at least one three-dimensional key point is displaced Finally, the identification of each 3D key point is still clear, and the corresponding relationship between each 3D key point and each vertex of the 3D standard model is still clear, and then the adjustment results of different 3D key points can be connected according to the first topology structure to construct the second a grid.

After the first mesh is constructed between the adjustment results of the various 3D key points, a target model of the target object is formed.

In step S502, according to the pixel information of the projection point of the 3D key point on the image to be processed, the coordinate information of the projection point of the adjustment result of the 3D key point on the image to be processed, and the first A grid, rendering the adjustment result of the target object on the image to be processed to obtain a target image.

The pixel information of the projection point of the three-dimensional key point on the image to be processed can be extracted as the pixel information of the three-dimensional key point, wherein the pixel information includes, for example, RGB values, etc. In addition, the three-dimensional key point is on the image to be processed The pixel information of the projection point of the three-dimensional key point is unchanged before and after the three-dimensional key point adjustment, so the pixel information of the projection point of the three-dimensional key point on the image to be processed can also be used as the pixel of the three-dimensional key point adjustment result information; according to the pixel information of the adjustment result of the three-dimensional key point, render the first grid to obtain the pixel information in the first grid; according to the adjustment result of the three-dimensional key point in the image to be processed The coordinate information of the projected point on the above, and the pixel information of the three-dimensional key point and the pixel information in the first grid are projected onto the image to be processed to obtain the target image. For example, if the target object is a facial image, then the target image is an image that has undergone beautification processing of the facial image.

Compared with the image to be processed, certain positions of the target object in the target image are adjusted, for example, the eyes in the face in the image to be processed become larger, the mouth becomes smaller, etc. These adjustments change the pixels in the image, such as eyes When it becomes larger, the pixel points of the eye part are displaced, the number increases, and the shape composed of all the pixel points of the eye changes. However, in this disclosure, the pixel points are not directly adjusted on the two-dimensional image to be processed, but the pixel points on the image to be processed are adjusted by adjusting the three-dimensional key points extracted from the two-dimensional image to be processed. . That is to say, the 3D key point corresponds to its projection point on the image to be processed, so the adjustment result of the 3D key point can be used to represent the adjustment result of the 3D key point corresponding to the projection point on the image to be processed, that is, the projection point of the projection point The position adjustment of the pixel point can be, for example, extracting the pixel information of the projection point of the three-dimensional key point on the image to be processed, and re-projecting the pixel information to the adjustment result of the three-dimensional key point on the projection point of the image to be processed.

The first grid includes a plurality of closed sub-grids, and these sub-grids are surrounded by at least three three-dimensional key points. When rendering the pixel information in the first grid, for example, when rendering the pixel information inside the first grid formed by three 3D key points, the pixel information in each sub-grid can be rendered as the Pixel information of three 3D keypoints. This is because, when extracting three-dimensional key points from the image to be processed, the points on the boundary of the pixel area with the same pixel information can be extracted, so the pixel points in such a pixel area are the same because they have the same The pixel information, and then the pixel information in the sub-grid composed of the three-dimensional key points corresponding to the points on the boundary of the pixel area is also the same as the pixel information of these three-dimensional key points.

In an example, the image to be processed can be rendered on the output image first, for example, through an image copy operation of a rendering API such as OpenGL, the image to be processed can be copied to the output image, that is, as the background of the target image, wherein the output image includes The pixel information of the three-dimensional key points and the projection of the pixel information in the first grid on the two-dimensional plane; the image to be processed is used as the input image of the rendering process, and the coordinates of the projection points of each three-dimensional key point on the image to be processed are used as Input the texture sampling coordinates of the image, so that the pixel information of the projection point of the 3D key point on the image to be processed can be extracted. Since the pixel information of the 3D key point remains unchanged before and after adjustment, the projection point of the 3D key point on the image to be processed The pixel information of the 3D key point can be used as the pixel information of the adjustment result of the 3D key point; through the model transformation matrix of the 3D adjustment model, the coordinates of the adjustment result of the 3D key point are converted to the world coordinate system, and then through the transformation matrix from the world coordinate system to the clipping space, Convert the coordinates of the adjustment results of the 3D key points in the world coordinate system to the clipping coordinate system, and send them to the subsequent fragment rendering stage, that is, render the pixel information in the first grid according to the pixel information of the adjustment results of the 3D key points, And rendering the adjustment result of the three-dimensional key point and the pixel information in the first grid on the output image, so as to render the adjustment result of the target object on the image to be processed, and obtain the target image.

In this embodiment, the first grid is constructed between the adjustment results of each three-dimensional key point to form the target model of the target object; and then the pixel information of the adjustment result of the three-dimensional key point is extracted, and the first grid The rendering of pixel information is to paste pixel information on the target model, and finally render the target model with pixel information to the output image to obtain the final target image.

In some embodiments of the present disclosure, the image may also be processed in the manner shown in FIG. 6 , including steps S601 to S602.

In step S601, generate a 3D extended model surrounding the 3D key point, wherein the vertices in the 3D extended model include the adjustment result of the first edge point and the second edge point obtained based on the extension of the first edge point , the first edge point is a three-dimensional key point corresponding to the boundary of the target object.

In this step, the first edge point corresponding to the boundary of the target object among the three-dimensional key points may be obtained first.

In some embodiments, the projection point of the three-dimensional key point in the image to be processed is obtained; according to the projection point of the target object and the three-dimensional key point in the image to be processed, determine the target object A projection point on the boundary: determining a 3D key point corresponding to the projection point on the boundary of the target object as the first edge point.

Exemplarily, when determining the projection point on the boundary of the target object, a plurality of straight lines can be set in the preset direction in the image to be processed, for example, a plurality of horizontal straight lines can be set in advance, and then projected on the basis of the three-dimensional key points The points on the plurality of straight lines are determined to determine the projected points on the boundary of the target object.

In this step, it may be determined on the extension line of the line between the central point of all the three-dimensional key points (for example, the central point of the target object, for example, located on the nose) and the first edge point, and determine the connection with the first edge point. A point at which the edge points are separated by a preset distance is a second edge point corresponding to the first edge point.

In some embodiments, the calculation may be performed according to the coordinate information of each 3D key point, so as to obtain the center point among the multiple 3D key points, that is, the 3D key point located at the center of the target object. Then construct a straight line between the center point and each edge point of the target object, and further extend the preset distance outward on the straight line (the distance can be preset in advance and can be set according to the size of the target object, for example, for the target object 1/10 of the width), then the vertex of the extension line is the second edge point corresponding to the edge point.

In this step, a second grid may be constructed between the adjustment result of the first edge point and the second edge point according to the second pre-established topological structure, so as to obtain the three-dimensional extended model.

The pre-established second topology refers to the connection relationship of points in the grid between multiple first edge points and multiple second edge points, for example, which other points each point needs to be connected to. For example, all the first edge points are connected sequentially to form the inner circle of the 3D extended model; all the second edge points are connected sequentially to form the outer circle of the 3D extended model; then each first edge point and its corresponding second The edge points are connected with a line, and the first edge point and the second edge point next to the corresponding second edge point are also connected with a line, thereby forming a second grid in the three-dimensional extended model. The second topology is recorded in the order of the first edge point and the second edge point, so after the first edge point is adjusted, the second grid can still be constructed according to the topology.

The three-dimensional extended model is a model that surrounds the three-dimensional key points located on the contour of the target object. The outer circle of the model (ie, the circle surrounded by the second edge point) is not adjusted, and the inner circle of the model (ie, The circle surrounded by the first edge point) is adjusted, because the first edge point is adjusted.

It has been mentioned in the above-mentioned embodiments that after the first grid is constructed between the adjustment results of the various three-dimensional key points, the target model of the target object is formed. After the second grid is constructed between the adjustment result of the first edge point and the second edge point, a 3D extended model is added around the target model, and the target model and the 3D extended model pass through the first edge point For the convenience of description, the target model and the 3D extended model connected as one can be referred to as an integrated model in the following. Please refer to FIG. 7 , which shows a schematic diagram of an integrated model when the target object is a face.

In step S602, the 3D extended model is rendered on the image to be processed according to the 3D extended model and the image to be processed.

The pixel information of the projection point of the first edge point on the image to be processed can be extracted. Since the pixel information of the first edge point is unchanged before and after the adjustment of the first edge point, the first edge point is The pixel information of the projection point on the image to be processed may also be used as the pixel information of the adjustment result of the first edge point; the pixel information of the projection point of the second edge point on the image to be processed is extracted as The pixel information of the second edge point; according to the pixel information of the adjustment result of the first edge point and the pixel information of the second edge point, render the second grid to obtain the second grid pixel information; according to the adjustment result of the first edge point and the coordinate information of the projection point of the second edge point on the image to be processed, the pixel information of the adjustment result of the first edge point, the The pixel information of the second edge point and the pixel information in the second grid are projected onto the image to be processed.

The above operation is basically the same as the operation in step S502, so step S602 can be executed with reference to the operation in step S502. In other words, before performing step S502, a three-dimensional extended model can be constructed around the target model according to the method provided in this embodiment to obtain a comprehensive model, and then the comprehensive model can be operated according to the operation method introduced in step S502, so that the The target object and surrounding images are projected onto the image to be processed to obtain the target image.

In this embodiment, by determining the second edge points surrounding the target model composed of three-dimensional key points, and further determining the three-dimensional extension model, the three-dimensional extension The adjusted results of the model are projected onto the image to be processed. Since the outer ring of the 3D extended model has not been adjusted, the connection between the area corresponding to the 3D extended model and the surrounding area in the target image is relatively natural, and there is no abrupt connection; and although the inner ring of the 3D extended model has been adjusted Yes, but the pixel information in the second grid in the 3D extended model is rendered, so the connection between the area corresponding to the 3D extended model in the target image and the area corresponding to the target object is relatively natural, and there will be no abrupt connection . In other words, the outside of the integrated model including the target model and the 3D extended model has not been adjusted, so the edge area of the entire integrated model on the target image is the same as the corresponding area on the image to be processed; while the inside of the integrated model (target model) After adjustment, the adjusted target model can overcome the uncoordinated and unnatural problems caused by the adjustment because the pixel information of the pixels in each sub-grid is the same during the rendering process, so this part (comprehensive model The corresponding region on the target object is also natural.

Taking the fine-tuning processing of the face as an example, a complete process of the image processing method provided by the present disclosure will be introduced below.

First import a 3D standard model (for example, a standard face 3D model), and use the method of calculating the Euclidean distance to establish the second corresponding relationship between the vertices in the prefabricated 3D adjustment model and the vertices in the 3D standard model, and store the second corresponding relationship In the lookup table, the vertices in the three-dimensional adjustment model and the vertices in the three-dimensional standard model all include, for example, the identification of the reflection order, and each vertex of the three-dimensional adjustment model also records an array representing preset deformation data corresponding to different adjustment items; extraction to be processed The three-dimensional key points of the face in the image, wherein each three-dimensional key point includes, for example, an identification reflecting the order, and a third corresponding relationship is established between the three-dimensional key points with the same identification and the vertices in the three-dimensional standard model, according to the second corresponding relationship and the third The corresponding relationship is to establish a first corresponding relationship between the 3D key point and the vertex in the 3D adjustment model, and update the identification of the 3D key point to the array of the corresponding vertex in the 3D adjustment model according to the first corresponding relationship.

Next, determine the projection point of each 3D key point on the image to be processed, and then construct a horizontal line on the face of the image to be processed, and determine the one located on the boundary of the face among the multiple projection points on each horizontal line Projection points, and determine the 3D key point corresponding to the projection point on the boundary as the first edge point, calculate the center point of all 3D key points, and extend the connection line between the center point and the first edge point by one unit length The second edge point is obtained, and then a second grid is constructed between the first edge point and the second edge point to form a three-dimensional extended model.

At least one 3D key point is moved according to currently set adjustment items (such as big eyes, thin face, etc.) and adjustment parameter values (ie, adjustment degree), that is, deformation data of vertices in the 3D adjustment model. In some embodiments, the array of each vertex in the three-dimensional adjustment model can be traversed, and the deformation data of the vertex under the adjustment item is multiplied by the adjustment parameter value to obtain the displacement data of the vertex, and the displacement data of the vertex is used to compare The three-dimensional key points corresponding to the array are displaced, and the displacement results of the three-dimensional key points are obtained. Then construct the first grid between the displacement results of the three-dimensional key points, and connect the first grid and the second grid as a whole through the first edge point to obtain a comprehensive model.

Finally, the image to be processed is pasted on the integrated model, and then the integrated model is rendered on the image to be processed to obtain the target image for presenting the face adjustment effect. In some embodiments, the image to be processed can be rendered on the output image first, for example, the image to be processed can be copied to the output image through an image copy operation of a rendering API such as OpenGL; then the image to be processed can be used as the input image of the rendering process , the coordinates of the projection points of each 3D key point on the image to be processed are taken as the texture sampling coordinates of the input image; the coordinates of the adjustment results of the 3D key points are converted to the world coordinate system through the model transformation matrix of the 3D adjustment model, and the world The transformation matrix from the coordinate system to the clipping space converts the coordinates of the adjustment results of the 3D key points in the world coordinate system to the clipping coordinate system, and sends them to the subsequent fragment rendering stage to realize the presentation of the real-time 3D adjustment effect.

According to the second aspect of the embodiments of the present disclosure, an image processing device is provided. Please refer to FIG. The processed image includes the target object; the first correspondence module 802 is used to extract the 3D key points of the target object, and determine the first correspondence between the 3D key points and the vertices in the 3D adjustment model; the adjustment module 803 uses Determine the adjustment result of the 3D key point according to the adjustment instruction, the preset deformation data of the vertices in the 3D adjustment model and the first corresponding relationship; the target module 804 is used to determine the adjustment result of the 3D key point according to the adjustment result and the image to be processed to determine a target image.

In some embodiments of the present disclosure, a second corresponding module is also included, configured to: obtain a second corresponding relationship between the vertices in the 3D adjustment model and the vertices in the 3D standard model; the first corresponding module is configured to: According to the identification of the 3D key point and the identification of the vertex in the 3D standard model, determine the third corresponding relationship between the 3D key point and the vertex in the 3D standard model; according to the second corresponding relationship and the vertex in the 3D standard model The third corresponding relationship is determined, and the first corresponding relationship is determined.

In some embodiments of the present disclosure, the second corresponding module is used to: convert the coordinates of vertices in the three-dimensional adjustment model and the coordinates of vertices in the three-dimensional standard model into the same coordinate system; according to the The Euclidean distance between each vertex in the three-dimensional adjustment model and each vertex in the three-dimensional standard model is used to determine the second corresponding relationship.

In some embodiments of the present disclosure, the first corresponding module is configured to determine the 3D key point and the vertex in the 3D standard model according to the identification of the 3D key point and the identification of the vertex in the 3D standard model For the third corresponding relationship, it is used to: determine the same 3D key point and the vertices of the 3D standard model as a pair of vertices corresponding to each other, so as to obtain the third corresponding relationship.

In some embodiments of the present disclosure, the adjustment module is configured to: determine the adjustment item of the target object according to the adjustment instruction; obtain the preset deformation data of the vertex in the three-dimensional adjustment model, which corresponds to the adjustment item First deformation data: Displacing the 3D key point corresponding to at least one vertex in the 3D adjustment model according to the first deformation data, to obtain an adjustment result of the 3D key point.

In some embodiments of the present disclosure, the adjustment module is further configured to: determine the adjustment parameter value of the adjustment item according to the adjustment instruction before obtaining the adjustment result of the three-dimensional key point; The first deformation data is adjusted.

In some embodiments of the present disclosure, the target module is configured to: construct a first grid between the adjustment results of the 3D key points according to a pre-established first topology structure; The pixel information of the projection point on the image to be processed, the coordinate information of the projection point of the adjustment result of the three-dimensional key point on the image to be processed, and the first grid, and render the image on the image to be processed The adjustment result of the target object is obtained to obtain the target image.

In some embodiments of the present disclosure, the target module is further configured to: extract the pixel information of the projection point of the 3D key point on the image to be processed, as the pixel information of the adjustment result of the 3D key point; Rendering the first grid according to the pixel information of the adjustment result of the three-dimensional key point to obtain pixel information in the first grid; according to the adjustment result of the three-dimensional key point on the image to be processed Projecting the coordinate information of the point, projecting the pixel information of the three-dimensional key point and the pixel information in the first grid onto the image to be processed to obtain the target image.

In some embodiments of the present disclosure, an extension module is also included, configured to: generate a 3D extended model surrounding the 3D key point, wherein the vertices in the 3D extended model include the adjustment result of the first edge point and based on the The second edge point obtained by extending the first edge point, the first edge point is a three-dimensional key point corresponding to the boundary of the target object; according to the three-dimensional extended model and the image to be processed, in the to-be-processed The three-dimensional extended model is rendered on the image.

In some embodiments of the present disclosure, the expansion module is further configured to: obtain the first edge point; on an extension line of a line connecting the center point of the three-dimensional key point and the first edge point, Determining a point at a preset distance from the first edge point as the second edge point; according to a pre-established second topology, between the adjustment result of the first edge point and the second edge point Construct the second grid to obtain the 3D extended model.

In some embodiments of the present disclosure, the extension module is further configured to: obtain the projection point of the 3D key point in the image to be processed; according to the target object and the 3D key point in the image to be processed The projection point in the image is determined as the projection point on the boundary of the target object; and the three-dimensional key point corresponding to the projection point on the boundary of the target object is determined as the first edge point.

In some embodiments of the present disclosure, the expansion module is further configured to: set a plurality of straight lines in preset directions in the image to be processed; points, determine the projected points that lie on the boundary of the target object.

In some embodiments of the present disclosure, the extension module is further configured to: extract pixel information of a projection point of the first edge point on the image to be processed, as a pixel of the adjustment result of the first edge point information, and extract the pixel information of the projection point of the second edge point on the image to be processed as the pixel information of the second edge point; according to the pixel information of the adjustment result of the first edge point and the pixel information of the second edge point, render the second grid, and obtain pixel information in the second grid; according to the adjustment result of the first edge point and the second edge point in the Processing the coordinate information of the projection point on the image, projecting the pixel information of the adjustment result of the first edge point, the pixel information of the second edge point, and the pixel information in the second grid to the waiting processing image.

In some embodiments of the present disclosure, the target object includes at least one of the following objects in the image to be processed: face, hand, and limb.

Regarding the apparatus in the above embodiments, the specific manner in which each module executes operations has been described in detail in the embodiment of the method in the first aspect, and will not be described in detail here.

In a third aspect, at least one embodiment of the present disclosure provides a device. Please refer to FIG. 9 , which shows the structure of the device. The device includes a memory 904, a processor 901, and a network interface 903. Coupled with 902, the memory is used to store computer instructions that can be run on a processor, and the processor is used to process the image based on the method described in any one of the first aspect when executing the computer instructions.

In a fourth aspect, at least one embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method described in any one of the first aspect is implemented.

In a fifth aspect, at least one embodiment of the present disclosure provides a computer program product, the product includes a computer program, and when the computer program is executed by a processor, the method described in any one of the first aspect is implemented

This disclosure relates to the field of augmented reality. By acquiring the image information of the target object in the real environment, and then using various visual correlation algorithms to detect or identify the relevant features, states and attributes of the target object, and thus obtain the image information that matches the specific application. AR effect combining virtual and reality. Exemplarily, the target object may involve faces, limbs, gestures, actions, etc. related to the human body, or markers and markers related to objects, or sand tables, display areas or display items related to venues or places. Vision-related algorithms can involve visual positioning, SLAM, 3D reconstruction, image registration, background segmentation, object key point extraction and tracking, object pose or depth detection, etc. Specific applications can not only involve interactive scenes such as guided tours, navigation, explanations, reconstructions, virtual effect overlays and display related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, body beautification, special effect display, virtual Interactive scenarios such as model display. The relevant features, states and attributes of the target object can be detected or identified through the convolutional neural network. The above-mentioned convolutional neural network is a network model obtained by performing model training based on a deep learning framework.

In the present disclosure, the terms "first" and "second" are used for descriptive purposes only, and should not be understood as indicating or implying relative importance. The term "plurality" means two or more, unless otherwise clearly defined.

Other embodiments of the disclosure will be readily apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. The present disclosure is intended to cover any modification, use or adaptation of the present disclosure. These modifications, uses or adaptations follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. . The specification and examples are to be considered exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It should be understood that the present disclosure is not limited to the precise constructions which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

An image processing method, comprising:

acquiring an image to be processed, wherein the image to be processed includes a target object;

extracting 3D key points of the target object, and determining a first correspondence between the 3D key points and vertices in the 3D adjustment model;

determining the adjustment result of the three-dimensional key points according to the adjustment instruction, the preset deformation data of the vertices in the three-dimensional adjustment model, and the first corresponding relationship;

A target image is determined according to the adjustment result of the three-dimensional key points and the image to be processed.
The image processing method according to claim 1, further comprising:

Acquiring a second corresponding relationship between vertices in the three-dimensional adjustment model and vertices in the three-dimensional standard model;

Determining the first corresponding relationship between the 3D key points and the vertices in the 3D adjustment model includes:

determining a third correspondence between the three-dimensional key points and the vertices in the three-dimensional standard model according to the identification of the three-dimensional key points and the identification of vertices in the three-dimensional standard model;

Determine the first correspondence according to the second correspondence and the third correspondence.
The image processing method according to claim 2, wherein obtaining the second corresponding relationship between the vertices in the three-dimensional adjustment model and the vertices in the three-dimensional standard model comprises:

converting the coordinates of the vertices in the three-dimensional adjustment model and the coordinates of the vertices in the three-dimensional standard model into the same coordinate system;

The second corresponding relationship is determined according to the Euclidean distance between each vertex in the three-dimensional adjustment model and each vertex in the three-dimensional standard model.
The image processing method according to claim 2, wherein, according to the identification of the 3D key point and the identification of the vertex in the 3D standard model, determine the first position between the 3D key point and the vertex in the 3D standard model Three correspondences, including:

Determining the same 3D key point and vertices of the 3D standard model as a pair of vertices corresponding to each other to obtain the third corresponding relationship.
The image processing method according to any one of claims 1 to 4, wherein the three-dimensional key is determined according to the adjustment instruction, the preset deformation data of vertices in the three-dimensional adjustment model and the first correspondence Point adjustment results, including:

determining adjustment items of the target object according to the adjustment instruction;

Obtaining the first deformation data corresponding to the adjustment item among the preset deformation data of vertices in the three-dimensional adjustment model;

According to the first deformation data, a 3D key point corresponding to at least one vertex in the 3D adjustment model is displaced to obtain an adjustment result of the 3D key point.
The image processing method according to claim 5, wherein, before obtaining the adjustment result of the three-dimensional key points, further comprising:

determining an adjustment parameter value of the adjustment item according to the adjustment instruction;

The first deformation data is adjusted according to the adjustment parameter value.
The image processing method according to any one of claims 1 to 4, wherein, according to the adjustment result of the three-dimensional key points and the image to be processed, determining the target image includes:

Constructing a first grid between the adjustment results of the 3D key points according to a pre-established first topology structure;

According to the pixel information of the projection point of the 3D key point on the image to be processed, the coordinate information of the projection point of the adjustment result of the 3D key point on the image to be processed, and the first grid, in The adjustment result of the target object is rendered on the image to be processed to obtain a target image.
The image processing method according to claim 7, wherein, according to the pixel information of the projection point of the three-dimensional key point on the image to be processed, the projection of the adjustment result of the three-dimensional key point on the image to be processed Coordinate information of the point and the first grid, rendering the adjustment result of the target object on the image to be processed, and obtaining the target image, including:

Extracting the pixel information of the projection point of the three-dimensional key point on the image to be processed, as the pixel information of the adjustment result of the three-dimensional key point;

Rendering the first grid according to the pixel information of the adjustment result of the three-dimensional key points to obtain the pixel information in the first grid;

Projecting the pixel information of the 3D key point and the pixel information in the first grid to the image to be processed according to the coordinate information of the projection point of the adjustment result of the 3D key point on the image to be processed on, get the target image.
The image processing method according to any one of claims 1 to 4, further comprising:

generating a 3D extended model surrounding the 3D key point, wherein the vertices in the 3D extended model include an adjustment result of a first edge point and a second edge point obtained based on the extension of the first edge point, the first The edge point is a three-dimensional key point corresponding to the boundary of the target object;

Render the 3D extended model on the image to be processed according to the 3D extended model and the image to be processed.
The image processing method according to claim 9, wherein generating a three-dimensional extended model surrounding the three-dimensional key point comprises:

obtaining the first edge point;

On the extension line of the line connecting the central point in the three-dimensional key point and the first edge point, determine a point at a preset distance from the first edge point as the second edge point;

According to the pre-established second topology structure, a second grid is constructed between the adjustment result of the first edge point and the second edge point to obtain the three-dimensional extended model.
The image processing method according to claim 10, wherein obtaining the first edge point comprises:

Acquiring projection points of the three-dimensional key points in the image to be processed;

determining a projection point on the boundary of the target object according to the projection point of the target object and the three-dimensional key point in the image to be processed;

A three-dimensional key point corresponding to a projection point on the boundary of the target object is determined as the first edge point.
The image processing method according to claim 11, wherein, according to the target object and the projection point of the three-dimensional key point in the image to be processed, determining the projection point on the boundary of the target object comprises:

setting a plurality of straight lines in preset directions in the image to be processed;

Based on the points projected on the plurality of straight lines among the three-dimensional key points, a projected point on the boundary of the target object is determined.
The image processing method according to claim 9, wherein rendering the 3D extended model on the image to be processed according to the 3D extended model and the image to be processed comprises:

extracting the pixel information of the projection point of the first edge point on the image to be processed as the pixel information of the adjustment result of the first edge point, and extracting the pixel information of the second edge point on the image to be processed The pixel information of the projected point is used as the pixel information of the second edge point;

Rendering the second grid according to the pixel information of the adjustment result of the first edge point and the pixel information of the second edge point to obtain pixel information in the second grid;

According to the adjustment result of the first edge point and the coordinate information of the projection point of the second edge point on the image to be processed, the pixel information of the adjustment result of the first edge point, the second edge point The pixel information of the points and the pixel information in the second grid are projected onto the image to be processed.
The image processing method according to any one of claims 1 to 4, wherein the target object includes at least one of the following objects in the image to be processed: face, hand, and limb.
An image processing device, comprising:

An acquisition module, configured to acquire an image to be processed, wherein the image to be processed includes a target object;

The first correspondence module is used to extract the three-dimensional key points of the target object, and determine the first correspondence between the three-dimensional key points and the vertices in the three-dimensional adjustment model;

An adjustment module, configured to determine the adjustment result of the three-dimensional key points according to the adjustment instruction, the preset deformation data of the vertices in the three-dimensional adjustment model, and the first corresponding relationship;

The target module is configured to determine a target image according to the adjustment result of the 3D key points and the image to be processed.
An electronic device comprising a memory and a processor, the memory is used to store computer instructions executable on the processor, and the processor is used to implement any one of claims 1 to 14 when executing the computer instructions method described in the item.
A computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.
A computer program product comprising a computer program, which implements the method of any one of claims 1 to 14 when said program is executed by a processor.