CN111488778A - Image processing method and apparatus, computer system, and readable storage medium - Google Patents
Image processing method and apparatus, computer system, and readable storage medium Download PDFInfo
- Publication number
- CN111488778A CN111488778A CN201910459983.8A CN201910459983A CN111488778A CN 111488778 A CN111488778 A CN 111488778A CN 201910459983 A CN201910459983 A CN 201910459983A CN 111488778 A CN111488778 A CN 111488778A
- Authority
- CN
- China
- Prior art keywords
- image
- user
- target part
- head
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
- G06V40/175—Static expression
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/60—Editing figures and text; Combining figures or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Image Processing (AREA)
Abstract
The present disclosure provides an image processing method, including: acquiring a user image, processing the user image, and determining the coordinates of one or more key points of a target part image in the user image; correcting the target part image based on one or more key point coordinates of the target part image to obtain a corrected target part image; acquiring user attribute information, and recommending modification content for modifying the corrected target part image based on the user attribute information; and modifying the corrected target part based on the recommended modification content to obtain a target image. The present disclosure also provides an image processing apparatus, a computer system, and a computer-readable storage medium.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and more particularly, to an image processing method, an image processing apparatus, a computer system, and a computer-readable storage medium.
Background
In work and life, a certificate photo or a formal photo similar to the certificate photo is generally needed. The certificate photo or the formal photo similar to the certificate photo generally meets the following requirements: firstly, correcting the posture, for example, correcting the face and having natural expression; secondly, the clothes are regular and normally worn; and thirdly, the background is simple and pure, and is usually blue, white and red background cloth. Because of its special requirements, users generally need to go to a special photo studio to take photos, or ask professionals to perform PS and other processes to meet the requirements. However, the pace of life of modern people is relatively fast, and the time cost of taking a set of identification photographs is relatively high.
In order to facilitate the users to shoot the certificate photo or the regular photo similar to the certificate photo, some convenient modes are continuously provided in the market, such as a certificate photo self-service shooting machine of a subway station. However, the current self-service camera for the certificate photo still needs to be elaborately dressed up by a user and can be shot after the user goes to the self-service camera, and the convenience degree is still not high.
In implementing the disclosed concept, the inventors found that there are at least the following problems in the related art: when an image satisfying a specific condition is acquired, not only the requirement for the original image is relatively high, but also the requirement for the user itself is relatively high.
Disclosure of Invention
In view of the above, the present disclosure provides an image processing method, an image processing apparatus, a computer system, and a computer-readable storage medium.
One aspect of the present disclosure provides an image processing method, including: acquiring a user image, processing the user image, and determining the coordinates of one or more key points of a target part image in the user image; correcting the target part image based on one or more key point coordinates of the target part image to obtain a corrected target part image; acquiring user attribute information, and recommending modification content for modifying the corrected target part image based on the user attribute information; and modifying the corrected target part based on the recommended modification content to obtain a target image.
According to an embodiment of the present disclosure, in a case where the target part image is an image of a head, correcting the target part image based on coordinates of one or more key points of the target part image includes: estimating the pose of the head based on the coordinates of one or more key points of the image of the head; and correcting the estimated head posture to be the target posture.
According to an embodiment of the present disclosure, in a case where the target part image is an image of a head, correcting the target part image based on coordinates of one or more key points of the target part image includes: recognizing the facial expression of the head based on the coordinates of one or more key points of the image of the head; and correcting the facial expression obtained by recognition into a target type expression.
According to an embodiment of the present disclosure, acquiring the user attribute information includes: acquiring user attribute information generated based on user operation; or identifying the user image to acquire the user attribute information.
According to an embodiment of the present disclosure, modifying the corrected target site based on the recommended modification content includes: and modifying the corrected target site based on the recommended modification content by using the conditional countermeasure network.
Another aspect of the present disclosure provides an image processing apparatus including: the first acquisition module is used for acquiring a user image, processing the user image and determining the coordinates of one or more key points of a target part image in the user image; a correction module, configured to correct the target portion image based on one or more keypoint coordinates of the target portion image to obtain a corrected target portion image; a second obtaining module, configured to obtain user attribute information, and recommend modification content for modifying the corrected target region image based on the user attribute information; and the modification module is used for modifying the corrected target part based on the recommended modification content so as to obtain a target image.
According to an embodiment of the present disclosure, the correction module includes: an estimation unit configured to estimate a posture of the head based on coordinates of one or more key points of the image of the head when the target part image is the image of the head; and a first correction unit for correcting the estimated head posture to a target posture.
According to an embodiment of the present disclosure, the correction module includes: a recognition unit configured to recognize a facial expression of the head based on coordinates of one or more key points of the image of the head when the target portion image is an image of the head; and a second correction unit for correcting the recognized facial expression to a target type expression.
According to an embodiment of the present disclosure, the second obtaining module includes: a first acquisition unit configured to acquire user attribute information generated based on a user operation; or the second acquiring unit is used for identifying the user image and acquiring the user attribute information.
According to an embodiment of the present disclosure, the modification module is configured to modify the corrected target site based on a recommended modification content by using a conditional countermeasure network.
Another aspect of the present disclosure provides a computer system comprising: one or more processors; a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method as described above.
Another aspect of the disclosure provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to implement the method as described above.
Another aspect of the disclosure provides a computer program comprising computer executable instructions for implementing the method as described above when executed.
According to the embodiment of the disclosure, after the user image is acquired, the target part image is corrected based on the key point coordinates of the target part image, so that the target part image of the acquired user image can be automatically corrected under the condition that the target part image is not at a correct angle or a target angle. Meanwhile, it is possible to recommend decoration content for decorating the corrected target site image based on the user attribute information, and decorate the corrected target site image based on the recommended decoration content. Through the embodiment of the disclosure, the image meeting the specific condition can be automatically generated from the life photograph or the self-photographing of the user, the requirement on the original image is relatively low, the requirement on the user per se is relatively low, the technical problems that the requirement on the original image is relatively high and the requirement on the user per se is relatively high when the image meeting the specific condition is acquired in the related technology are solved, and the technical effect of reducing the time cost for the user to acquire the image meeting the specific condition is further achieved.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:
fig. 1 schematically illustrates an exemplary system architecture to which the image processing method and apparatus may be applied, according to an embodiment of the present disclosure;
FIG. 2 schematically shows a flow chart of an image processing method according to an embodiment of the present disclosure;
FIG. 3A schematically illustrates a first schematic of a target site image before correction, according to an embodiment of the disclosure;
FIG. 3B schematically illustrates a second schematic of a target site image before correction, in accordance with an embodiment of the present disclosure;
FIG. 3C schematically illustrates a diagram of a corrected target site image according to an embodiment of the disclosure;
FIG. 4 schematically illustrates a flow chart for correcting a target site image based on coordinates of one or more keypoints of the target site image, according to an embodiment of the disclosure;
FIG. 5 schematically illustrates a flow chart for correcting a target site image based on coordinates of one or more keypoints of the target site image, according to another embodiment of the present disclosure;
FIG. 6 schematically shows a flow chart of an image processing method according to another embodiment of the present disclosure;
fig. 7 schematically shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure; and
FIG. 8 schematically illustrates a block diagram of a computer system suitable for implementing the above-described method, according to an embodiment of the present disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
The embodiment of the disclosure provides an image processing method, which includes acquiring a user image, processing the user image, and determining coordinates of one or more key points of a target part image in the user image; correcting the target part image based on one or more key point coordinates of the target part image to obtain a corrected target part image; acquiring user attribute information, and recommending modification content for modifying the corrected target part image based on the user attribute information; and modifying the corrected target part based on the recommended modification content to obtain a target image.
Fig. 1 schematically shows an exemplary system architecture to which the image processing method and apparatus may be applied, according to an embodiment of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.
As shown in fig. 1, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired and/or wireless communication links, and so forth.
The user may use the terminal devices 101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The terminal devices 101, 102, 103 may have installed thereon various communication client applications, such as a shopping-like application, a web browser application, a search-like application, an instant messaging tool, a mailbox client, and/or social platform software, etc. (by way of example only).
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 105 may be a server providing various services, such as a background management server (for example only) providing support for websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and perform other processing on the received data such as the user request, and feed back a processing result (e.g., a webpage, information, or data obtained or generated according to the user request) to the terminal device.
It should be noted that the image processing method provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the image processing apparatus provided by the embodiment of the present disclosure may be generally disposed in the server 105. The image processing method provided by the embodiment of the present disclosure may also be executed by a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Accordingly, the image processing apparatus provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Alternatively, the image processing method provided by the embodiment of the present disclosure may also be executed by the terminal device 101, 102, or 103, or may also be executed by another terminal device different from the terminal device 101, 102, or 103. Accordingly, the image processing apparatus provided by the embodiment of the present disclosure may also be provided in the terminal device 101, 102, or 103, or in another terminal device different from the terminal device 101, 102, or 103.
For example, the image to be processed may be stored in any one of the terminal apparatuses 101, 102, or 103 (for example, but not limited to the terminal apparatus 101), or may be stored on an external storage apparatus and may be imported into the terminal apparatus 101. Then, the terminal device 101 may locally execute the image processing method provided by the embodiment of the present disclosure, or send the image to be processed to another terminal device, server, or server cluster, and execute the image processing method provided by the embodiment of the present disclosure by another terminal device, server, or server cluster that receives the image to be processed.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Fig. 2 schematically shows a flow chart of an image processing method according to an embodiment of the present disclosure.
As shown in fig. 2, the method includes operations S201 to S204.
In operation S201, a user image is acquired and processed to determine coordinates of one or more key points of a target portion image in the user image.
According to an embodiment of the present disclosure, the user image may be a life photograph or a self-photograph of the user, may be a whole body photograph, a half body photograph, or a head photograph of the user, or the like.
According to the embodiment of the disclosure, the user image can be processed by using an image segmentation technology, and the main body region can be segmented. For example, the human face part and the clothes part are segmented from the user image, and the human face part can be used as a target part image for subsequent processing.
According to an embodiment of the present disclosure, for example, fig. 3A schematically illustrates a first schematic view of a target site image before correction according to an embodiment of the present disclosure.
As shown in fig. 3A, the target part image may be a head image of the user, and at this time, the head image of the user is tilted to the left, and the head is not in a correct state. According to the embodiment of the disclosure, a three-dimensional coordinate system may be established on the head image, for example, a certain point of the head is selected as an origin of the three-dimensional coordinate system, an X axis, a Y axis and a Z axis are established, and a coordinate region where the head image is located in the user image is determined.
Fig. 3B schematically illustrates a second schematic of a target site image before correction according to an embodiment of the disclosure.
As shown in fig. 3B, the target part image may be a head image of the user, which is tilted upward at this time, and the head is not in a correct state.
According to the embodiment of the disclosure, the key point coordinates of the head image can be extracted through a key point extraction technology, and then the head posture is estimated by using the key point coordinates, so that the three-dimensional direction of the head in the user image is obtained.
In operation S202, the target region image is corrected based on one or more keypoint coordinates of the target region image, resulting in a corrected target region image.
According to the embodiment of the present disclosure, taking the target part image as an example of the head, the estimation of the head pose may be performed based on one or more key point coordinates of the target part image, the three-dimensional direction of the head in the user image is obtained, and then the head is rotated to the frontal face direction by using the affine transformation matrix, for example, the head is rotated along the Y axis and/or the Z axis.
For example, fig. 3C schematically illustrates a schematic view of a corrected target site image according to an embodiment of the present disclosure.
As shown in fig. 3C, the target part image may be a head image of the user, and the corrected target part image (i.e., the corrected head image of the user) is in a correct state.
According to examples of the present disclosure, the type of target site is not limited, for example, including but not limited to head, face, neck, eyes, and the like.
In operation S203, user attribute information is acquired, and decoration content for decorating the corrected target site image is recommended based on the user attribute information.
According to an embodiment of the present disclosure, the user attribute information may be generated based on the following two ways.
And (I) generating user attribute information based on user operation.
According to an embodiment of the present disclosure, the user attribute information may be attributes of the user's age, gender, expression, face shape, and the like. The user attribute information generated based on the user operation is an interactive mode, the user can directly select the attribute information from the template library, and the background can automatically select the makeup, the hair style and the clothes which are most suitable for the user characteristics by utilizing big data. This approach is suitable for users with explicit preferences or aesthetic confidence to themselves.
And (II) identifying the user image to generate user attribute information.
According to the embodiment of the disclosure, the user attribute information generated by identifying the user image is an automatic identification mode, the user image can be directly identified, and the attribute information of the age, the sex, the expression and the like of the user in the image can be determined.
According to an embodiment of the present disclosure, the decoration may be makeup, hairstyle, clothing, and the like of the user.
According to the embodiment of the disclosure, the user can select interactively according to the preference of the user and also can select an automatic identification mode, so that the system can select the modification content which is most suitable for the user from the background big data by analyzing the user attribute.
In operation S204, the corrected target site is modified based on the recommended modification content to obtain a target image.
According to an embodiment of the present disclosure, modifying the corrected target site based on the recommended modification content includes: and modifying the corrected target part based on the recommended modification content by using the conditional generation countermeasure network.
According to an embodiment of the present disclosure, a conditional generative confrontation network employed by the present disclosure is different from a conventional generative confrontation network in which a generator (G) and an arbiter (D) add additional information y as a condition, which may be, for example, makeup, clothes, background color, etc. of a user selected through user interaction or through big data recommendation.
According to the embodiment of the disclosure, the model type corresponding to the conditional generative confrontation network is not limited, and the additional information y can be added on the basis of the existing generative confrontation network model as a function of the condition. The objective function of the additional information y as a condition may be as shown in the following equation (1).
L＝Ladv(x|y)+Latt(x|y)+Lid(x|y)+R (1)
Wherein L represents the overall loss, LadvRepresenting resistance to loss, LattRepresenting attribute classification loss, LidRepresenting the computational loss of identity information, R represents a regularization term commonly used in training generative confrontation networks.
Confrontation loss LadvComprising two parts, wherein one part of the counter-loss is the counter-loss L of the generator GG＝E[(D(G(x|y))-1)2]. Where E represents a desire, x represents an input image of a user, such as a life photograph, y represents an attribute tag, and y ═ y is formed by a plurality of vectors1；y2；…；yn]，yiAnd a label vector representing the ith attribute, wherein the length of the label vector is equal to the number of categories of the attribute, such as the first attribute represents a hair style. If the model provides m hairstyles, then y1Is a vector of m dimensions, the value of j dimension is 1, the other values are 0, which indicates that j type hair style is selected. The purpose of minimizing this loss is to make the picture generated by the generator as realistic as possible given x and y, so that the discriminator cannot distinguish between true and false.
Another part of the countermeasure loss is that of the discriminator D
LD＝E[(D(G(x|y)))2]+E[(D(x)-1)2]The purpose of this loss is to improve the ability of the discriminator D to discriminate between authenticity and falsehood.
In order to ensure that the generated picture meets the requirements, the attributes of the generated image need to be classified to meet the classification requirements, with a classification attribute loss of Latt。LattComposed of a plurality of sub-losses, Latt＝λ1*Latt1+λ2*Latt2+…+λn*LattnWherein LattiIndicating the classification loss of the ith attribute if the vector yiIs 1, and the remaining dimensions are 0, LattiThe classifier representing the attribute i predicts the generated picture G (x | y) as a loss of the category j, thereby ensuring that the generated picture satisfies the category requirement. Lambda [ alpha ]iRepresenting the loss weights of different attributes.
In order to ensure the identity information of the person in the generated picture, the calculated loss L of the identity information needs to be addedidA trained high-precision face classifier can be used, Lid(x | y) represents the loss of x identity tag predicted from the generated picture G (x | y).
In the related art, editing an image requires a lot of manual intervention, even professional intervention, to combine different elements more naturally, such as peeling a face, replacing clothes, and the like. The method can automatically generate natural and vivid synthetic graphs by utilizing the condition generating type countermeasure network, the countermeasure error in the condition generating type countermeasure network can ensure the naturalness of the generated images, and the condition generating type countermeasure network can ensure the authenticity of the generated images.
By the embodiment of the disclosure, the target image is taken as the identification photo as an example, the portrait area can be automatically segmented from the life photo of the user or the self-photographing, manual marking or fine adjustment of the user is not needed, the optimal makeup and clothing recommendation can be carried out according to the selection of the user or the characteristics of the user, and finally the identification photo is generated through the condition generation type confrontation network.
According to the embodiment of the disclosure, after the target image is obtained, post-processing including cropping, image compression, super-resolution image generation and the like can be performed according to user requirements, and finally the result is output.
According to the embodiment of the disclosure, after the user image is acquired, the target part image is corrected based on the key point coordinates of the target part image, so that the target part image of the acquired user image can be automatically corrected under the condition that the target part image is not at a correct angle or a target angle; meanwhile, it is possible to recommend modification content for modifying the corrected target site image based on the user attribute information, and modify the corrected target site based on the recommended modification content. Through the embodiment of the disclosure, the image meeting the specific condition can be automatically generated from the life photograph or the self-photographing of the user, the requirement on the original image is relatively low, the requirement on the user per se is relatively low, the technical problems that the requirement on the original image is relatively high and the requirement on the user per se is relatively high when the image meeting the specific condition is acquired in the related technology are solved, and the technical effect of reducing the time cost for the user to acquire the image meeting the specific condition is further achieved.
The method shown in fig. 2 is further described with reference to fig. 4-7 in conjunction with specific embodiments.
FIG. 4 schematically illustrates a flow chart for correcting a target site image based on coordinates of one or more keypoints of the target site image according to an embodiment of the disclosure.
According to an embodiment of the present disclosure, in the case where the target part image is an image of the head, as shown in fig. 4, correcting the target part image based on the coordinates of one or more key points of the target part image includes operations S401 to S402.
In operation S401, a pose of a head is estimated based on coordinates of one or more key points of an image of the head.
According to an embodiment of the present disclosure, estimation of the head pose may be performed based on one or more keypoint coordinates of the image of the head, obtaining the three-dimensional direction of the head in the user image.
In operation S402, the estimated head pose is corrected to a target pose.
According to an embodiment of the present disclosure, for example, the estimated head pose may be rotated to the frontal face direction using an affine transformation matrix. According to an embodiment of the present disclosure, the type of the target pose is not limited, and includes, but is not limited to, a frontal face direction, a side face with different angular directions, a head-up or head-down pose, and the like.
FIG. 5 schematically illustrates a flow chart for correcting a target site image based on coordinates of one or more keypoints of the target site image according to another embodiment of the present disclosure.
According to an embodiment of the present disclosure, in the case where the target part image is an image of the head, correcting the target part image based on the coordinates of one or more key points of the target part image, as shown in fig. 5, includes operations S501 to S502.
In operation S501, a facial expression of a head is recognized based on coordinates of one or more key points of an image of the head.
According to an embodiment of the present disclosure, a partial region may be determined based on coordinates of one or more key points of an image of a head, and facial expression recognition may be performed on the partial region. According to an embodiment of the present disclosure, for example, the facial expression in the partial region may be determined using an expression recognition model.
In operation S502, the recognized facial expression is corrected to a target type expression.
According to an embodiment of the present disclosure, for example, the identified facial expression is a sad expression, and the target type expression is a smile expression, and the sad expression may be corrected to a smile expression.
According to the embodiment of the disclosure, the gesture and expression correction can be automatically performed on the face area in the input image by using the technologies such as the face key point and the like, and the gesture and expression of the face in the input image do not need to be manually processed.
Fig. 6 schematically shows a flow chart of an image processing method according to another embodiment of the present disclosure.
As shown in fig. 6, the image processing method includes operations S601 to S606.
In operation S601, a user image is input.
In operation S602, a face region is extracted. The technology belongs to semantic segmentation in computer vision, an image is input, each pixel value can be endowed with a semantic label through a human body analysis algorithm, and therefore the image can be known which region represents hair, which region represents clothes and the like. And the method provides possibility for subsequent attribute analysis, image editing and the like. Semantic segmentation algorithms that may be used to parse user images include the full probabilistic neural network (FCNN), the Deeplab network, the PSPNet, and the like.
In operation S603, the human Face key points may be extracted first, and the coordinate values of the designated human Face key points may be output for the input human Face image, and the commonly used human Face key points include 86 points, 106 points, etc. the algorithms available for detecting the human Face key points include Multi-task shell conditional Networks (MTCNN), Deep Alignment Networks (DAN), L ook at Boundary: a Boundary-Aware Face Alignment Algorithm (L AB), etc.
After face key points are extracted, head pose estimation can be carried out, mapping conversion and calibration among a world coordinate system (3D), a 2D landmark input image and a camera coordinate system are completed through an algorithm, and a rotation and translation matrix can be solved through a method of iterative solution through a Direct L initial transform (D L T) algorithm in combination with least square.
After extracting key points of the face, expression recognition can be carried out, a face image is input, and the expression type and degree of the face are output, such as surprise, smile, laugh and the like.
The user attribute analysis may simultaneously learn a plurality of tasks in a Multi-task learning (Multi-task L earning) manner.
In operation S605, a target image is generated using the condition generating type countermeasure network. For example, a certificate photo is generated.
In operation S606, the target image is output.
Fig. 7 schematically shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure.
As shown in fig. 7, the image processing apparatus 700 includes a first acquiring module 710, a correcting module 720, a second acquiring module 730, and a modifying module 740.
The first obtaining module 710 is configured to obtain a user image, process the user image, and determine coordinates of one or more key points of a target portion image in the user image;
the correction module 720 is configured to correct the target portion image based on one or more keypoint coordinates of the target portion image, so as to obtain a corrected target portion image;
the second obtaining module 730 is configured to obtain the user attribute information, and recommend modification content for modifying the corrected target region image based on the user attribute information; and
the modification module 740 is configured to modify the corrected target portion based on the recommended modification content to obtain a target image.
According to the embodiment of the disclosure, after the user image is acquired, the target part image is corrected based on the key point coordinates of the target part image, so that the target part image of the acquired user image can be automatically corrected under the condition that the target part image is not at a correct angle or a target angle; meanwhile, it is possible to recommend modification content for modifying the corrected target site image based on the user attribute information, and modify the corrected target site based on the recommended modification content. Through the embodiment of the disclosure, the image meeting the specific condition can be automatically generated from the life photograph or the self-photographing of the user, the requirement on the original image is relatively low, the requirement on the user per se is relatively low, the technical problems that the requirement on the original image is relatively high and the requirement on the user per se is relatively high when the image meeting the specific condition is acquired in the related technology are solved, and the technical effect of reducing the time cost for the user to acquire the image meeting the specific condition is further achieved.
According to an embodiment of the present disclosure, the correction module 720 includes: an estimation unit configured to estimate a pose of the head based on coordinates of one or more key points of the image of the head, in a case where the target part image is an image of the head; and a first correction unit for correcting the estimated head posture to a target posture.
According to an embodiment of the present disclosure, the correction module 720 includes: a recognition unit configured to recognize a facial expression of the head based on coordinates of one or more key points of the image of the head, in a case where the target part image is an image of the head; and a second correction unit for correcting the recognized facial expression to a target type expression.
According to an embodiment of the present disclosure, the second obtaining module 730 includes a first obtaining unit, configured to obtain user attribute information generated based on a user operation; and/or the second acquisition module comprises a second acquisition unit for identifying the user image and acquiring the user attribute information.
According to an embodiment of the present disclosure, the modification module 740 is configured to modify the corrected target site based on the recommended modification content using the conditional generation countermeasure network.
Any one or more of the modules, sub-modules, units, sub-units, or sub-units according to embodiments of the present disclosure may be implemented at least in part as hardware circuitry, e.g., a Field Programmable Gate Array (FPGA), a programmable logic array (P L a), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or any other reasonable manner of integrating or packaging circuitry, or in any one of or a suitable combination of software, hardware, and firmware implementations.
For example, any number of the first obtaining module 710, the correcting module 720, the second obtaining module 730, and the modifying module 740 may be combined into one module/unit/sub-unit, or any one of the modules/units/sub-units may be split into a plurality of modules/units/sub-units, or at least part of the functions of one or more of these modules/units/sub-units may be combined with at least part of the functions of the other modules/units/sub-units and implemented in one module/unit/sub-unit according to embodiments of the present disclosure, at least one of the first obtaining module 710, the correcting module 720, the second obtaining module 730, and the modifying module 740 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a programmable logic array (P L a), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or any other reasonable manner by integrating or encapsulating circuits, or any combination of the first obtaining module 710, the correcting module 720, the second obtaining module 730, the modifying module 740, or the modifying module 740 may be implemented as a computer, or a computer, where appropriate combination of the first obtaining module, the correcting module, the second obtaining module, the computer, or the computer, or the computer.
It should be noted that the image processing apparatus portion in the embodiments of the present disclosure corresponds to the image processing method portion in the embodiments of the present disclosure, and the description of the image processing apparatus portion specifically refers to the image processing method portion, and is not repeated herein.
FIG. 8 schematically illustrates a block diagram of a computer system suitable for implementing the above-described method, according to an embodiment of the present disclosure. The computer system illustrated in FIG. 8 is only one example and should not impose any limitations on the scope of use or functionality of embodiments of the disclosure.
As shown in fig. 8, a computer system 800 according to an embodiment of the present disclosure includes a processor 801 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. The processor 801 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flows according to embodiments of the present disclosure.
In the RAM 803, various programs and data necessary for the operation of the system 800 are stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. The processor 801 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 802 and/or RAM 803. Note that the programs may also be stored in one or more memories other than the ROM 802 and RAM 803. The processor 801 may also perform various operations of method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.
According to AN embodiment of the present disclosure, the system 800 may further include AN input/output (I/O) interface 805, the input/output (I/O) interface 805 also being connected to the bus 804. the system 800 may further include one or more of AN input section 806 including a keyboard, a mouse, and the like, AN output section 807 including a Cathode Ray Tube (CRT), a liquid crystal display (L CD), and the like, a speaker, and the like, a storage section 807 including a hard disk, and the like, and a communication section 809 including a network interface card such as a L AN card, a modem, and the like, the communication section 809 performs communication processing via a network such as the Internet, a drive 810 is also connected to the I/O interface 805 as necessary, a removable medium 811 such as a magnetic disk, AN optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 810 as necessary, so that a computer program read therefrom is mounted into the storage section 808 as necessary.
According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. The computer program, when executed by the processor 801, performs the above-described functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.
The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.
According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples may include, but are not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 802 and/or RAM 803 described above and/or one or more memories other than the ROM 802 and RAM 803.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.
The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.
Claims (12)
1. An image processing method comprising:
acquiring a user image, processing the user image, and determining the coordinates of one or more key points of a target part image in the user image;
correcting the target part image based on one or more key point coordinates of the target part image to obtain a corrected target part image;
acquiring user attribute information, and recommending modification content for modifying the corrected target part image based on the user attribute information; and
and modifying the corrected target part based on the recommended modification content to obtain a target image.
2. The method of claim 1, wherein, in the case that the target part image is an image of a head, correcting the target part image based on coordinates of one or more keypoints of the target part image comprises:
estimating a pose of the head based on coordinates of one or more keypoints of the image of the head; and
and correcting the head posture obtained by estimation into a target posture.
3. The method of claim 1, wherein, in the case that the target site image is an image of a head, correcting the target site image based on coordinates of one or more keypoints of the target site image comprises:
recognizing a facial expression of the head based on coordinates of one or more keypoints of the image of the head; and
and correcting the facial expression obtained by recognition into a target type expression.
4. The method of claim 1, wherein obtaining user attribute information comprises:
acquiring user attribute information generated based on user operation; or
And identifying the user image to acquire the user attribute information.
5. The method of claim 1, wherein modifying the corrected target site based on recommended modification content comprises:
and modifying the corrected target part by using a condition generating countermeasure network based on the recommended modification content.
6. An image processing apparatus comprising:
the first acquisition module is used for acquiring a user image, processing the user image and determining the coordinates of one or more key points of a target part image in the user image;
the correction module is used for correcting the target part image based on one or more key point coordinates of the target part image to obtain a corrected target part image;
the second acquisition module is used for acquiring user attribute information and recommending modification content for modifying the corrected target part image based on the user attribute information; and
and the modification module is used for modifying the corrected target part based on the recommended modification content so as to obtain a target image.
7. The apparatus of claim 6, wherein the correction module comprises:
an estimation unit configured to estimate a pose of the head based on coordinates of one or more key points of the image of the head, if the target part image is an image of the head; and
a first correction unit for correcting the estimated head posture to a target posture.
8. The apparatus of claim 6, wherein the correction module comprises:
a recognition unit configured to recognize a facial expression of the head based on coordinates of one or more key points of the image of the head, in a case where the target part image is an image of the head; and
and the second correction unit is used for correcting the recognized facial expression into a target type expression.
9. The apparatus of claim 6, wherein the second obtaining means comprises:
a first acquisition unit configured to acquire user attribute information generated based on a user operation; or
And the second acquisition unit is used for identifying the user image and acquiring the user attribute information.
10. The apparatus of claim 6, wherein the modification module is configured to modify the corrected target site based on recommended modification content using a conditional generation countermeasure network.
11. A computer system, comprising:
one or more processors;
a memory for storing one or more programs,
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-5.
12. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to carry out the method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910459983.8A CN111488778A (en) | 2019-05-29 | 2019-05-29 | Image processing method and apparatus, computer system, and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910459983.8A CN111488778A (en) | 2019-05-29 | 2019-05-29 | Image processing method and apparatus, computer system, and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111488778A true CN111488778A (en) | 2020-08-04 |
Family
ID=71791304
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910459983.8A Pending CN111488778A (en) | 2019-05-29 | 2019-05-29 | Image processing method and apparatus, computer system, and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111488778A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113096049A (en) * | 2021-04-26 | 2021-07-09 | 北京京东拓先科技有限公司 | Recommendation method and device for picture processing scheme |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6504546B1 (en) * | 2000-02-08 | 2003-01-07 | At&T Corp. | Method of modeling objects to synthesize three-dimensional, photo-realistic animations |
| JP2005141705A (en) * | 2003-11-04 | 2005-06-02 | Hiroto Hamazaki | Automatic generation of face image of character looking like animated character and information storage medium |
| CN106663340A (en) * | 2014-08-29 | 2017-05-10 | 汤姆逊许可公司 | Method and device for editing a facial image |
| CN107274354A (en) * | 2017-05-22 | 2017-10-20 | 奇酷互联网络科技（深圳）有限公司 | image processing method, device and mobile terminal |
| CN107408315A (en) * | 2015-02-23 | 2017-11-28 | Fittingbox公司 | The flow and method of glasses try-in accurate and true to nature for real-time, physics |
| CN108090465A (en) * | 2017-12-29 | 2018-05-29 | 国信优易数据有限公司 | A kind of dressing effect process model training method and dressing effect processing method |
| CN109472795A (en) * | 2018-10-29 | 2019-03-15 | 三星电子（中国）研发中心 | A kind of image edit method and device |
-
2019
- 2019-05-29 CN CN201910459983.8A patent/CN111488778A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6504546B1 (en) * | 2000-02-08 | 2003-01-07 | At&T Corp. | Method of modeling objects to synthesize three-dimensional, photo-realistic animations |
| JP2005141705A (en) * | 2003-11-04 | 2005-06-02 | Hiroto Hamazaki | Automatic generation of face image of character looking like animated character and information storage medium |
| CN106663340A (en) * | 2014-08-29 | 2017-05-10 | 汤姆逊许可公司 | Method and device for editing a facial image |
| CN107408315A (en) * | 2015-02-23 | 2017-11-28 | Fittingbox公司 | The flow and method of glasses try-in accurate and true to nature for real-time, physics |
| CN107274354A (en) * | 2017-05-22 | 2017-10-20 | 奇酷互联网络科技（深圳）有限公司 | image processing method, device and mobile terminal |
| CN108090465A (en) * | 2017-12-29 | 2018-05-29 | 国信优易数据有限公司 | A kind of dressing effect process model training method and dressing effect processing method |
| CN109472795A (en) * | 2018-10-29 | 2019-03-15 | 三星电子（中国）研发中心 | A kind of image edit method and device |
Non-Patent Citations (1)
| Title |
|---|
| 包仁达: "《基于区域敏感生成对抗网络的自动上妆算法》", 《软件学报》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113096049A (en) * | 2021-04-26 | 2021-07-09 | 北京京东拓先科技有限公司 | Recommendation method and device for picture processing scheme |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11481869B2 (en) | Cross-domain image translation | |
| US10872420B2 (en) | Electronic device and method for automatic human segmentation in image | |
| CN107704838B (en) | Target object attribute identification method and device | |
| US10789622B2 (en) | Generating and providing augmented reality representations of recommended products based on style compatibility in relation to real-world surroundings | |
| JP2020522285A (en) | System and method for whole body measurement extraction | |
| US20200151849A1 (en) | Visual style transfer of images | |
| WO2018121777A1 (en) | Face detection method and apparatus, and electronic device | |
| TWI573093B (en) | Method of establishing virtual makeup data, electronic device having method of establishing virtual makeup data and non-transitory computer readable storage medium thereof | |
| US11538096B2 (en) | Method, medium, and system for live preview via machine learning models | |
| Liang et al. | Facial skin beautification using adaptive region-aware masks | |
| US11922661B2 (en) | Augmented reality experiences of color palettes in a messaging system | |
| US9443325B2 (en) | Image processing apparatus, image processing method, and computer program | |
| US12118601B2 (en) | Method, system, and non-transitory computer-readable medium for analyzing facial features for augmented reality experiences of physical products in a messaging system | |
| US11915305B2 (en) | Identification of physical products for augmented reality experiences in a messaging system | |
| CN111008935B (en) | Face image enhancement method, device, system and storage medium | |
| KR20220163430A (en) | Identification of Physical Products for Augmented Reality Experiences in Messaging Systems | |
| CN108428214A (en) | A kind of image processing method and device | |
| US20210312678A1 (en) | Generating augmented reality experiences with physical products using profile information | |
| US12008811B2 (en) | Machine learning-based selection of a representative video frame within a messaging application | |
| WO2020037962A1 (en) | Facial image correction method and apparatus, and storage medium | |
| JP2014149677A (en) | Makeup support apparatus, makeup support system, makeup support method and makeup support system | |
| US20210256174A1 (en) | Computer aided systems and methods for creating custom products | |
| US20220207917A1 (en) | Facial expression image processing method and apparatus, and electronic device | |
| CN111488778A (en) | Image processing method and apparatus, computer system, and readable storage medium | |
| US11871104B2 (en) | Recommendations for image capture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200804 |
|
| RJ01 | Rejection of invention patent application after publication |