CN112598591A

CN112598591A - Image processing method, image processing device, electronic equipment and storage medium

Info

Publication number: CN112598591A
Application number: CN202011510796.7A
Authority: CN
Inventors: 肖雪; 秦文煜; 刘晓坤
Original assignee: Beijing Dajia Internet Information Technology Co Ltd
Current assignee: Beijing Dajia Internet Information Technology Co Ltd
Priority date: 2020-12-18
Filing date: 2020-12-18
Publication date: 2021-04-02
Anticipated expiration: 2040-12-18
Also published as: CN112598591B

Abstract

The disclosure relates to an image processing method, an image processing device, an electronic device and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: carrying out face recognition on the first image to obtain a first face identification of a target face; searching first flaw point information corresponding to the first face identification in a database; the first flaw information comprises first positions of N flaws on the target face in the second image; performing position mapping on the first positions of the N flaw points to obtain second positions of the N flaw points in the first image; for each flaw, determining a first area and a second area in the first image according to the second position of the flaw in the first image; the second area comprises the flaw point after position mapping, and the first area is obtained by outwards expanding the target size on the basis of the second area; and repositioning the defect point according to the color mean values of the first area and the second area. The method can improve the positioning precision of the flaw point and has high positioning speed.

Description

Image processing method, image processing device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method and apparatus, an electronic device, and a storage medium.

Background

Because the beauty of the human face is affected by the flaws (such as acne, nevus, spots, etc.), the removal of the flaws becomes a very important link when the facial image is beautified. Wherein, before removing the facial blemish, the blemish needs to be positioned on the face, namely, the position and the size of the blemish are detected.

In addition, it is the basis for accurately positioning the blemishes to accurately remove the blemishes, and therefore, a new image processing method is urgently needed at present to realize high-precision positioning of the blemishes, so as to ensure the beauty effect of the portrait.

Disclosure of Invention

The present disclosure provides an image processing method, an image processing apparatus, an electronic device, and a storage medium, which can achieve both the positioning accuracy and the positioning speed of a flaw. Namely, the method not only realizes the high-precision positioning of the flaw point, but also has higher positioning speed.

According to a first aspect of embodiments of the present disclosure, there is provided an image processing method, the method including:

carrying out face recognition on the first image to obtain a first face identification of a target face;

searching first flaw point information corresponding to the first face identification in a database; the database is used for storing a plurality of face identifications and defect point information corresponding to the face identifications; the first flaw information comprises first positions of N flaw points on the target face in a second image, wherein N is a positive integer;

performing position mapping on first positions of the N defective points to obtain second positions of the N defective points in the first image;

for each said flaw, determining a first area and a second area in said first image according to a second position of said flaw in said first image; wherein the second area includes the flaw after position mapping, and the first area is obtained by expanding a target size outwards on the basis of the second area; repositioning the flaw in the first area according to the color mean values of the first area and the second area.

In some embodiments, the first flaw information further includes a first size of the N flaws in the second image;

determining a first area and a second area in the first image according to the second position of the flaw in the first image, including:

determining a second size of the flaw in the first image according to the size of the first detection frame, the size of the second detection frame and the first size of the flaw in the second image;

determining the first area in the first image according to the second size and the target size by taking the second position of the flaw as a center; and determining the second area in the first image according to the second size with the second position of the flaw as a center;

the first detection frame and the second detection frame are used for positioning the target face; the size of the first detection frame is obtained by carrying out face recognition on the first image; the size of the second detection frame is stored in the database and is obtained by performing face recognition on the second image.

In some embodiments, the method further comprises:

performing semantic segmentation processing on the first image to obtain a first type region and a second type region; the first-class area is a human face skin area which is not covered by a covering object in the first image; the second type of area is other areas except the first type of area in the first image;

determining M defect points in the second type area in the N defect points; wherein the M flaw points are flaw points which are not relocated; m is a positive integer and M is less than N.

In some embodiments, said determining, for each said flaw, a first area and a second area in said first image based on a second location of said flaw in said first image comprises:

for each defect of the remaining N-M defects, determining the first area and the second area in the first image according to a second position of the defect in the first image; wherein the first region and the second region are both located within the first type of region.

In some embodiments, said repositioning said flaw based on color means of said first area and said second area comprises:

determining a difference between the color mean of the first region and the color mean of the second region; repositioning the flaw in the first zone in response to the absolute value of the difference being not greater than a target threshold.

In some embodiments, the second location of the flaw after location mapping is taken as the current flaw point location in response to the absolute value of the difference being greater than the target threshold.

In some embodiments, said relocating said blemish within said first area comprises:

extracting a plurality of candidate feature points in the first region;

performing feature matching on each candidate feature point with the flaw point in the second image respectively;

and taking the position of the candidate feature point with the highest matching degree as a new flaw point position.

In some embodiments, the method further comprises:

acquiring face data, wherein the face data comprises a plurality of candidate images;

for each candidate image, carrying out face recognition on the candidate image to obtain a second face identifier and a candidate detection frame; performing flaw detection on the candidate image to obtain second flaw information;

the second defect information comprises the positions and sizes of X defect points in the candidate face in the candidate image, and X is a positive integer;

establishing a binding relationship between the candidate face identification and the candidate detection frame, establishing a binding relationship between the candidate face identification and the second defect point information, and storing the candidate face identification, the size of the candidate detection frame and the second defect point information into the database.

In some embodiments, said determining a second size of said flaw in said first image based on a size of a first detection box, a size of a second detection box, and a first size of said flaw in said second image comprises:

acquiring a first sum of the width and the height of the first detection frame;

acquiring a second sum of the width and the height of the second detection frame;

acquiring a first ratio between the first sum and a target value;

acquiring a second ratio between the second sum and the target value;

and acquiring a division result of the first ratio and the second ratio, and taking a product value of the first size and the division result as a second size of the flaw point in the first image.

According to a second aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, the apparatus including:

the face recognition module is configured to perform face recognition on the first image to obtain a first face identification of the target face;

the searching module is configured to search first defect point information corresponding to the first face identification in a database; the database is used for storing a plurality of face identifications and defect point information corresponding to the face identifications; the first flaw information comprises first positions of N flaw points on the target face in a second image, wherein N is a positive integer;

a position mapping module configured to perform position mapping on first positions of the N defect points to obtain second positions of the N defect points in the first image;

a first processing module configured to determine, for each said flaw, a first area and a second area in said first image according to a second position of said flaw in said first image; wherein the second area includes the flaw after position mapping, and the first area is obtained by expanding a target size outwards on the basis of the second area; repositioning the flaw in the first area according to the color mean values of the first area and the second area.

In some embodiments, the first flaw information further includes a first size of the N flaws in the second image; the first processing module comprises:

a first determining unit configured to determine a second size of the flaw in the first image according to the size of the first detection frame, the size of the second detection frame, and the first size of the flaw in the second image;

a second determining unit configured to determine the first area in the first image according to the second size and a target size with a second position of the flaw as a center; and determining the second area in the first image according to the second size with the second position of the flaw as a center;

In some embodiments, the apparatus further comprises:

the second processing module is configured to perform semantic segmentation processing on the first image to obtain a first type region and a second type region; the first-class area is a human face skin area which is not covered by a covering object in the first image; the second type of area is other areas except the first type of area in the first image; determining M defect points in the second type area in the N defect points; wherein the M flaw points are flaw points which are not relocated; m is a positive integer and M is less than N.

In some embodiments, the first processing module is configured to determine, for each of the remaining N-M blemishes, the first area and the second area in the first image in accordance with a second position of the blemish in the first image; wherein the first region and the second region are both located within the first type of region.

In some embodiments, the first processing module further comprises:

a first processing unit configured to determine a difference between a color mean of the first region and a color mean of the second region; repositioning the flaw in the first zone in response to the absolute value of the difference being not greater than a target threshold.

In some embodiments, the first processing module further comprises:

a second processing unit configured to take a second position of the flaw after position mapping as a current flaw point position in response to the absolute value of the difference being greater than the target threshold.

In some embodiments, the first processing unit is configured to extract several candidate feature points within the first region; performing feature matching on each candidate feature point with the flaw point in the second image respectively; and taking the position of the candidate feature point with the highest matching degree as a new flaw point position.

In some embodiments, the apparatus further comprises:

the system comprises a preprocessing module, a display module and a display module, wherein the preprocessing module is configured to acquire face data, and the face data comprises a plurality of candidate images; for each candidate image, carrying out face recognition on the candidate image to obtain a second face identifier and a candidate detection frame; performing flaw detection on the candidate image to obtain second flaw information; the second defect information comprises the positions and sizes of X defect points in the candidate face in the candidate image, and X is a positive integer; establishing a binding relationship between the candidate face identification and the candidate detection frame, establishing a binding relationship between the candidate face identification and the second defect point information, and storing the candidate face identification, the size of the candidate detection frame and the second defect point information into the database.

In some embodiments, the first determination unit is configured to obtain a first sum of a width and a height of the first detection frame; acquiring a second sum of the width and the height of the second detection frame; acquiring a first ratio between the first sum and a target value; acquiring a second ratio between the second sum and the target value; and acquiring a division result of the first ratio and the second ratio, and taking a product value of the first size and the division result as a second size of the flaw point in the first image.

According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including:

one or more processors;

one or more memories for storing the one or more processor-executable instructions;

wherein the one or more processors are configured to perform the image processing method described above.

According to a fourth aspect of embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium, wherein instructions of the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the above-mentioned image processing method.

According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product, wherein instructions of the computer program product, when executed by a processor of an electronic device, enable the electronic device to perform the above-mentioned image processing method.

The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:

when the facial defect point detection is performed, the embodiment of the disclosure performs face recognition on an image to be detected to obtain a face identifier, and then matches defect point information in a database according to the face identifier, wherein the database stores a plurality of face identifiers and defect point information corresponding to the face identifiers. That is, the defect is located in advance, and when the image is detected in real time, the position mapping can be quickly completed according to the defect point information stored in the database, so that the initial mapping defect information included in the image to be detected is obtained; this greatly speeds up the positioning speed; in addition, after the position mapping, the embodiment of the disclosure also performs the facial flaw relocation in the local area through the flaw local search, and further ensures the positioning accuracy. In conclusion, the method can give consideration to both the positioning accuracy and the positioning speed of the flaw. Namely, the method not only realizes the high-precision positioning of the flaw point, but also has higher positioning speed.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.

FIG. 1 is a flow diagram illustrating an image processing method according to an exemplary embodiment.

FIG. 2 is a flow diagram illustrating another image processing method according to an exemplary embodiment.

FIG. 3 is a schematic flow diagram illustrating a pre-computation phase according to an exemplary embodiment.

FIG. 4 is a flow diagram illustrating a real-time computation phase according to an exemplary embodiment.

Fig. 5 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.

FIG. 6 is a block diagram illustrating an electronic device in accordance with an example embodiment.

Detailed Description

In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

The user information to which the present disclosure relates may be information authorized by the user or sufficiently authorized by each party.

Some terms referred to in the embodiments of the present disclosure are explained first.

Beautifying human image: the method refers to beautifying and modifying the human face, wherein the human face beautifying process comprises but is not limited to face thinning, makeup application, facial blemish removal and the like.

Face blemish: refers to characteristic points which are appeared on skin areas (except eyebrows, eyes, mouth, etc.) of the human face, such as acne, moles or spots, and the like, and affect the appearance.

The following describes an implementation environment related to an image processing scheme according to an embodiment of the present disclosure.

The image processing scheme provided by the embodiment of the disclosure is used for positioning the facial defect points of the human face. Because the positions of the facial flaws of the same face are usually not changed within a certain period of time, the positions of the facial flaws can be located in advance by adopting a pre-calculation mode, for example, by adopting a 3D digital media (3D) model face reconstruction technology, the positions of the facial flaws are pre-calculated in advance, and the position mapping of the facial flaws is completed in a real-time calculation stage.

In addition, in order to improve the positioning accuracy of the facial defect, the embodiment of the present disclosure may add a local search process of the facial defect after the position mapping to relocate the facial defect. The scheme can improve the positioning precision of the facial flaws, and the time consumption is short, so that the real-time effect can be achieved.

In some embodiments, this scheme can be applied in any scenario where there is a need for detecting a facial flaw, which is not limited by the embodiments of the present disclosure. For example, a medical scene, a live scene, an online or offline portrait scene, and the like.

The execution main body of the scheme is electronic equipment. Illustratively, the electronic device is a terminal, and the type of the terminal includes, but is not limited to, a smart phone, a desktop computer, a notebook computer, a tablet computer, and the like. In addition, the electronic device may also be a server, for example, the terminal uploads the face image to the server, and as the server performs facial defect point positioning, this is not specifically limited in the embodiment of the present disclosure.

In other embodiments, taking the above electronic device as a terminal as an example, the face image currently processed by the terminal may be a locally stored image, a newly shot image, a video frame in a video call or a video live broadcast, or an image sent by another terminal, which is not limited in this disclosure. Taking the above electronic device as an example, the face image currently processed by the server may be a face image uploaded by the terminal.

The target tracking scheme of the embodiments of the present disclosure is explained in detail by the following embodiments.

Fig. 1 is a flowchart illustrating an image processing method for use in an electronic device, according to an exemplary embodiment, including the following steps.

In 101, face recognition is performed on the first image to obtain a first face identifier of the target face.

In 102, searching a database for first defect point information corresponding to a first face identification; the database is used for storing a plurality of face identifications and flaw point information corresponding to the face identifications; the first flaw information comprises first positions of N flaws on the target face in the second image, wherein N is a positive integer.

In 103, the first positions of the N blemishes are subjected to position mapping to obtain second positions of the N blemishes in the first image.

At 104, for each flaw, determining a first area and a second area in the first image based on a second location of the flaw in the first image; the second area comprises the flaw point after position mapping, and the first area is obtained by outwards expanding the target size on the basis of the second area; and repositioning the defect point in the first area according to the color mean values of the first area and the second area.

In the method provided by the embodiment of the present disclosure, when detecting a facial defect point, the embodiment of the present disclosure first performs face recognition on an image to be detected to obtain a face identifier, and then matches defect point information in a database according to the face identifier, where the database stores a plurality of face identifiers and defect point information corresponding to the face identifiers. That is, the defect is located in advance, and when the image is detected in real time, the position mapping can be quickly completed according to the defect point information stored in the database, so that the initial mapping defect information included in the image to be detected is obtained; this greatly speeds up the positioning speed; in addition, after the position mapping, the embodiment of the disclosure also performs the facial flaw relocation in the local area through the flaw local search, and further ensures the positioning accuracy. In conclusion, the method can give consideration to both the positioning accuracy and the positioning speed of the flaw. Namely, the method not only realizes the high-precision positioning of the flaw point, but also has higher positioning speed.

In some embodiments, the method further comprises:

extracting a plurality of candidate feature points in the first region;

In some embodiments, the method further comprises:

acquiring a first sum of the width and the height of the first detection frame;

acquiring a first ratio between the first sum and a target value;

acquiring a second ratio between the second sum and the target value;

All the above optional technical solutions may be combined arbitrarily to form the optional embodiments of the present disclosure, and are not described herein again.

FIG. 2 is a flow diagram illustrating an image processing method according to an exemplary embodiment. The method comprises a pre-calculation stage and a real-time calculation stage. The method is used in the electronic equipment and comprises the following steps.

Precomputation phase

In 201, face data is obtained, wherein the face data comprises a plurality of candidate images.

In some embodiments, the face data may be derived from one or more face data sets, which are not limited by the disclosed embodiments. In addition, for the convenience of subsequent distinction, the face images involved in the pre-calculation stage are collectively referred to as candidate images in the embodiment of the disclosure. And the face identification and the face detection frame obtained by carrying out face recognition on the candidate image are respectively called as a second face identification and a candidate detection frame.

In 202, for each candidate image, face recognition is performed on the candidate image to obtain a second face identifier and a candidate detection frame.

In the embodiment of the present disclosure, the face recognition is performed by a face recognition model. For face recognition, a face recognized in a face image is usually marked with a detection frame, and position information of the detection frame, that is, a position of the recognized face in the face image, is given. In some embodiments, the face recognition may be performed by using a deep learning face recognition algorithm, or may be performed by using a non-deep learning face recognition algorithm, which is not limited in the embodiments of the present disclosure. As one example, a face recognition model may be trained for face recognition based on pre-labeled face data.

It should be noted that, in addition to performing face recognition on each candidate image, the embodiment of the present disclosure performs the following

steps

203 and 204 on each candidate image.

In 203, the candidate image is subjected to flaw detection to obtain second flaw information.

In the embodiment of the present disclosure, flaw detection is performed by a face flaw detection model.

In some embodiments, the training process of the facial blemish detection model includes, but is not limited to:

2031. and acquiring a training sample image and labeling information of the training sample image.

The training sample image can be a face image with serious pox, a face image with more spots or a face image with more scars. Each training sample image may include a defect feature of one category or may include defect features of multiple categories, which is not limited in the embodiments of the present disclosure. In addition, the training sample image may further include a male face image and a female face image of various different flaw categories.

In some embodiments, the blemish in the training sample image may be marked by manual marking, wherein the marking information is used to record the location and size of the blemish being marked.

2032. Inputting a training sample image into a deep learning model; and determining whether the prediction result of the deep learning model for the training sample image is matched with the annotation information or not based on the target loss function.

As an example, the target loss function is a cross-entropy loss function, and the deep learning model is a convolutional neural network, which is not limited in this disclosure.

2033. And when the output prediction result is not matched with the labeling information, iteratively updating the network parameters of the deep learning model repeatedly and circularly until the model converges to obtain the facial flaw detection model.

On the basis of guaranteeing sample diversity, through carrying out the mark processing of flaw characteristic to training sample image, can promote the detection precision of the facial flaw detection model that the training came out. In addition, the detection accuracy of the trained facial flaw detection model can be further ensured due to the large number of samples.

The second defect information comprises the positions and sizes of X defect points in the candidate face in the candidate image, and X is a positive integer. In the present embodiment, X flaw points are all flaw points in the candidate face.

In 204, a binding relationship between the candidate face identifier and the candidate detection frame is established, a binding relationship between the candidate face identifier and the second defect point information is established, and the candidate face identifier, the size of the candidate detection frame and the second defect point information are stored in a database.

Wherein the size of the candidate detection frame comprises the height and width of the candidate detection frame.

In addition, applying the operation processing from step 202 to step 204 to the large batch of face data in step 201, a database with rich flaw data is obtained. Namely, the database is at least used for storing a plurality of face identifications and defect point information corresponding to the face identifications; in some embodiments, the flaw point information includes, but is not limited to: position information and size information of the flaw. The method can realize the advance prediction of flaw positions based on the database, ensures the positioning speed of flaw points in the subsequent real-time calculation stage, and provides guarantee for real-time facial flaw detection.

Based on this database, the following real-time computation phase may be entered in the disclosed embodiment.

Real-time computing phase

In 205, face recognition is performed on the first image to be detected to obtain a first face identifier of the target face.

The step 202 may be referred to as an implementation manner of performing face recognition on the first image, and details are not repeated here.

At 206, first defect point information corresponding to the first face identification is looked up in the database.

In this step, face ID matching is performed from the database according to the first face identifier. Wherein the second image is another image including the target face. In other words, the first image and the second image include the same face.

In the embodiment of the present disclosure, the first defect information includes position information and size information of N defects on the target face in the second image, where N is a positive integer. Here, the position information is referred to as a first position in the present disclosure, and the size information is referred to as a first size in the present disclosure. That is, the first defect information includes first positions and first sizes of N defects on the target person's face in the second image.

In 207, the first positions of the N blemishes on the target person's face in the second image are position-mapped to obtain the second positions of the N blemishes in the first image.

Since the same face may have different sizes in different images, the defect on the same face may have inconsistent sizes and positions in different images, and therefore, after matching the defect information in the database, position mapping is required. In some embodiments, the disclosed embodiments use a 3DMM face reconstruction model for position mapping. The 3D reconstruction of the human face refers to reconstructing a three-dimensional model of the human face from one or more two-dimensional human face images. In the embodiment of the present disclosure, the first positions of the N blemishes on the target face in the second image are input into the 3DMM face reconstruction model, so as to obtain the second positions of the N blemishes in the first image.

It should be noted that the first position is also referred to as a predicted position of the flaw in the embodiment of the present disclosure, and the second position is also referred to as a primary mapping position of the flaw in the embodiment of the present disclosure.

Assume that there are 4 blemishes on the target person's face and the predicted positions of these four blemishes are (x1, y1), (x2, y2), (x3, y3), (x4, y4), respectively, and accordingly, the size of each blemish is (d1, d2, d3, d 4). Then after the position mapping, the initial mapping positions (x1 ', y 1'), (x2 ', y 2'), (x3 ', y 3'), (x4 ', y 4') of the 4 blemishes are obtained.

In 208, performing semantic segmentation processing on the first image to obtain a first class region and a second class region; determining M defective points located in the second type area in the N defective points; and removing the M flaw points.

The first type of area is a human face skin area which is not shielded by a shielding object in the first image; the second type of area is the other area of the first image except the first type of area.

In some embodiments, the disclosed embodiments use a facial skin segmentation model to segment out facial skin regions (first type regions), excluding regions that are occluded by hands, hair, or other obstructions (second type regions). In other words, the embodiment of the present disclosure performs semantic segmentation processing on the first image based on a pre-trained facial skin segmentation model, so as to obtain the first-class region and the second-class region. The facial skin segmentation model is usually sensitive to edges as an image semantic segmentation model, so that more accurate segmentation edges can be obtained by using the image semantic segmentation model, and the segmentation effect is ensured.

In some embodiments, the training process of the facial skin segmentation model includes, but is not limited to:

2081. and acquiring a training sample image and an annotation segmentation result of the training sample image.

The method comprises the steps of obtaining training sample images, wherein the training sample images comprise images of a large number of human face areas which are shielded by shielding objects such as hands or objects, and labeling segmentation results obtained by manually labeling the training sample images. Illustratively, the visible face area which is not blocked and the invisible face area which is blocked in each training sample image are given by human in the annotation segmentation result.

2082. Inputting a training sample image into a deep learning model; and determining whether the prediction segmentation result of the training sample image output by the deep learning model is matched with the annotation segmentation result or not based on the target loss function.

As an example, the target loss function may be a cross-entropy loss function, and the deep learning model may be a convolutional neural network, such as a full convolutional neural network, which is not specifically limited by the embodiments of the present disclosure.

2083. And when the prediction segmentation result is not matched with the labeling segmentation result, iteratively updating the network parameters of the deep learning model repeatedly and circularly until the model converges to obtain the facial skin segmentation model.

In addition, the defect points in the second type area on the target human face can be removed, the remaining defect points are repositioned, namely the remaining defect points are further verified, the real defect points are detected, and the positioning accuracy of the defect points is further improved.

It should be noted that the embodiment of the present disclosure sequentially traverses the remaining N-M defect points, i.e., sequentially relocates each remaining defect point through the defect local search shown in the following steps 209 to 210.

In 209, for each defect of the N-M defects, determining a first area and a second area in the first image according to the second position of the defect in the first image; wherein the second area comprises the defect point after position mapping, and the first area is obtained after outward expanding the target size on the basis of the second area.

In some embodiments, determining the first area and the second area in the first image based on the first location of the flaw in the second image and the second location in the first image comprises:

2091. and determining the second size of the flaw in the first image according to the first size of the flaw in the second image, the size of the first detection frame and the size of the second detection frame.

The first detection frame and the second detection frame are used for positioning a target face in different images; the size of the first detection frame is obtained by carrying out face recognition on the first image; the size of the second detection frame is stored in the database and is obtained by face recognition of the second image in the pre-calculation stage.

In some embodiments, determining a second size of the defective dot in the first image based on the size of the first inspection box, the size of the second inspection box, and the first size of the defective dot in the second image comprises: acquiring a first sum of the width and the height of the first detection frame; acquiring a second sum of the width and the height of the second detection frame; acquiring a first ratio between the first sum and a target value; acquiring a second ratio between the second sum and the target value; and acquiring a division result of the first ratio and the second ratio, and taking a product value of the first size and the division result as a second size of the flaw point in the first image.

Taking the above flaw (x1 ', y 1') as an example, assuming that the distance between the primary mapping position and the actual position of the flaw obtained through the 3DMM face reconstruction model is not far, the embodiment of the present disclosure will take the primary mapping position as the center and cut out a square area (first area) nearby, where the new size of the flaw is:

d1’＝((face_width’+face_height’)/2)*d1/(face_width+face_height)/2

wherein, face _ height 'and face _ width' refer to the size of the first detection frame, face _ width + face _ height, the size of the second detection frame, and d1 refers to the first size of the defect (x1 ', y 1') in the second image; d1 ' refers to the second size of the blemish (x1 ', y1 ') in the first image.

From the above equation, the size of the defect point is updated according to the scaling of the face size. In other words, in the real-time calculation stage, the size of the defective spot is updated according to the scaling of the face size, so that the calculation accuracy of the size of the defective spot is ensured.

2092. Determining a first area in the first image according to the second size and the target size by taking the second position of the defective point as a center; and determining a second area in the first image according to a second size with the second position of the defective point as a center.

Continuing with the above flaw (x1 ', y1 ') as an example, the size of the first region is recD1 ═ d1 ' + extended _ width, where extended _ width refers to the target dimension, and the value of extended _ width can be self-defined. For example, the value of the expanded _ width is not too large, and only the candidate feature point in the first region needs to be ensured, and in some embodiments, the value of the expanded _ width is 40. The second region recD2 is a region centered at (x1 ', y1 ') and having a radius of d1 '. Illustratively, the region is also a square region.

In summary, the first area includes the defect point after position mapping and the face skin area located around the defect point, and the second area includes the defect point after position mapping, that is, the first area is obtained after the target size is expanded outward on the basis of the second area. In addition, after the position mapping, the embodiment of the disclosure also performs the facial flaw relocation through the flaw local search, and further ensures the positioning accuracy. Illustratively, the first region and the second region are both located within the first-type region.

In step 210, the defect point is repositioned based on the color mean of the first region and the second region.

In some embodiments, repositioning the defect point based on the color mean of the first region and the second region comprises: determining a difference value between the color mean value of the first area and the color mean value of the second area; repositioning the defect point in response to the absolute value of the difference being not greater than the target threshold.

Exemplarily, the color mean value of the first region refers to the color mean value of all pixel points in the first region in the R channel, the color mean value in the G channel, and the color mean value in the B channel; the color mean value of the second area refers to the color mean value of all pixel points in the second area in an R channel, the color mean value in a G channel and the color mean value in a B channel; the difference between the color mean of the first region and the color mean of the second region may be the difference between the average values of the three channels, which is not limited in the embodiments of the present disclosure.

Taking the color mean value of the first region as M0 and the color mean value of the second region as M1 as an example, if the absolute value of the difference between M0 and M1 is greater than the target threshold M, it represents that the defect point (x1 ', y 1') obtained after the position mapping of the 3DMM face reconstruction model is greatly different from the mean value of the surrounding skin regions, and the defect point is most likely to be a real defect point and may not need to be further relocated; if the absolute value of the difference between M0 and M1 is less than or equal to the target threshold M, it means that the difference between the defect point (x1 ', y 1') obtained after the position mapping of the 3DMM face reconstruction model and the mean value of the surrounding skin area is very small, and the defect point is likely not a true defect point and needs to be further repositioned.

Because the colors of the blemishes and the common skin area are usually different, whether a certain blemish is a real blemish or not is judged more accurately by utilizing the color mean value, and the subsequent blemish positioning accuracy is further ensured.

In other embodiments, the defect point is relocated, including but not limited to:

2101. a plurality of candidate feature points are extracted in the first region.

Illustratively, SIFT features are extracted within the first region. That is, the disclosed embodiments extract SIFT features within the truncated square region. The SIFT feature extraction is generally divided into the following steps:

a. and extracting the characteristic points. Illustratively, several feature points within the first region may be detected by a method of establishing a Difference of Gaussian (DOG) function.

b. Each feature point is added with detailed information (local feature), i.e., a descriptor.

For each feature point, there are three pieces of information: location, scale, and orientation. Next, a descriptor is established for each detected feature point, i.e. the feature points are described by a set of vectors so as not to change with various changes, such as illumination changes, view angle changes, and the like. Also, the descriptors should be highly unique in order to increase the probability of a correct match of feature points. Extracting feature points and adding detailed information to the feature points may be referred to as SIFT feature generation, that is, extracting feature vectors from an image that are not related to scale scaling, rotation, and brightness change.

Illustratively, 64-dimensional or 128-dimensional SIFT features may be extracted for each feature point. For example, in practical applications, in order to enhance the robustness of matching, 16 seed points of 4 × 4 may be used for each feature point to describe, and each seed point has vector information in 8 directions, so that a feature point can generate a 128-dimensional SIFT feature vector. That is, several candidate feature points are extracted within the first region, including but not limited to: detecting the characteristic points in the first area to obtain a plurality of characteristic points; and respectively establishing a descriptor for each detected feature point, wherein the descriptor is characterized by a feature vector with a fixed dimension.

2102. Performing feature matching on each candidate feature point and the flaw point in the second image respectively; and taking the position of the candidate feature point with the highest matching degree as a new flaw point position.

In the embodiment of the present disclosure, assuming that several candidate features ((p1, q1), (p2, q 2)..) are obtained, each candidate feature point is respectively subjected to feature matching with a flaw point (x1, y1) from the database one by one, and the position of the candidate feature point with the highest matching degree is taken as the final position (x1 ", y 1") after the flaw is relocated.

In some embodiments, feature matching is performed by calculating a feature distance between each candidate feature and the flaw (x1, y 1). Illustratively, the characteristic distance is a euclidean distance, which is not limited by the embodiments of the present disclosure.

In the defect repositioning process, the candidate characteristic points are searched, the characteristic matching is carried out, and the position of the candidate characteristic point with the highest matching degree is used as the final position after the defect repositioning, so that after position mapping, a more accurate defect position can be searched, and the positioning accuracy of the defect is ensured.

In the method provided by the embodiment of the present disclosure, when detecting a facial defect point, the embodiment of the present disclosure first performs face recognition on an image to be detected to obtain a face identifier, and then matches defect point information in a database according to the face identifier, where the database stores a plurality of face identifiers and defect point information corresponding to the face identifiers. That is, the defect is located in advance, and when the image is detected in real time, the position mapping can be quickly completed according to the defect point information stored in the database, so that the initial mapping defect information included in the image to be detected is obtained; this greatly speeds up the positioning speed; in addition, after the position mapping, the embodiment of the disclosure also performs the facial flaw relocation through the flaw local search, and further ensures the positioning accuracy. In conclusion, the method can give consideration to both the positioning accuracy and the positioning speed of the flaw. Namely, the method not only realizes the high-precision positioning of the flaw point, but also has higher positioning speed.

The following describes an overall execution flow of the image processing method provided by the embodiment of the present disclosure with reference to fig. 3 and 4.

Fig. 3 shows the execution flow of the pre-calculation stage, which includes the following steps 301 to 304.

In 301, for a face image img, a face recognition model is used to perform face recognition to obtain a face ID and a face detection frame, where the face detection frame is face _ height and face _ width.

In 302, flaw detection is performed on the face image img based on the face flaw detection model. If the detection precision of the model is high, the position of a flaw point in the face image can be accurately positioned.

Illustratively, assuming that the face image includes 4 defects in the target face, the model outputs the predicted positions ((x1, y1), (x2, y2), (x3, y3), (x4, y4)) and the predicted sizes (d1, d2, d3, d4) of the 4 defects.

In 303, the face ID, the position and size information of the 4 flaw points, and the size of the face detection frame are bound and stored in the database.

At 304, the above operations are applied to a large batch of face data to obtain a large database of flaw points.

Fig. 4 shows an execution flow of the real-time computation phase, which includes the following steps 401 to 407.

In 401, another image img ' of the target face is input, and face recognition is performed through a face recognition model to obtain a face ID and a face detection frame, where the face detection frame is face _ height ' and face _ width '.

At 402, matching is performed based on the database, and defect point information corresponding to the face ID is searched.

From the database, it can be known that there are 4 defects on the target face, and the predicted positions of the 4 defects are ((x1, y1), (x2, y2), (x3, y3), (x4, y4)), and the size of each defect is (d1, d2, d3, d 4).

In 403, a position mapping is performed based on the 3DMM face reconstruction model.

That is, the predicted positions of the 4 blemishes ((x1, y1), (x2, y2), (x3, y3), (x4, y4)) are input, and the outputs of the model are the initial mapped positions (x1 ', y 1'), (x2 ', y 2'), (x3 ', y 3'), (x4 ', y 4') of the 4 blemishes.

In 404, the image img' is semantically segmented based on a facial skin segmentation model, cutting out facial skin regions, excluding regions such as hands, hair, or regions occluded by other obstructions. Assuming that the defect (x4 ', y 4') is located in the shielded area among the 4 defects, the defect is rejected.

And traversing the remaining 3 defect points in sequence, and completing defect point relocation through defect local search.

Exemplarily, taking a flaw (x1 ', y 1') as an example, assuming that the distance between the primary mapping position and the actual position of the flaw obtained through the 3DMM face reconstruction model is not far, the embodiment of the present disclosure may take the primary mapping position as the center, and cut out a square area (first area) nearby, where the new size of the flaw is:

d1’＝((face_width’+face_height’)/2)*d1/(face_width+face_height)/2

continuing with the above flaw (x1 ', y1 '), the first region has a size recD1 ═ d1 ' + extended _ width, where extended _ width can be defined by itself. For example, the value of the expanded _ width is not too large, and only the candidate feature point in the first region needs to be ensured, and in some embodiments, the value of the expanded _ width is 40.

The second region recD2 is a region centered at (x1 ', y1 ') and having a radius of d1 '.

In 405, acquiring a color mean value m0 of the first region and a color mean value m1 of the second region; if the absolute value of the difference between M0 and M1 is greater than the target threshold M, it represents that the average value of the flaw points (x1 ', y 1') obtained after the position mapping of the 3DMM face reconstruction model is greatly different from the average value of the surrounding skin area, and the flaw points are most likely to be real flaw points and do not need to be further repositioned; if the absolute value of the difference between M0 and M1 is less than or equal to the target threshold M, it means that the difference between the defect point (x1 ', y 1') obtained after the position mapping of the 3DMM face reconstruction model and the mean value of the surrounding skin area is very small, and the defect point is likely not a true defect point and needs to be further repositioned.

In 406, SIFT features are extracted from the truncated square region, and assuming that a plurality of candidate features ((p1, q1), (p2, q 2.) are obtained, each candidate feature is feature-matched with the flaw point (x1, y1) in the database one by one, and the position of the candidate feature with the highest matching degree is taken as the final position (x1 ', y 1') after the flaw is relocated.

In 407, all of the remaining 3 defect points are subjected to the above steps 405 to 407, and the relocation is completed.

Fig. 5 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment. Referring to fig. 5, the apparatus includes a face recognition module 501, a lookup module 502, a location mapping module 503, and a first processing module 504.

The face recognition module 501 is configured to perform face recognition on the first image to obtain a first face identifier of a target face;

a searching module 502 configured to search a database for first defect point information corresponding to the first face identifier; the database is used for storing a plurality of face identifications and defect point information corresponding to the face identifications; the first flaw information comprises first positions of N flaw points on the target face in a second image, wherein N is a positive integer;

a position mapping module 503 configured to perform position mapping on first positions of the N defect points, to obtain second positions of the N defect points in the first image;

a first processing module 504 configured to determine, for each said flaw, a first area and a second area in said first image according to a second position of said flaw in said first image; wherein the second area includes the flaw after position mapping, and the first area is obtained by expanding a target size outwards on the basis of the second area; repositioning the flaw in the first area according to the color mean values of the first area and the second area.

When the device provided by the embodiment of the disclosure detects a facial defect point, the embodiment of the disclosure firstly performs face recognition on an image to be detected to obtain a face identifier, and then matches defect point information in a database according to the face identifier, wherein the database stores a plurality of face identifiers and defect point information corresponding to the face identifiers. That is, the defect is located in advance, and when the image is detected in real time, the position mapping can be quickly completed according to the defect point information stored in the database, so that the initial mapping defect information included in the image to be detected is obtained; this greatly speeds up the positioning speed; in addition, after the position mapping, the embodiment of the disclosure also performs the facial flaw relocation through the flaw local search, and further ensures the positioning accuracy. In conclusion, the method can give consideration to both the positioning accuracy and the positioning speed of the flaw. Namely, the method not only realizes the high-precision positioning of the flaw point, but also has higher positioning speed.

In some embodiments, the apparatus further comprises:

In some embodiments, the first processing module further comprises:

In some embodiments, the apparatus further comprises:

With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Fig. 6 shows a block diagram of an electronic device 600 according to an exemplary embodiment of the present disclosure. In general, the apparatus 600 includes: a processor 601 and a memory 602.

The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so on. The processor 601 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.

The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in the memory 602 is used to store at least one instruction for execution by the processor 601 to implement the image processing method provided by the method embodiments in the present disclosure.

In some embodiments, the apparatus 600 may further optionally include: a peripheral interface 603 and at least one peripheral. The processor 601, memory 602, and peripheral interface 603 may be connected by buses or signal lines. Various peripheral devices may be connected to the peripheral interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes: and a power supply 609.

The peripheral interface 603 may be used to connect at least one peripheral related to I/O (Input/Output) to the processor 601 and the memory 602. In some embodiments, the processor 601, memory 602, and peripheral interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral interface 603 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

The power supply 604 is used to power the various components in the device 600. The power source 604 may be alternating current, direct current, disposable batteries, or rechargeable batteries. When the power source 604 includes a rechargeable battery, the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery. The wired rechargeable battery is a battery charged through a wired line, and the wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery may also be used to support fast charge technology.

Those skilled in the art will appreciate that the configuration shown in fig. 6 does not constitute a limitation of the device 600, and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components may be employed.

In an exemplary embodiment, a computer-readable storage medium comprising instructions, such as a memory comprising instructions, executable by a processor of the electronic device 600 to perform the image processing method described above is also provided. Alternatively, the storage medium may be a non-transitory computer readable storage medium, which may be, for example, a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

In an exemplary embodiment, a computer program product is also provided, in which instructions, when executed by a processor of the electronic device 600, enable the electronic device 600 to perform the image processing method as in the above-described method embodiments.

Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.

It will be understood that the invention is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.

Claims

1. An image processing method, characterized in that the method comprises:

2. The image processing method according to claim 1, wherein the first flaw information further includes a first size of the N flaws in the second image;

3. The image processing method according to claim 1, characterized in that the method further comprises:

4. An image processing method according to claim 3, wherein said determining, for each said flaw, a first area and a second area in said first image based on a second position of said flaw in said first image comprises:

5. The image processing method according to any one of claims 1 to 4, wherein said repositioning the flaw in the first area according to the color mean of the first area and the second area comprises:

determining a difference between the color mean of the first region and the color mean of the second region;

repositioning the flaw in the first zone in response to the absolute value of the difference being not greater than a target threshold.

6. The image processing method according to any one of claims 1 to 4, wherein said repositioning the flaw in the first area comprises:

extracting a plurality of candidate feature points in the first region;

7. An image processing apparatus, characterized in that the apparatus comprises:

8. An electronic device, comprising:

one or more processors;

wherein the one or more processors are configured to perform the image processing method of any one of claims 1-6.

9. A non-transitory computer readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the image processing method of any of claims 1-6.

10. A computer program product, wherein instructions in the computer program product, when executed by a processor of an electronic device, enable the electronic device to perform the image processing method of any of claims 1-6.