CN116052240A - Post-makeup face changing method based on deep convolutional neural network and face key point detection - Google Patents

Post-makeup face changing method based on deep convolutional neural network and face key point detection Download PDF

Info

Publication number
CN116052240A
CN116052240A CN202211405068.9A CN202211405068A CN116052240A CN 116052240 A CN116052240 A CN 116052240A CN 202211405068 A CN202211405068 A CN 202211405068A CN 116052240 A CN116052240 A CN 116052240A
Authority
CN
China
Prior art keywords
portrait
post
face
image
makeup
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211405068.9A
Other languages
Chinese (zh)
Inventor
赵辉煌
张法超
梁小满
林睦纲
孙雅琪
王耀南
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hengyang Normal University
Original Assignee
Hengyang Normal University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hengyang Normal University filed Critical Hengyang Normal University
Priority to CN202211405068.9A priority Critical patent/CN116052240A/en
Publication of CN116052240A publication Critical patent/CN116052240A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/24—Aligning, centring, orientation detection or correction of the image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20224—Image subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Ophthalmology & Optometry (AREA)
  • Processing Or Creating Images (AREA)

Abstract

The invention discloses a face changing method after making up based on deep convolutional neural network and face key point detection, which comprises the steps of firstly selecting a target portrait image and a post-making up portrait image which need to be converted, then respectively detecting the face key points of the two images, detecting the face key points by adopting a Dlib frame, correcting the obtained key points, and then storing the key point coordinates of the face in the two images; obtaining a face contour by using a BiSeNet network model; generating a picture after face change by using a StyleBlit algorithm; and finally, combining the eye and mouth selected areas generated by face segmentation with the selected areas of the whole face to obtain a final face-changing picture after makeup. According to the invention, the facial key points of the post-makeup portrait image and the target portrait image are detected, the post-makeup face change is completed by using an algorithm, and the identity of the main body of the target portrait image is kept in the face change process, so that the face change result is ensured to be more similar to the target portrait image in appearance.

Description

Post-makeup face changing method based on deep convolutional neural network and face key point detection
Technical Field
The invention relates to the technical field of deep learning, in particular to a face changing method after making up based on a deep convolutional neural network and face key point detection.
Background
With rapid development of technology, deep neural grids are widely used in a variety of fields. Post-cosmetic face changes belong to one field of style migration. Deep neural mesh based style migration is a new topic of intense research in the field of recent artificial intelligence, whose rationale is to utilize neural network models for two known images (called style image and content image, respectively) and to migrate styles (style) from "style image" to "content image". The purpose is to be able to generate new images of different styles of the same content from images of different styles.
Gatys et al, first published article "Image Style Transfer Using Convolutional Neural Networks" in 2016 CVPR (International Convergence of computer vision and pattern recognition) demonstrated the surprising ability of Convolutional Neural Networks (CNNs) to exhibit image style migration: by separating and recombining the picture content and style, CNNs can create works with artistic appeal. Since then, there has been great interest in neurostimulation migration in academic research and industrial applications, which has become one of the hot research problems in academia and industry for short years, based on deep neural mesh image style migration. Style migration has been studied extensively and in depth by teams including universities of Qinghua, universities of Beijing, universities of Stanford and universities of UC Berkeley, the world universities at home and abroad, the institute (institute) and the laboratory.
However, when performing style migration on a face, for example, using a post-makeup portrait as a style image, the conventional style migration method performs style migration on an non-makeup portrait as a content image, which has some problems, mainly including: the style migration of the image has great randomness, so that the effect is quite unsatisfactory in many cases, and some errors can be generated in some cases, for example, the partial characteristics of eyes in the style image are migrated to the mouth, or the background characteristics of the image are migrated to a foreground object, so that the migration effect is quite unsatisfactory.
Disclosure of Invention
In order to solve the technical problem that the effect is not ideal when the style migration is carried out on the face at present, the invention provides a face-changing method after makeup based on a deep convolutional neural network and face key point detection, which can effectively realize the style migration of the face.
In order to achieve the above purpose, the present invention provides the following technical solutions: a face changing method after make-up based on deep convolutional neural network and face key point detection comprises the following steps:
a face changing method after make-up based on deep convolutional neural network and face key point detection comprises the following steps:
step 1, respectively detecting the key points of the human face of the post-makeup portrait and the target portrait to obtain coordinates, correcting the coordinates of the key points of the human face of the post-makeup portrait, and then storing the corrected coordinates;
step 2, respectively calculating gradient images of the post-cosmetic portrait and the target portrait as position guidelines of the post-cosmetic portrait and the target portrait
Figure BDA0003936688550000021
And->
Figure BDA0003936688550000022
Step 3, respectively carrying out the following operations on the post-makeup portrait image and the target portrait image: converting into a gray level image, performing continuous three-time downsampling operation on the gray level image, then adjusting the gray level image into an original size, and finally performing Gaussian blur operation to obtain a blurred image;
step 4, subtracting the blurred image of the post-makeup portrait and the blurred image of the target portrait from the gray level images of the post-makeup portrait and the target portrait which are not subjected to downsampling respectively to obtain the appearance guidelines of the post-makeup portrait and the target portrait
Figure BDA0003936688550000023
And
Figure BDA0003936688550000024
re-modification->
Figure BDA0003936688550000025
The histogram makes it and->
Figure BDA0003936688550000026
Matching;
step 5, establishing a cube lookup table lookupcube, and based on
Figure BDA0003936688550000027
And->
Figure BDA0003936688550000028
Storing RGB color channel information of the post-makeup portrait image in a logo up cube;
step 6, obtaining a first contour of a human face according to the human face key point coordinates of a target portrait and combining a skin detection algorithm, obtaining a second contour of the human face and contours of eyes and mouth on the basis of a BiSeNet model for the target portrait, obtaining an accurate human face contour by intersecting the first contour and the second contour, and obtaining a human face mask by combining the contours of the eyes and the mouth;
step 7, using the face key points of the target portrait image and the face key points of the post-makeup portrait image
Figure BDA0003936688550000029
Obtaining +.f. of deformed post-cosmetic portrait using mobile least square algorithm>
Figure BDA00039366885500000210
Step 8, will
Figure BDA00039366885500000211
Inputting the post-makeup portrait image and the lookupcube into a StyleBlit algorithm to obtain stylized pictures StylezedImg;
and 9, replacing the area except the face mask in the StylizedImg with the area except the face mask in the target portrait image by using the face mask.
In the step 1, a Dlib frame is adopted to detect the face key points of the post-cosmetic portrait graph to obtain 68 face key point coordinates { (x) of the post-cosmetic portrait graph 1 ，y 1 )，(x 2 ，y 2 )，(x 3 ，y 3 )...(x 68 ，y 68 ) -a }; then for the key point coordinates of the mouth { (x) 49 ，y 49 )，(x 50 ，y 50 )...(x 68 ，y 68 ) Correcting, namely selecting a correct key point coordinate of a mouth, extracting color values of three channels of the key point coordinate RGB respectively, and then moving coordinates of other inaccurate key points until the color difference value between the key point coordinate and the selected correct key point coordinate is smaller than a preset threshold value to obtain corrected 68 face key point coordinates { (x) 1 ，y 1 )，(x 2 ，y 2 )，(x 3 ，y 3 )...(x 68 ，y 68 ) And put itSome key points are saved.
The method for changing the face after making up based on the depth convolution neural network and the face key point detection selects a correct key point coordinate of a mouth, and randomly selects one key point from key points for positioning an upper lip to serve as the correct key point coordinate.
The method for changing the face after making up based on the deep convolutional neural network and the face key point detection comprises the following steps in the step 2
Figure BDA0003936688550000031
Or->
Figure BDA0003936688550000032
RGB channel values for each pixel point:
Figure BDA0003936688550000033
wherein the method comprises the steps of
Figure BDA0003936688550000034
Pixel R channel of position guide for post-cosmetic portrait or target portrait, < >>
Figure BDA0003936688550000035
Pixel point G channel of position guide of post-cosmetic portrait or target portrait, +.>
Figure BDA0003936688550000036
The method comprises the steps that a channel B is a pixel point of a position guide of a post-cosmetic portrait image or a target portrait image, w is the width of the post-cosmetic portrait image or the target portrait image, h is the height of the post-cosmetic portrait image or the target portrait image, x and y are x coordinates and y coordinates corresponding to each pixel point in the post-cosmetic portrait image or the target portrait image, meanwhile, the x coordinates correspond to a red channel, the y coordinates correspond to a green channel, and the blue channel is set to 0;
the values of the three channels are then saved and generated using the OpenCV library
Figure BDA0003936688550000037
And->
Figure BDA0003936688550000038
In the step 4, the method for changing the face after making up based on the deep convolutional neural network and the face key point detection is firstly obtained by using the following formula
Figure BDA0003936688550000039
And->
Figure BDA00039366885500000310
Figure BDA00039366885500000314
Wherein orig is gray The target portrait or the grey-scale image converted from the post-makeup portrait, the bluImg is the blurred image of the post-makeup portrait or the target portrait obtained in the step 3, G app An appearance guide for a post-cosmetic portrait or a target portrait;
then use OpencCV library modifications
Figure BDA00039366885500000311
To be equal to->
Figure BDA00039366885500000312
Is matched with the histogram of the object.
In the method for changing face after making up based on the deep convolutional neural network and the face key point detection, in the step 5, coordinates of example pixels of the best matching pattern of red and green channel values of the post-making up portrait map are stored in a look-up cube, wherein the coordinates are obtained by the following steps:
firstly, calculating RGB channel error values between pixel points of two position guides according to the following steps of traversing all pixel points of the position guides of the post-makeup portrait image and the target portrait image:
Figure BDA00039366885500000313
wherein the method comprises the steps of
Figure BDA0003936688550000041
Pixel point G channel for position guide of post-cosmetic portrait map, +.>
Figure BDA0003936688550000042
Pixel point G channel of position guide for target portrait map, +.>
Figure BDA0003936688550000043
Pixel R channel of position guide for post-cosmetic portrait>
Figure BDA0003936688550000044
The pixel point R channel of the position guide of the target portrait image, and alpha and beta are weight values;
then, after all the pixel points are guided by the pixel points of the position guide of each target portrait image after matching the position guide of the portrait image after makeup, the pixel in the calculated minimum error value is taken as an example pixel of the optimal matching pattern, and the pixel is corresponding to the example pixel
Figure BDA0003936688550000045
The coordinates of the corresponding pixel are stored in a lookupcube while the corresponding +.>
Figure BDA0003936688550000046
Together with the gray scale intensities in the bookup.
In the step 6, the face under the eyebrow and the face under the eyebrow are obtained through face key point detection, the face of the forehead part is obtained through skin detection, and the first contour of the face is obtained through combination.
In the method for changing face after makeup based on the deep convolutional neural network and the face key point detection, in the step 8, the StyleBlit algorithm is used for changing face after makeup, and the distance between a post-makeup portrait image and a plurality of coordinates of a target portrait image is defined as follows:
Figure BDA0003936688550000047
wherein P is a post-makeup portrait drawing, O is a target portrait drawing, and alpha 'and beta' are weights;
Figure BDA0003936688550000048
and->
Figure BDA0003936688550000049
Obtained from a lookupcube, taking the set of pixels of the post-makeup image as { P ] 1 ，P 2 ，P 3 ，...P n The pixel set of the target image is { O } 1 ，O 2 ，O 3 ，...O n Then render the target picture O: traversing all pixels { O } of the target image 1 ，O 2 ，...O n And after finding the nearest pixel point in the corresponding post-makeup picture, calculating the error rate E between the two pixel points, and copying the pixels of the post-makeup picture into the target picture if the E is smaller than a preset threshold value to finally obtain StylizedImg.
The invention has the technical effects that the facial key points of the post-makeup portrait image and the target portrait image are detected, the post-makeup face change is completed by using an algorithm, the identity of the main body of the target portrait is kept in the face change process, and the face change result is ensured to be more similar to the target portrait image in appearance. Meanwhile, in the face changing process, the method does not need GPU to participate in calculation, long-time training and a large data set are not needed, and only tens of pictures are needed. Meanwhile, the method and the device for locating and identifying the post-makeup portrait based on the segmentation algorithm solve the problems caused by the fact that the face segmentation is carried out by using a skin detection method before, can quickly and accurately segment the post-makeup portrait, avoid the occurrence of errors such as wrong segmentation, incomplete segmentation and the like of the post-makeup portrait, and improve the effect of face changing after makeup. The invention also provides a calibration mode aiming at the problem of inaccurate positioning of part of facial key points, thereby realizing accurate calibration of the facial key points of the people after make-up.
Drawings
FIG. 1 is a schematic flow chart of the invention for obtaining a face mask;
FIG. 2 is a system flow diagram of the present invention;
FIG. 3 is a diagram of a model architecture of the present invention;
FIG. 4 is a target portrait view as used in an embodiment of the present invention;
FIG. 5 is a post-cosmetic portrait view of an embodiment of the present invention;
FIG. 6 is a schematic diagram of keypoints of the 68 face keypoint detection method used in the present invention;
FIG. 7 is a stylizideimg employed in an example of the present invention;
FIG. 8 is a graph of the face-changing results after makeup of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The post-makeup portrait mentioned in this example is based on the post-makeup appearance of the Beijing opera small raw facial makeup, and can be performed as a post-makeup portrait in practical use. The target portrait image mentioned in this embodiment is a portrait image of the person whose face is changed.
The method for changing the face after making up based on the deep convolutional neural network and the face key point detection provided by the embodiment comprises the following steps:
and step 1, respectively detecting key points of the human face on the portrait picture after the make-up and the target portrait picture to obtain coordinates, correcting and storing the coordinates. Wherein the key points in the embodiment are detected by 68 face key points commonly used at present, and the specific distribution of the key points is shown in fig. 6As shown. In this embodiment, dlib frame is used to detect the face key points, and 68 face key point coordinates { (x) are obtained correspondingly 1 ，y 1 )，(x 2 ，y 2 )，(x 3 ，y 3 )...(x 68 ，y 68 ) }. However, when positioning the lips of the post-makeup portrait image, the Dlib frame often suffers from positioning inaccuracy, particularly the lower lips of the lips, due to the characteristic difference between the post-makeup and non-makeup faces. The positioning of the upper lip is always accurate because the distinguishing features between the upper lip part and the person part are obvious, i.e. the limit is relatively sharp. However, the lower lip portion often suffers from inaccurate positioning because there is no obvious boundary with the upper lip or other reasons. The coordinates { (x) of the key points for the upper lip portion are calculated by using the 5 key points for locating the upper lip portion as the correct key points in the present embodiment 49 ，y 49 )，(x 50 ，y 50 )...(x 68 ，y 68 ) Specifically, in this embodiment, any one of the 5 key points of the upper lip is selected as the correct key point, the color values of the three channels of the key point coordinate RGB are extracted respectively, and then the coordinates of other inaccurate key points are moved until the color difference between the key point coordinate and the selected correct key point coordinate is smaller than a preset threshold value, so as to obtain corrected 68 face key point coordinates { (x) 1 ，y 1 )，(x 2 ，y 2 )，(x 3 ，y 3 )...(x 68 ，y 68 ) And save these keypoints. The reason why the present application adjusts the coordinates of the inaccurate keypoints in this way is that, considering that the lips of the face in the post-makeup portrait drawing must be lips after makeup, i.e., lips coated with lipstick, and the color of lipstick is uniform over the whole lips, it is only necessary to check the color difference between the lips except the upper lips and the accurate keypoints when judging whether the keypoints are accurate. As the 68 face key point detection is the existing common technology, the embodiment adopts the technology such as Kazemi V, sullivan J.one millisecond face alignment with an ensemble of regression trees[C]Proceedings of the IEEE conference on computer vision and pattern recovery.2014: 1867-1874, the accuracy of the method is not a major issue, so the inaccuracy of positioning is not particularly large, and only the method in this embodiment is needed to adjust, so that correction can be achieved. The preset threshold value can be adjusted according to the actual situation so as to meet the requirement of the actual situation.
Step 2, respectively calculating gradient images of the post-cosmetic portrait and the target portrait as position guidelines of the post-cosmetic portrait and the target portrait
Figure BDA0003936688550000069
And->
Figure BDA00039366885500000610
(Position Guide). Wherein +.>
Figure BDA00039366885500000611
Or->
Figure BDA00039366885500000612
RGB channel values for each pixel point:
Figure BDA0003936688550000061
wherein the method comprises the steps of
Figure BDA0003936688550000062
Pixel R channel of position guide for post-cosmetic portrait or target portrait, < >>
Figure BDA0003936688550000063
Pixel point G channel of position guide of post-cosmetic portrait or target portrait, +.>
Figure BDA0003936688550000064
For post-cosmetic or target portraitThe pixel point B channel of the position guide is w is the width of the post-cosmetic portrait drawing or the target portrait drawing, h is the height of the post-cosmetic portrait drawing or the target portrait drawing, x and y are the x coordinate and the y coordinate corresponding to each pixel point in the post-cosmetic portrait drawing or the target portrait drawing, meanwhile, the x coordinate corresponds to the red channel, the y coordinate corresponds to the green channel, and the blue channel is set to 0;
the values of the three channels are then saved and generated using the OpenCV library
Figure BDA0003936688550000065
And->
Figure BDA0003936688550000066
Step 3, respectively carrying out the following operations on the post-makeup portrait image and the target portrait image: converting into a gray level image, performing continuous three-time downsampling operation on the gray level image, adjusting the gray level image into an original size, and finally performing Gaussian blur operation to obtain a blurred image.
Step 4, subtracting the blurred image of the post-makeup portrait and the blurred image of the target portrait from the gray level images of the post-makeup portrait and the target portrait which are not subjected to downsampling respectively to obtain the appearance guidelines of the post-makeup portrait and the target portrait
Figure BDA0003936688550000067
And
Figure BDA0003936688550000068
(Appearance Guide) and then modify +.>
Figure BDA0003936688550000071
The histogram makes it and->
Figure BDA0003936688550000072
Matching. Specifically, in this embodiment, the following formula is first used to obtain +.>
Figure BDA00039366885500000715
And->
Figure BDA0003936688550000073
Figure BDA00039366885500000716
Wherein orig is gray The target portrait or the grey-scale image converted from the post-makeup portrait, the bluImg is the blurred image of the post-makeup portrait or the target portrait obtained in the step 3, G app An appearance guide for a post-cosmetic portrait or a target portrait;
then modified using the qpeccv library
Figure BDA0003936688550000074
To be equal to->
Figure BDA0003936688550000075
Is matched with the histogram of the object.
Step 5, establishing a cube lookup table lookupcube, and based on
Figure BDA0003936688550000076
And->
Figure BDA0003936688550000077
RGB color channel information for the post-cosmetic portrait map is stored in the lookupcube. Specifically, RGB channel error values between the two position guide pixels are first calculated by traversing all pixels of the position guide of the post-makeup portrait map and the target portrait map according to: />
Figure BDA0003936688550000078
Wherein the method comprises the steps of
Figure BDA0003936688550000079
Pixel point G channel for position guide of post-cosmetic portrait map, +.>
Figure BDA00039366885500000710
Pixel point G channel of position guide for target portrait map, +.>
Figure BDA00039366885500000711
Pixel R channel of position guide for post-cosmetic portrait>
Figure BDA00039366885500000712
The pixel point R channel of the position guide of the target portrait image, and alpha and beta are weight values;
then, after all the pixel points are guided by the pixel points of the position guide of each target portrait image after matching the position guide of the portrait image after makeup, the pixel in the calculated minimum error value is taken as an example pixel of the optimal matching pattern, and the pixel is corresponding to the example pixel
Figure BDA00039366885500000713
The coordinates of the corresponding pixel are stored in a lookupcube while the corresponding +.>
Figure BDA00039366885500000714
Together with the gray scale intensities in the bookup. The two position guide pixels are matched one by error values, and then the pixel corresponding to the calculated minimum error value is used as a matched pixel and stored.
And 6, acquiring a first contour of a human face according to the human face key point coordinates of the target portrait and combining a skin detection algorithm, acquiring a second contour of the human face and contours of eyes and mouth based on a BiSeNet model for the target portrait, acquiring an accurate human face contour by intersecting the first contour and the second contour, and acquiring a human face mask by combining the contours of the eyes and the mouth. The first contour of the face is obtained by detecting key points of the face to obtain the eyebrows and the face below the eyebrows, and then obtaining the face of the forehead part by detecting the skin, and comprehensively obtaining the face. In this example, as used in the document Yu C, wang J, peng C, et al Bisenet: bilateral segmentation network for real-time semantic segmentation [ C ]. Proceedings of the European conference on computer vision (ECCV): 2018:325-341. Methods are presented in.
Step 7, using the face key points of the target portrait image and the face key points of the post-makeup portrait image
Figure BDA0003936688550000085
Obtaining +.f. of deformed post-cosmetic portrait using mobile least square algorithm>
Figure BDA0003936688550000086
. The moving least squares algorithm used in this embodiment can be referred to in Schaefer S, mcPhail T, warren J.image deformation using moving least squares [ M ]].ACM SIGGRAPH 2006Papers.2006：533-540。
Step 8, will
Figure BDA0003936688550000081
The post-makeup portrait map and the lookupcube are input into a StyleBlit algorithm to obtain a stylized picture stylizideimg. The StyleBlit algorithm used in this embodiment can be referred to as +.>
Figure BDA0003936688550000087
D，
Figure BDA0003936688550000088
ka O，Texler O，et al.StyleBlit：Fast Example-Based Stylization with Local Guidance[C]Computer Graphics forum.2019, 38 (2): 83-91. Specifically, in this embodiment, the StyleBlit algorithm is first used to make a post-makeup face change, and the distance between the coordinates of the post-makeup portrait map and the target portrait map is defined as:
Figure BDA0003936688550000082
wherein P is a post-makeup portrait drawing, O is a target portrait drawing, and alpha 'and beta' are weights;
Figure BDA0003936688550000083
and->
Figure BDA0003936688550000084
Obtained from a lookupcube, taking the set of pixels of the post-makeup image as { P ] 1 ，P 2 ，P 3 ，...P n The pixel set of the target image is { O } 1 ，O 2 ，O 3 ，...O n Then render the target picture O: traversing all pixels { O } of the target image 1 ，O 2 ，...O n And after finding the nearest pixel point in the corresponding post-makeup picture, calculating the error rate E between the two pixel points, and copying the pixels of the post-makeup picture into the target picture if the E is smaller than a preset threshold value to finally obtain StylizedImg.
Step 9, using the face mask, replacing the area except the face mask in the stylizideimg with the area except the face mask of the target portrait image, thereby obtaining the image as in fig. 8.
The invention detects the key points of the face of the post-makeup portrait drawing and the target portrait drawing based on the convolutional neural network, then divides the complete face parts of the post-makeup portrait drawing and the target portrait drawing according to the key points of the face, generates an appearance guide and a position guide, ensures the retention of the appearance of the target main body and ensures the consistency of semantics in the face changing process, and realizes the face changing function after makeup.
In summary, the invention detects the key points of the human face on the post-makeup portrait image and the target portrait image, completes the post-makeup face change by using the algorithm, maintains the main identity of the target portrait in the face change process, and ensures that the face change result is more similar to the target portrait image in appearance. Meanwhile, in the face changing process, the method does not need GPU to participate in calculation, does not need long-time training and does not need a large data set, and only needs tens of pictures; the invention provides a post-makeup portrait segmentation algorithm, which avoids the problems caused by the prior face segmentation by using a skin detection method, can quickly and accurately segment the post-makeup portrait, avoids the generation of errors such as wrong segmentation, incomplete segmentation and the like of the post-makeup portrait, and improves the effect of post-makeup face change ； The invention proposesAn improved facial key point calibration algorithm is provided to achieve accurate calibration of facial key points of a post-cosmetic character.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.

Claims (8)

1. A face changing method after make-up based on deep convolutional neural network and face key point detection is characterized in that: the method comprises the following steps:
step 1, respectively detecting the key points of the human face of the post-makeup portrait and the target portrait to obtain coordinates, correcting the coordinates of the key points of the human face of the post-makeup portrait, and then storing the corrected coordinates;
step 2, respectively calculating gradient images of the post-cosmetic portrait and the target portrait as position guidelines of the post-cosmetic portrait and the target portrait
Figure FDA0003936688540000011
And->
Figure FDA0003936688540000012
Step 3, respectively carrying out the following operations on the post-makeup portrait image and the target portrait image: converting into a gray level image, performing continuous three-time downsampling operation on the gray level image, then adjusting the gray level image into an original size, and finally performing Gaussian blur operation to obtain a blurred image;
step 4, subtracting the blurred image of the post-cosmetic portrait and the model of the target portrait from the gray level image of the non-downsampled post-cosmetic portrait and the gray level image of the target portrait respectivelyPasting images to obtain an appearance guide of the post-makeup portrait and the target portrait
Figure FDA0003936688540000013
And
Figure FDA0003936688540000014
re-modification->
Figure FDA0003936688540000015
The histogram makes it and->
Figure FDA0003936688540000016
Matching;
step 5, establishing a cube lookup table lookupcube, and based on
Figure FDA0003936688540000017
And->
Figure FDA0003936688540000018
Storing RGB color channel information of the post-makeup portrait image in a logo up cube;
step 6, obtaining a first contour of a human face according to the human face key point coordinates of a target portrait and combining a skin detection algorithm, obtaining a second contour of the human face and contours of eyes and mouth on the basis of a BiSeNet model for the target portrait, obtaining an accurate human face contour by intersecting the first contour and the second contour, and obtaining a human face mask by combining the contours of the eyes and the mouth;
step 7, using the face key points of the target portrait image and the face key points of the post-makeup portrait image
Figure FDA0003936688540000019
Obtaining +.f. of deformed post-cosmetic portrait using mobile least square algorithm>
Figure FDA00039366885400000110
Step 8, will
Figure FDA00039366885400000111
Inputting the post-makeup portrait image and the lookupcube into a StyleBlit algorithm to obtain stylized pictures StylezedImg;
and 9, replacing the area except the face mask in the StylizedImg with the area except the face mask in the target portrait image by using the face mask.
2. The post-cosmetic face-changing method based on deep convolutional neural network and face key point detection as recited in claim 1, wherein the method comprises the following steps: in the step 1, a Dlib frame is adopted to detect the key points of the human face of the post-cosmetic portrait image, and 68 key point coordinates { (x) of the human face of the post-cosmetic portrait image are obtained 1 ,y 1 ),(x 2 ,y 2 ),(x 3 ,y 3 )…(x 68 ,y 68 ) -a }; then for the key point coordinates of the mouth { (x) 49 ,y 49 ),(x 50 ,y 50 )…(x 68 ,y 68 ) Correcting, namely selecting a correct key point coordinate of a mouth, extracting color values of three channels of the key point coordinate RGB respectively, and then moving coordinates of other inaccurate key points until the color difference value between the key point coordinate and the selected correct key point coordinate is smaller than a preset threshold value to obtain corrected 68 face key point coordinates { (x) 1 ,y 1 ),(x 2 ,y 2 ),(x 3 ,y 3 )…(x 68 ,y 68 ) And save these keypoints.
3. The method of claim 2, wherein selecting a correct key point coordinate for the mouth is randomly selecting one of the key points for locating the upper lip as the correct key point coordinate.
4. The depth convolutional neural network and face keypoints-based system of claim 1The detected face changing method after make-up is characterized in that: in the step 2, the method is obtained by using the following formula
Figure FDA0003936688540000021
Or->
Figure FDA0003936688540000022
RGB channel values for each pixel point: />
Figure FDA0003936688540000023
Wherein the method comprises the steps of
Figure FDA0003936688540000024
Pixel R channel of position guide for post-cosmetic portrait or target portrait, < >>
Figure FDA0003936688540000025
Pixel point G channel of position guide of post-cosmetic portrait or target portrait, +.>
Figure FDA0003936688540000026
The method comprises the steps that a channel B is a pixel point of a position guide of a post-cosmetic portrait image or a target portrait image, w is the width of the post-cosmetic portrait image or the target portrait image, h is the height of the post-cosmetic portrait image or the target portrait image, x and y are x coordinates and y coordinates corresponding to each pixel point in the post-cosmetic portrait image or the target portrait image, meanwhile, the x coordinates correspond to a red channel, the y coordinates correspond to a green channel, and the blue channel is set to 0;
the values of the three channels are then saved and generated using the OpenCV library
Figure FDA0003936688540000027
And->
Figure FDA0003936688540000028
5. The post-cosmetic face-changing method based on deep convolutional neural network and face key point detection as recited in claim 1, wherein the method comprises the following steps: in the step 4, the following formula is used to obtain
Figure FDA0003936688540000029
And->
Figure FDA00039366885400000210
Figure FDA00039366885400000211
Wherein orig is gray The target portrait or the grey-scale image converted from the post-makeup portrait, the bluImg is the blurred image of the post-makeup portrait or the target portrait obtained in the step 3, G app An appearance guide for a post-cosmetic portrait or a target portrait;
then use OpencCV library modifications
Figure FDA00039366885400000212
To be equal to->
Figure FDA00039366885400000213
Is matched with the histogram of the object.
6. The post-cosmetic face-changing method based on deep convolutional neural network and face key point detection as recited in claim 4, wherein the method comprises the following steps: in the step 5, coordinates of example pixels of the best matching patterns of the red and green channel values of the post-makeup portrait map are stored in a lookupcube, wherein the coordinates are obtained by:
firstly, calculating RGB channel error values between pixel points of two position guides according to the following steps of traversing all pixel points of the position guides of the post-makeup portrait image and the target portrait image:
Figure FDA0003936688540000031
wherein the method comprises the steps of
Figure FDA0003936688540000032
Pixel point G channel for position guide of post-cosmetic portrait map, +.>
Figure FDA0003936688540000033
Pixel point G channel of position guide for target portrait map, +.>
Figure FDA0003936688540000034
Pixel R channel of position guide for post-cosmetic portrait>
Figure FDA0003936688540000035
The pixel point R channel of the position guide of the target portrait image, and alpha and beta are weight values;
then, after all the pixel points are guided by the pixel points of the position guide of each target portrait image after matching the position guide of the portrait image after makeup, the pixel in the calculated minimum error value is taken as an example pixel of the optimal matching pattern, and the pixel is corresponding to the example pixel
Figure FDA0003936688540000036
The coordinates of the corresponding pixel are stored in a lookupcube while the corresponding +.>
Figure FDA0003936688540000037
Together with the gray scale intensities in the bookup.
7. The post-cosmetic face-changing method based on deep convolutional neural network and face key point detection as recited in claim 1, wherein the method comprises the following steps: in the step 6, the face under the eyebrow and the eyebrow is obtained through the face key point detection, the face of the forehead part is obtained through the skin detection, and the first contour of the face is obtained through the combination.
8. The post-cosmetic face-changing method based on deep convolutional neural network and face key point detection as recited in claim 6, wherein the method comprises the following steps: in the step 8, the StyleBlit algorithm is used to make a post-makeup face change, and the distance between the coordinates of the post-makeup portrait drawing and the target portrait drawing is defined as:
Figure FDA0003936688540000038
wherein P is a post-makeup portrait drawing, O is a target portrait drawing, and alpha 'and beta' are weights;
Figure FDA0003936688540000039
and->
Figure FDA00039366885400000310
Obtained from a lookupcube, taking the set of pixels of the post-makeup image as { P ] 1 ,P 2 ,P 3, …P n The pixel set of the target image is { O } 1 ,O 2 ,O 3, …O n Then render the target picture O: traversing all pixels { O } of the target image 1 ,O 2 ,…O n And after finding the nearest pixel point in the corresponding post-makeup picture, calculating the error rate E between the two pixel points, and copying the pixels of the post-makeup picture into the target picture if the E is smaller than a preset threshold value to finally obtain StylizedImg. />
CN202211405068.9A 2022-11-10 2022-11-10 Post-makeup face changing method based on deep convolutional neural network and face key point detection Pending CN116052240A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211405068.9A CN116052240A (en) 2022-11-10 2022-11-10 Post-makeup face changing method based on deep convolutional neural network and face key point detection

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211405068.9A CN116052240A (en) 2022-11-10 2022-11-10 Post-makeup face changing method based on deep convolutional neural network and face key point detection

Publications (1)

Publication Number Publication Date
CN116052240A true CN116052240A (en) 2023-05-02

Family

ID=86130873

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211405068.9A Pending CN116052240A (en) 2022-11-10 2022-11-10 Post-makeup face changing method based on deep convolutional neural network and face key point detection

Country Status (1)

Country Link
CN (1) CN116052240A (en)

Similar Documents

Publication Publication Date Title
CN109376582B (en) Interactive face cartoon method based on generation of confrontation network
CN103914699B (en) A kind of method of the image enhaucament of the automatic lip gloss based on color space
CN103456010B (en) A kind of human face cartoon generating method of feature based point location
CN101339606B (en) Human face critical organ contour characteristic points positioning and tracking method and device
CN110175558B (en) Face key point detection method and device, computing equipment and storage medium
CN108229278B (en) Face image processing method and device and electronic equipment
CN105139438B (en) video human face cartoon generation method
US20200034996A1 (en) Image processing method, apparatus, terminal, and storage medium
WO2018036462A1 (en) Image segmentation method, computer apparatus, and computer storage medium
US20220044491A1 (en) Three-dimensional face model generation method and apparatus, device, and medium
CN107507216B (en) Method and device for replacing local area in image and storage medium
CN106920274A (en) Mobile terminal 2D key points rapid translating is the human face model building of 3D fusion deformations
CN109657612B (en) Quality sorting system based on facial image features and application method thereof
CN109829930A (en) Face image processing process, device, computer equipment and readable storage medium storing program for executing
CN110096925A (en) Enhancement Method, acquisition methods and the device of Facial Expression Image
KR102386642B1 (en) Image processing method and apparatus, electronic device and storage medium
CN116997933A (en) Method and system for constructing facial position map
CN116583878A (en) Method and system for personalizing 3D head model deformation
CN108510500A (en) A kind of hair figure layer process method and system of the virtual figure image based on face complexion detection
Yuan et al. Watershed-based superpixels with global and local boundary marching
CN117157673A (en) Method and system for forming personalized 3D head and face models
KR20230085931A (en) Method and system for extracting color from face images
CN115689869A (en) Video makeup migration method and system
CN111243051A (en) Portrait photo-based stroke generating method, system and storage medium
CN105574814A (en) Portrait paper-cut special effect generation method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination