CN110363116A

CN110363116A - Irregular face antidote, system and medium based on GLD-GAN

Info

Publication number: CN110363116A
Application number: CN201910575810.2A
Authority: CN
Inventors: 孙锬锋; 蒋兴浩; 许可; 陆翼龙
Original assignee: Shanghai Jiaotong University
Current assignee: Shanghai Jiaotong University
Priority date: 2019-06-28
Filing date: 2019-06-28
Publication date: 2019-10-22
Anticipated expiration: 2039-06-28
Also published as: CN110363116B

Abstract

The present invention provides a kind of irregular face antidote, system and medium based on GLD-GAN, comprising: step S1: identifying and the face in interception image, obtains face figure P₀；Step S2: according to the face figure P of acquisition₀, background redundancy is rejected using the image Segmentation Technology based on CRF-RNN；Step S3: image classification is carried out using Inception model, side face is divided into multiple classes according to angle；Step S4: the GLD-GAN network model of each class of training.The present invention will generate regularization of the confrontation application of net in side face and study, learn to realize the conversion of individual side face to corresponding positive face by confrontation, simultaneously by side face rule method in conjunction with super-resolution rebuilding technology, the end-to-end mapping of the side face portrait under multi-pose, different illumination conditions to the positive face view of high quality is realized.

Description

Irregular face antidote, system and medium based on GLD-GAN

Technical field

The present invention relates to image identification technical fields, and in particular, to the irregular face correction side based on GLD-GAN Method, system and medium.

Background technique

Recognition of face is increasingly mature as a technology, and the application in reality is also very extensive, such as security protection Monitoring, attendance are registered etc..It is applicable in mostly however, more, current applications is also compared in limitation suffered by existing recognition of face In face that is clear, facing, once there is illumination variation or face is deflected or even blocked, existing face recognition technology Accuracy rate begin to decline to a great extent.Therefore, recognition of face is carried out under conditions of complicated difficult remain one rich in challenge The problem of property.The invention proposes a kind of irregular face correction algorithm based on GLD-GAN, by correcting irregular face At clearly positive face, thus the face recognition accuracy rate under improving complex environment.

Related patents published at present are not more, wherein publication number and publication date be respectively CN108510061A, 2018.09.07, the China of entitled " method of more positive faces of monitor video human face segmentation of confrontation network is generated based on condition " is specially Benefit provides a kind of method of positive face of the human face segmentation under more monitor videos, this method need to acquire in monitor video without about Beam deflection angle face and positive face filter out positive face, therefore its synthesis inputted to realize positive face for relying on multiple side faces；Separately Outside, it is " a kind of based on generating confrontation network that publication number and publication date are respectively CN108537743A, 2018.09.14, entitled Face-image Enhancement Method ", the thinking this method provides another kind based on the irregular face correction of generation confrontation network implementations, It has mainly used the dense facial alignment schemes of 3D to pre-process the face-image of many attitude, then places into network It is trained, it is true that the image that this method obtains can retain primary light photograph, visual perception, and remains original identity letter Breath.But due to the facial alignment schemes dense which employs 3D, the speed of service is partially slow；Simultaneously when wide-angle occur inclined for face When the case where turning or blocking, accuracy rate also will appear apparent decline.In addition there are a kind of patent based on mobile terminal, Publication number and publication date are respectively CN108491775A, 2018.09.04, entitled " a kind of image correcting method and mobile end End ", which is intended to improve the display effect of shooting works, and the pupil in portrait is carried out towards upper amendment.Though this method So there is certain correction to irregular face, but the information of left/right eye would generally be lost in the case where wide-angle side face, this Kind method just fails.

The present invention has studied the rule method of side face portrait, i.e., from non-frontal posture and illumination condition it is unsatisfactory two Dimension facial image reconstruct desired light shine under the conditions of positive facial image, specifically include portrait posture, illumination correction and Automatic compensation to face texture missing, it is therefore an objective to realize that the end of single side face portrait to the front portrait view of multi-angle is arrived End mapping, solves the problems, such as recognition of face complicated under actual environment to a certain extent, further promotes face in practical application and knows Other accuracy.

Patent document CN106874861A (application number: 201710053937.9) discloses a kind of face antidote and is System, method include: setting one standard face, obtain standard point, the standard point includes at least: left eye ball center, right eye ball center, 5 characteristic points of nose, the left corners of the mouth and the right corners of the mouth；Different characteristic point is selected from above-mentioned standard point, and the difference of input is deflected The face of angle corrects the face of more deflection angles using the different modes for calculating similarity transformation parameter；Work as deflection angle In the first deflection angle threshold value, then the parameter of similarity transformation is calculated using the coordinate based on two eyeball centers of left and right；Work as deflection Angle is then calculated using the coordinate of the centre based on two eyeballs and the coordinate among two corners of the mouths similar in the second deflection angle threshold value The parameter of transformation；When deflection angle is in third deflection angle threshold value, then using based on unilateral characteristic point calculating similarity transformation parameter.

Summary of the invention

For the defects in the prior art, the object of the present invention is to provide a kind of, and the irregular face based on GLD-GAN is rectified Correction method, system and medium.

A kind of irregular face antidote based on GLD-GAN provided according to the present invention, comprising:

Step S1: identifying and the face in interception image, obtains face figure P₀；

Step S2: according to the face figure P of acquisition₀, background redundancy is rejected using the image Segmentation Technology based on CRF-RNN；

Step S3: image classification is carried out using Inception model, side face is divided into multiple classes according to angle；

Step S4: the GLD-GAN network model of each class of training；

Step S5: side face image is inputted into GLD-GAN network model, obtains face image a；

Step S6: the TP-GAN that side face image is input to open source is obtained into face image b；

Step S7: face image a and b are merged according to the ratio of a:b=3:1, and the positive face of fusion is replaced in image b Positive face obtains face image c；

Step S8: the face image c that adjustment obtains is optimized using bilateral filtering algorithm；

Step S9: training super-resolution network SRGAN；

Step S10: utilize SRGAN by face image c super-resolution processing.

Preferably, the step S1 includes:

Step S101: whether there is face in detection image: being continued to execute if so, then entering step S102, otherwise, jump out and work as Preceding circulation simultaneously reports the mistake that face is not detected；

Step S102: its face coordinate and corresponding bounding box (bounding box) are calculated according to detection；

Step S103: positioning the position of nose in calculated face coordinate first, and ensures that nose is the people of interception The central axes of face figure, the position of bounding box is adjusted according to central axes, and intercepts bounding box as new face figure P₀；

The step S102:

Using the open source library face_recognition based on dlib, image and discriminance analysis face are read, and returns to it The coordinate and bounding box of face；

The step S103, the position that bounding box is adjusted according to central axes:

Remember that right is the right margin coordinate of former bounding box, left is the left margin coordinate of former bounding box, remembers that former central axes are sat It marks x=(right-left)/2, remembers that the abscissa of nose is mid, if x > mid, right=right- (x-mid), left is not Become；If x < mid, left=left+ (mid-x), right is constant；If x=mid, without operation.

Preferably, the step S2 includes:

Step S201: using open source MODEL C RF-RNN, by the face figure P of acquisition₀Image segmentation is carried out, a figure is obtained As the chromaticity diagram P of segmentation₁；

Step S202: traversal P₀With P₁Each of pixel, if in P₁In pixel correspond to the first color, then make P₀In corresponding pixel become the second color, to achieve the effect that divide facial image, obtaining will be after face and background segment , the only image P of face₂；

The step S3 includes:

All training images for being used for side face correction are divided into multiple classes according to the angle of side face, are then placed in Classification based training is carried out in Inception V3 model, the Inception V3 network after being trained；

The input of the Inception V3 network is a facial image, when output facial image relative to positive face institute partially The angle turned；

The step S4 includes:

Step S401: preparing the first training, test and validation data set, and first trains, in test and validation data set Image one opens the corresponding positive face figure of side face figure；

Step S402: the corresponding positive face figure of side face figure in the first training, test and validation data set is spliced one It opens on figure；

Step S403: a GLD-GAN network model, GLD- are trained for each class of the angular divisions according to side face The loss function of GAN network are as follows:

Wherein,

G^*Indicate the loss function of GLD-GAN network

Indicate the loss of GAN network itself, i.e., by Min-max to reach one A stable locally optimal solution；

λ₁Indicate the parameter of L1 loss；

λ₂Indicate the parameter of fuzzy loss；

λ₃Indicate that identity retains the parameter of loss；

Indicate L1 loss, i.e. norm loss；

Indicate fuzzy loss；

Indicate that identity retains loss；

G indicates to generate the loss function of network；

D indicates the loss function of differentiation network, i.e., the part between generation image and truthful data ground truth Loss function；

Min-max is the game between production confrontation network to reach a stable locally optimal solution；

L1 indicates to generate the whole loss function between image and truthful data ground truth；

L2 indicates to generate the loss of sharpness function of image；

L3 indicates to generate the identity information loss function between image and truthful data ground truth.

Preferably, the step S5 includes:

Step S501: according to Inception V3 network, by the facial image P of input₂Assign to the GLD-GAN net of corresponding class In network model；

Step S502: in GLD-GAN network model, by the facial image P of input₂It is converted into the picture of pre-set dimension, And export corresponding face image P₃；

The step S6 includes:

In open source TP-GAN network model, by the facial image P of input₂It is converted into the picture of pre-set dimension, and is exported Corresponding face image P₄；

The step S7 includes:

Step S701: by face image P₃、P₄Face fusion is carried out according to the first preset ratio, face fusion algorithm is based on Increase income library dlib and OpenCV, and wherein dlib is used to extract the characteristic model of face, reuses OpenCV proportionally for feature Model is merged, and fusion face is obtained；

Step S702: original face image P is replaced using obtained fusion face₄, obtain image P₅, face replacement calculation Method is equally based on dlib and OpenCV；

The fusion face that the utilization obtains replaces original face image P₄:

Convex closure is calculated using the facial key point of dlib detection, then to key point progress Delaunay Triangulation, and by Affine transformation is mapped to positive face figure P for face is merged₄On.

Preferably, the step S8 includes:

Using two-sided filter, by image P₅It optimizes, obtains image P₆；

The two-sided filter generates distance template using two-dimensional Gaussian function, generates codomain mould using one-dimensional Gaussian function Plate, distance template coefficient are as follows:

Wherein,

Indicate distance template coefficient；

K, l are the centre coordinate of template window；

I, j are the coordinate of other coefficients of template window；

σ_dFor the standard deviation of its Gaussian function；

The generation formula of codomain coefficients are as follows:

Wherein,

Indicate codomain coefficients；

Indicates coordinate is i, the pixel value of the point of j；

F (k, l) indicates coordinate is k, the pixel value of the point of l；

σ_rIndicate the standard deviation of its Gaussian function；

Wherein function f represents image to be processed；

To sum up, the template of two-sided filter are as follows:

Wherein,

Indicate the template of two-sided filter；

The step S9 includes:

Prepare the second training, test and validation data set, training super-resolution network model SRGAN；

Second training, the low-resolution image in test and validation data set and high-definition picture size be the Two preset ratios.

Preferably, the step S10 includes:

By image P₆Super-resolution network model in super-resolution network model SRGAN after input training, after training SRGAN carries out super-resolution processing to input picture, improves the resolution ratio of image to default resolution ratio, and export final image P₇；

Wherein, the loss function of the generation network of SRGAN network are as follows:

Wherein,

Indicate the loss function that network is generated in SRGAN network；

W_i,jIndicate the width of the VGG-19 network characterization figure of the i-th row jth column；

H_i,jIndicate the height of the VGG-19 network characterization figure of the i-th row jth column；

φ_i,jIndicate the spy obtained before i-th of maximum pond layer of the jth layer convolution (after activation) of VGG-19 network Sign figure；

I^HRIndicate high-definition picture；

(I^HR)_x,yIndicate that coordinate is x, the pixel of y in high-definition picture；

Indicate the corresponding characteristic pattern of image；

I^LRIndicate low-resolution image；

Indicate that coordinate is x, the pixel of y in the characteristic pattern of low-resolution image.

A kind of irregular face correction system based on GLD-GAN provided according to the present invention, comprising:

Module S1: identifying and the face in interception image, obtains face figure P₀；

Module S2: according to the face figure P of acquisition₀, background redundancy is rejected using the image Segmentation Technology based on CRF-RNN；

Module S3: image classification is carried out using Inception model, side face is divided into multiple classes according to angle；

Module S4: the GLD-GAN network model of each class of training；

Module S5: side face image is inputted into GLD-GAN network model, obtains face image a；

Module S6: the TP-GAN that side face image is input to open source is obtained into face image b；

Module S7: merging face image a and b according to preset ratio, and the positive face of fusion replaced the positive face in image b, Obtain face image c；

Module S8: the face image c that adjustment obtains is optimized using bilateral filtering algorithm；

Module S9: training super-resolution network SRGAN；

Module S10: utilize SRGAN by face image c super-resolution processing.

Preferably, the module S1 includes:

Module S101: whether there is face in detection image: if so, otherwise then calling module S102 jumps out previous cycle simultaneously The mistake of face is not detected in report；

Module S102: its face coordinate and corresponding bounding box (bounding box) are calculated according to detection；

Module S103: positioning the position of nose in calculated face coordinate first, and ensures that nose is the people of interception The central axes of face figure, the position of bounding box is adjusted according to central axes, and intercepts bounding box as new face figure P₀；

The module S102:

The module S103, the position that bounding box is adjusted according to central axes:

Remember that right is the right margin coordinate of former bounding box, left is the left margin coordinate of former bounding box, remembers that former central axes are sat It marks x=(right-left)/2, remembers that the abscissa of nose is mid, if x > mid, right=right- (x-mid), left is not Become；If x < mid, left=left+ (mid-x), right is constant；If x=mid, without operation；

The module S2 includes:

Module S201: using open source MODEL C RF-RNN, by the face figure P of acquisition₀Image segmentation is carried out, a figure is obtained As the chromaticity diagram P of segmentation₁；

Module S202: traversal P₀With P₁Each of pixel, if in P₁In pixel correspond to the first color, then make P₀In corresponding pixel become the second color, to achieve the effect that divide facial image, obtaining will be after face and background segment , the only image P of face₂；

The module S3 includes:

The module S4 includes:

Module S401: preparing the first training, test and validation data set, and first trains, in test and validation data set Image one opens the corresponding positive face figure of side face figure；

Module S402: the corresponding positive face figure of side face figure in the first training, test and validation data set is spliced one It opens on figure；

Module S403: a GLD-GAN network model, GLD- are trained for each class of the angular divisions according to side face The loss function of GAN network are as follows:

Wherein,

G^*Indicate the loss function of GLD-GAN network

λ₁Indicate the parameter of L1 loss；

λ₂Indicate the parameter of fuzzy loss；

λ₃Indicate that identity retains the parameter of loss；

Indicate L1 loss, i.e. norm loss；

Indicate fuzzy loss；

Indicate that identity retains loss；

G indicates to generate the loss function of network；

L2 indicates to generate the loss of sharpness function of image；

L3 indicates to generate the identity information loss function between image and truthful data ground truth；

The module S5 includes:

Module S501: according to Inception V3 network, by the facial image P of input₂Assign to the GLD-GAN net of corresponding class In network model；

Module S502: in GLD-GAN network model, by the facial image P of input₂It is converted into the picture of pre-set dimension, And export corresponding face image P₃；

The module S6 includes:

The module S7 includes:

Module S701: by face image P₃、P₄Face fusion is carried out according to the first preset ratio, face fusion algorithm is based on Increase income library dlib and OpenCV, and wherein dlib is used to extract the characteristic model of face, reuses OpenCV proportionally for feature Model is merged, and fusion face is obtained；

Module S702: original face image P is replaced using obtained fusion face₄, obtain image P₅, face replacement calculation Method is equally based on dlib and OpenCV；

The fusion face that the utilization obtains replaces original face image P₄:

Preferably, the module S8 includes:

Using two-sided filter, by image P₅It optimizes, obtains image P₆；

Wherein,

Indicate distance template coefficient；

K, l are the centre coordinate of template window；

I, j are the coordinate of other coefficients of template window；

σ_dFor the standard deviation of its Gaussian function；

The generation formula of codomain coefficients are as follows:

Wherein,

Indicate codomain coefficients；

Indicates coordinate is i, the pixel value of the point of j；

F (k, l) indicates coordinate is k, the pixel value of the point of l；

σ_rIndicate the standard deviation of its Gaussian function；

Wherein function f represents image to be processed；

To sum up, the template of two-sided filter are as follows:

Wherein,

Indicate the template of two-sided filter；

The module S9 includes:

Second training, the low-resolution image in test and validation data set and high-definition picture size be the Two preset ratios；

The module S10 includes:

Wherein,

Indicate the loss function that network is generated in SRGAN network；

I^HRIndicate high-definition picture；

Indicate the corresponding characteristic pattern of image；

I^LRIndicate low-resolution image；

A kind of computer readable storage medium for being stored with computer program provided according to the present invention, which is characterized in that Described in any item irregular face corrections based on GLD-GAN among the above are realized when the computer program is executed by processor The step of method.

Compared with prior art, the present invention have it is following the utility model has the advantages that

1, the present invention will generate regularization of the confrontation application of net in side face and study, and realize individual by confrontation study Conversion of the side face to corresponding positive face.

2, the present invention has merged TP-GAN network and GLD-GAN face generated, on the basis of side face regularization, protects Stay facial identity information feature.

3, present invention employs two-sided filters, will generate image and are filtered, eliminate the interference of noise.

4, the present invention realizes multi-pose, different illumination items by side face rule method in conjunction with super-resolution rebuilding technology The end-to-end mapping of side face portrait under part to the positive face view of high quality.

5, the present invention will generate regularization of the confrontation application of net in side face and study, and realize individual by confrontation study Side face to corresponding positive face conversion, while by side face rule method with super-resolution rebuilding technology in conjunction with, realization multi-pose, no With the side face portrait under the conditions of illumination to the end-to-end mapping of the positive face view of high quality.

Detailed description of the invention

Upon reading the detailed description of non-limiting embodiments with reference to the following drawings, other feature of the invention, Objects and advantages will become more apparent upon:

Fig. 1 is the flow diagram of the irregular face antidote provided by the invention based on GLD-GAN.

Specific embodiment

The present invention is described in detail combined with specific embodiments below.Following embodiment will be helpful to the technology of this field Personnel further understand the present invention, but the invention is not limited in any way.It should be pointed out that the ordinary skill of this field For personnel, without departing from the inventive concept of the premise, several changes and improvements can also be made.These belong to the present invention Protection scope.

Step S4: the GLD-GAN network model of each class of training；

Step S9: training super-resolution network SRGAN；

Step S10: utilize SRGAN by face image c super-resolution processing.

Specifically, the step S1 includes:

The step S102:

Specifically, the step S2 includes:

The step S3 includes:

The step S4 includes:

Wherein,

G^*Indicate the loss function of GLD-GAN network

λ₁Indicate the parameter of L1 loss；

λ₂Indicate the parameter of fuzzy loss；

λ₃Indicate that identity retains the parameter of loss；

Indicate L1 loss, i.e. norm loss；

Indicate fuzzy loss；

Indicate that identity retains loss；

G indicates to generate the loss function of network；

L2 indicates to generate the loss of sharpness function of image；

Specifically, the step S5 includes:

The step S6 includes:

The step S7 includes:

The fusion face that the utilization obtains replaces original face image P₄:

Specifically, the step S8 includes:

Using two-sided filter, by image P₅It optimizes, obtains image P₆；

Wherein,

Indicate distance template coefficient；

K, l are the centre coordinate of template window；

I, j are the coordinate of other coefficients of template window；

σ_dFor the standard deviation of its Gaussian function；

The generation formula of codomain coefficients are as follows:

Wherein,

Indicate codomain coefficients；

Indicates coordinate is i, the pixel value of the point of j；

F (k, l) indicates coordinate is k, the pixel value of the point of l；

σ_rIndicate the standard deviation of its Gaussian function；

Wherein function f represents image to be processed；

To sum up, the template of two-sided filter are as follows:

Wherein,

Indicate the template of two-sided filter；

The step S9 includes:

Specifically, the step S10 includes:

Wherein,

Indicate the loss function that network is generated in SRGAN network；

I^HRIndicate high-definition picture；

Indicate the corresponding characteristic pattern of image；

I^LRIndicate low-resolution image；

Irregular face correction system provided by the invention based on GLD-GAN, can give through the invention based on The step process of the irregular face antidote of GLD-GAN is realized.Those skilled in the art described can will be based on GLD-GAN Irregular face antidote, be interpreted as a preference of the irregular face correction system based on GLD-GAN.

Module S4: the GLD-GAN network model of each class of training；

Module S9: training super-resolution network SRGAN；

Module S10: utilize SRGAN by face image c super-resolution processing.

Specifically, the module S1 includes:

The module S102:

The module S2 includes:

The module S3 includes:

The module S4 includes:

Wherein,

G^*Indicate the loss function of GLD-GAN network

λ₁Indicate the parameter of L1 loss；

λ₂Indicate the parameter of fuzzy loss；

λ₃Indicate that identity retains the parameter of loss；

Indicate L1 loss, i.e. norm loss；

Indicate fuzzy loss；

Indicate that identity retains loss；

G indicates to generate the loss function of network；

L2 indicates to generate the loss of sharpness function of image；

The module S5 includes:

The module S6 includes:

The module S7 includes:

The fusion face that the utilization obtains replaces original face image P₄:

Specifically, the module S8 includes:

Using two-sided filter, by image P₅It optimizes, obtains image P₆；

Wherein,

Indicate distance template coefficient；

K, l are the centre coordinate of template window；

I, j are the coordinate of other coefficients of template window；

σ_dFor the standard deviation of its Gaussian function；

The generation formula of codomain coefficients are as follows:

Wherein,

Indicate codomain coefficients；

Indicates coordinate is i, the pixel value of the point of j；

F (k, l) indicates coordinate is k, the pixel value of the point of l；

σ_rIndicate the standard deviation of its Gaussian function；

Wherein function f represents image to be processed；

To sum up, the template of two-sided filter are as follows:

Wherein,

Indicate the template of two-sided filter；

The module S9 includes:

The module S10 includes:

Wherein,

Indicate the loss function that network is generated in SRGAN network；

I^HRIndicate high-definition picture；

Indicate the corresponding characteristic pattern of image；

I^LRIndicate low-resolution image；

Below by preference, the present invention is more specifically illustrated.

Preference 1:

As shown in Figure 1, the irregular face antidote based on GLD-GAN provided according to the present invention, including walk as follows It is rapid:

Step S1: identifying and the face in interception image；

Step S2: background redundancy is rejected using the image Segmentation Technology based on CRF-RNN；

Step S3: image classification is carried out using Inception model, side face is classified according to angle

Step S4: the GLD-GAN network model of each angle of training；

Step S5: side face image input GLD-GAN is obtained into face image a；

Step S9: training super-resolution network SRGAN；

Step S10: utilize SRGAN by face image c super-resolution processing.

The step S1 includes:

Step S1.1: whether there is face in detection image；

Step S1.2: if so, calculating the coordinate and bounding box of its face；

Specifically, using the open source library face_recognition based on dlib, image and discriminance analysis face are read, and Return to the coordinate and bounding box of its face；

Step S1.3: positioning the position of nose first in face coordinate, and ensures that nose is in the face figure intercepted Axis.The position of bounding box is adjusted according to central axes, and intercepts bounding box as new face figure P₀。

Specifically, remember original central axes x=(right-left)/2, remember that the abscissa of nose is mid, if x > mid, Length=mid-left, right=right- (x-mid), left is constant；If x < mid, length=right-mid, Left=left+ (mid-x), right is constant.

The step S2 includes:

Step S2.1: using open source MODEL C RF-RNN, the face figure P obtained in step 1₀Image segmentation is carried out, is obtained To the chromaticity diagram P of an image segmentation₁。

Step S2.2: traversal P₀With P₁Each of pixel, if in P₁In pixel correspond to red (192,0,0), Then make P₀In corresponding pixel become ater (0,0,0), thus achieve the effect that divide facial image；

Step S2.3: saving to handle according to step 2.2 and obtain, after face and background segment, the only figure of face As P₂。

The step S3 includes: to carry out all training images for being used for side face correction manually according to the angle of its side face Mark, is divided into 0,15,30,45,60,75,90 degree of seven classes, is then placed in Inception V3 model and carries out classification based training；

Specifically, prepare corresponding seven classes of seven files, corresponding training data is respectively placed in this seven files In, then simultaneously train classification models are read using Inception V3.Into the model, which all can one image of every input Return to a possibility that image belongs to each class.Only demand returns to the maximum result of possibility in this project.

The step S4 includes:

Step S4.1: prepare corresponding training, test and validation data set；

The data set of step S4.2:GLD-GAN network requirement is one-to-one, the i.e. corresponding positive face of a side face figure Figure, then by this two image mosaics on a figure；

Step S4.3: due to a total of seven classes, so requiring one GLD-GAN network mould of training for each class Type；The loss function of GLD-GAN network is

Wherein,

G^*Indicate the loss function of GLD-GAN network

Indicate the loss of GAN network itself, i.e., by Min-max to great achievement one A stable locally optimal solution.

λ₁Indicate the parameter of L1 loss

λ₂Indicate the parameter of fuzzy loss

λ₃Indicate that identity retains the parameter of loss

Indicate L1 loss, i.e. norm loss

Indicate fuzzy loss

Indicate that identity retains loss

Wherein G represent generate network loss function, D represent differentiate network loss function, i.e., generation image with The loss function of part between groundtruth, Min-max are the games between production confrontation network to reach One stable locally optimal solution, L₁Represent the whole loss function generated between image and ground truth, L₂It represents Generate image loss of sharpness function, L₃Represent the identity information loss letter generated between image and ground truth Number.

The step S5 includes:

Step S5.1: according to Inception V3 network, by the facial image P of input₂Assign to the GLD-GAN of corresponding angle In network.

Step S5.2: the facial image P no matter inputted₂Size is how many, all can be by GLD-GAN network model Resize and exports corresponding face image P at the picture of 256*256 size₃。

The step S6 includes: the facial image P no matter inputted₂Size is how many, in open source TP-GAN network model Will be by resize at the picture of 256*256 size, and export corresponding face image P₄。

The step S7 includes:

Step S7.1: by face image P₃: P₄Face fusion is carried out according to the ratio of 3:1, face fusion algorithm is based on open source Library dlib and OpenCV.Wherein dlib is used to extract the characteristic model of face, reuses OpenCV proportionally for characteristic model It is merged.

Step S7.2: original face image P is replaced using obtained fusion face₄, obtain image P₅.Face replacement is calculated Method is equally based on dlib and OpenCV.

Specifically, convex closure is calculated using the facial key point of dlib detection, then Delaunay triangle is carried out to key point and is cutd open Point, and positive face figure P is mapped to by face is merged by affine transformation₄On.Affine transformation formula is

Wherein, (x, y) is the characteristic point coordinate for merging face, and (x ', y ') is after converting in figure P₄On respective coordinates, M It is the multi-mode operations such as affine transformation matrix, including translation, rotation, scaling, m₀₀~m₁₂Refer respectively to the parameter of affine variation.

The step S8 includes: using bilateral filtering algorithm, by image P₅It optimizes, removes noise to a certain extent, Obtain image P₆.Two-sided filter generates distance template using two-dimensional Gaussian function, generates codomain mould using one-dimensional Gaussian function Plate.Distance template coefficient is

Wherein,

Indicate distance template coefficient；

K, l are the centre coordinate of template window；

I, j are the coordinate of other coefficients of template window；

σ_dFor the standard deviation of its Gaussian function.The generation formula of codomain coefficients is

Wherein,

Indicate codomain coefficients

Indicates coordinate is i, the pixel value of the point of j

F (k, l) indicates coordinate is k, the pixel value of the point of l

σ_rIndicate the standard deviation of its Gaussian function

Wherein function f represents image to be processed.To sum up, the template of two-sided filter is

Wherein,

Indicate the template of two-sided filter

The step S9 includes: to prepare corresponding training, test and validation data set；Low-resolution image and high-resolution The size of image is 1:4, in a practical situation it is not absolutely required to there is ready-made low-resolution image, can directly be passed through It compresses high-resolution and obtains low-resolution image

The step S10 includes: by image P₆It inputs in SRGAN network model, super-resolution model can be to input picture Super-resolution processing is carried out, greatly improves the resolution ratio of image to 1024*1024, and export final image P₇.Wherein SRGAN The loss function of the generation network of network is

Wherein,

Indicate the loss function that network is generated in SRGAN network

W_i,jIndicate the width of the VGG-19 network characterization figure of the i-th row jth column

H_i,jIndicate the height of the VGG-19 network characterization figure of the i-th row jth column

φ_i,jIndicate the spy obtained before i-th of maximum pond layer of the jth layer convolution (after activation) of VGG-19 network Sign figure

I^HRIndicate high-definition picture

(I^HR)_x,yIndicate that coordinate is x, the pixel of y in high-definition picture

Indicate the corresponding characteristic pattern of image

I^LRIndicate low-resolution image

Indicate that coordinate is x, the pixel of y in the characteristic pattern of low-resolution image

The loss function is exactly the loss pixel-by-pixel to the characteristic pattern of a certain layer as content loss in simple terms, rather than Finally export the loss pixel-by-pixel of result, the manifold space where the image that can learn in this way.And the damage of the differentiation network of SRGAN Lose function be actually negative logarithm summation, be conducive to train in this way.

Preference 2:

A kind of irregular face antidote based on the GLD-GAN independently studied, includes the following steps:

Step S1: identifying and the face in interception image；

Step S4: the GLD-GAN network model of each angle of training；

Step S5: side face image input GLD-GAN is obtained into face image a；

Step S9: training super-resolution network SRGAN；

Step S10: utilize SRGAN by face image c super-resolution processing.

The step S1 includes:

Step S1.1: whether there is face in detection image；

Step S1.2: if so, calculating the coordinate and bounding box of its face；

The step S2 includes:

Step S2.2: traversal P₀With P₁Each of pixel, if in P₁In pixel correspond to red (192,0,0), Then make P₀In corresponding pixel become ater (0,0,0)

The step S3 includes: that all training images for being used for side face correction are divided into 0 according to the angle of its side face, 15,30,45,60,75,90 degree of seven class is then placed in Inception V3 model and carries out classification based training；

The step S4 includes:

Step S4.1: prepare corresponding training, test and validation data set；

Step S4.2: due to a total of seven classes, so requiring one GLD-GAN network mould of training for each class Type；

The step S5 includes:

The step S7 includes:

Step S7.1: by face image P₃: P₄Face fusion is carried out according to the ratio of 3:1, face fusion algorithm is based on open source Library dlib and OpenCV.

The step S8 includes: using bilateral filtering algorithm, by image P₅It optimizes, removes noise to a certain extent, Obtain image P₆。

The step S9 includes: to prepare corresponding training, test and validation data set；

The step S10 includes: by image P₆It inputs in SRGAN network model, super-resolution model can be to input picture Super-resolution processing is carried out, greatly improves the resolution ratio of image to 1024*1024

In the description of the present application, it is to be understood that term " on ", "front", "rear", "left", "right", " is erected at "lower" Directly ", the orientation or positional relationship of the instructions such as "horizontal", "top", "bottom", "inner", "outside" is orientation based on the figure or position Relationship is set, description the application is merely for convenience of and simplifies description, rather than the device or element of indication or suggestion meaning are necessary It with specific orientation, is constructed and operated in a specific orientation, therefore should not be understood as the limitation to the application.

One skilled in the art will appreciate that in addition to realizing system provided by the invention in a manner of pure computer readable program code It, completely can be by the way that method and step be carried out programming in logic come so that provided by the invention other than system, device and its modules System, device and its modules are declined with logic gate, switch, specific integrated circuit, programmable logic controller (PLC) and insertion The form of controller etc. realizes identical program.So system provided by the invention, device and its modules may be considered that It is a kind of hardware component, and the knot that the module for realizing various programs for including in it can also be considered as in hardware component Structure；It can also will be considered as realizing the module of various functions either the software program of implementation method can be Hardware Subdivision again Structure in part.

Specific embodiments of the present invention are described above.It is to be appreciated that the invention is not limited to above-mentioned Particular implementation, those skilled in the art can make a variety of changes or modify within the scope of the claims, this not shadow Ring substantive content of the invention.In the absence of conflict, the feature in embodiments herein and embodiment can any phase Mutually combination.

Claims

1. a kind of irregular face antidote based on GLD-GAN characterized by comprising

Step S4: the GLD-GAN network model of each class of training；

Step S7: face image a and b are merged according to the ratio of a:b=3:1, and the positive face of fusion is replaced in image b just Face obtains face image c；

Step S9: training super-resolution network SRGAN；

Step S10: utilize SRGAN by face image c super-resolution processing.

2. the irregular face antidote according to claim 1 based on GLD-GAN, which is characterized in that the step S1 includes:

Step S101: whether there is face in detection image: being continued to execute if so, then entering step S102, otherwise, jump out and currently follow Ring simultaneously reports the mistake that face is not detected；

Step S103: positioning the position of nose in calculated face coordinate first, and ensures that nose is the face figure of interception Central axes, the position of bounding box is adjusted according to central axes, and intercepts bounding box as new face figure P₀；

The step S102:

Using the open source library face_recognition based on dlib, image and discriminance analysis face are read, and returns to its face Coordinate and bounding box；

Remember that right is the right margin coordinate of former bounding box, left is the left margin coordinate of former bounding box, remembers former axis line coordinates x =(right-left)/2 remembers that the abscissa of nose is mid, if x > mid, right=right- (x-mid), left is constant； If x < mid, left=left+ (mid-x), right is constant；If x=mid, without operation.

3. the irregular face antidote according to claim 2 based on GLD-GAN, which is characterized in that the step S2 includes:

Step S201: using open source MODEL C RF-RNN, by the face figure P of acquisition₀Image segmentation is carried out, an image segmentation is obtained Chromaticity diagram P₁；

Step S202: traversal P₀With P₁Each of pixel, if in P₁In pixel correspond to the first color, then make P₀In it is right The pixel answered becomes the second color, to achieve the effect that divide facial image, obtains after face and background segment, only The image P of face₂；

The step S3 includes:

All training images for being used for side face correction are divided into multiple classes according to the angle of side face, are then placed in Inception Classification based training is carried out in V3 model, the Inception V3 network after being trained；

The input of the Inception V3 network is a facial image, and facial image is deflected relative to positive face when output Angle；

The step S4 includes:

Step S401: prepare the first training, test and validation data set, the image in the first training, test and validation data set The corresponding positive face figure of one side face figure；

Step S402: the corresponding positive face figure splicing of side face figure in the first training, test and validation data set is schemed at one On；

Step S403: a GLD-GAN network model, GLD-GAN are trained for each class of the angular divisions according to side face The loss function of network are as follows:

Wherein,

G^*Indicate the loss function of GLD-GAN network

Indicate the loss of GAN network itself, i.e., by Min-max to reach a stabilization Locally optimal solution；

λ₁Indicate the parameter of L1 loss；

λ₂Indicate the parameter of fuzzy loss；

λ₃Indicate that identity retains the parameter of loss；

Indicate L1 loss, i.e. norm loss；

Indicate fuzzy loss；

Indicate that identity retains loss；

G indicates to generate the loss function of network；

D indicates to differentiate the loss function of network, i.e., the loss of the part between generation image and truthful data ground truth Function；

L2 indicates to generate the loss of sharpness function of image；

4. the irregular face antidote according to claim 3 based on GLD-GAN, which is characterized in that the step S5 includes:

Step S501: according to Inception V3 network, by the facial image P of input₂Assign to the GLD-GAN network mould of corresponding class In type；

Step S502: in GLD-GAN network model, by the facial image P of input₂It is converted into the picture of pre-set dimension, and is exported Corresponding face image P₃；

The step S6 includes:

The step S7 includes:

Step S701: by face image P₃、P₄Face fusion is carried out according to the first preset ratio, face fusion algorithm is based on open source Library dlib and OpenCV, wherein dlib is used to extract the characteristic model of face, reuses OpenCV proportionally for characteristic model It is merged, obtains fusion face；

Step S702: original face image P is replaced using obtained fusion face₄, obtain image P₅, it is same that face replaces algorithm Sample is based on dlib and OpenCV；

The fusion face that the utilization obtains replaces original face image P₄:

Convex closure is calculated using the facial key point of dlib detection, then Delaunay Triangulation is carried out to key point, and by affine Fusion face is mapped to positive face figure P by transformation₄On.

5. the irregular face antidote according to claim 4 based on GLD-GAN, which is characterized in that the step S8 includes:

Using two-sided filter, by image P₅It optimizes, obtains image P₆；

The two-sided filter generates distance template using two-dimensional Gaussian function, generates codomain template using one-dimensional Gaussian function, Distance template coefficient are as follows:

Wherein,

Indicate distance template coefficient；

K, l are the centre coordinate of template window；

I, j are the coordinate of other coefficients of template window；

σ_dFor the standard deviation of its Gaussian function；

The generation formula of codomain coefficients are as follows:

Wherein,

Indicate codomain coefficients；

Indicates coordinate is i, the pixel value of the point of j；

F (k, l) indicates coordinate is k, the pixel value of the point of l；

σ_rIndicate the standard deviation of its Gaussian function；

Wherein function f represents image to be processed；

To sum up, the template of two-sided filter are as follows:

Wherein,

Indicate the template of two-sided filter；

The step S9 includes:

The size of second training, the low-resolution image in test and validation data set and high-definition picture is second pre- If ratio.

6. the irregular face antidote according to claim 5 based on GLD-GAN, which is characterized in that the step S10 includes:

By image P₆Super-resolution network model SRGAN in super-resolution network model SRGAN after input training, after training Super-resolution processing is carried out to input picture, improves the resolution ratio of image to default resolution ratio, and export final image P₇；

Wherein,

Indicate the loss function that network is generated in SRGAN network；

φ_i,jIndicate the characteristic pattern obtained before i-th of maximum pond layer of the jth layer convolution (after activation) of VGG-19 network；

I^HRIndicate high-definition picture；

Indicate the corresponding characteristic pattern of image；

I^LRIndicate low-resolution image；

7. a kind of irregular face correction system based on GLD-GAN characterized by comprising

Module S4: the GLD-GAN network model of each class of training；

Module S7: face image a and b are merged according to preset ratio, and the positive face of fusion is replaced into the positive face in image b, is obtained Face image c；

Module S9: training super-resolution network SRGAN；

Module S10: utilize SRGAN by face image c super-resolution processing.

8. the irregular face correction system according to claim 7 based on GLD-GAN, which is characterized in that the module S1 includes:

Module S101: whether there is face in detection image: if so, otherwise then calling module S102 jumps out previous cycle and reports The mistake of face is not detected；

Module S103: positioning the position of nose in calculated face coordinate first, and ensures that nose is the face figure of interception Central axes, the position of bounding box is adjusted according to central axes, and intercepts bounding box as new face figure P₀；

The module S102:

Remember that right is the right margin coordinate of former bounding box, left is the left margin coordinate of former bounding box, remembers former axis line coordinates x =(right-left)/2 remembers that the abscissa of nose is mid, if x > mid, right=right- (x-mid), left is constant； If x < mid, left=left+ (mid-x), right is constant；If x=mid, without operation；

The module S2 includes:

Module S201: using open source MODEL C RF-RNN, by the face figure P of acquisition₀Image segmentation is carried out, an image segmentation is obtained Chromaticity diagram P₁；

Module S202: traversal P₀With P₁Each of pixel, if in P₁In pixel correspond to the first color, then make P₀In it is right The pixel answered becomes the second color, to achieve the effect that divide facial image, obtains after face and background segment, only The image P of face₂；

The module S3 includes:

The module S4 includes:

Module S401: prepare the first training, test and validation data set, the image in the first training, test and validation data set The corresponding positive face figure of one side face figure；

Module S402: the corresponding positive face figure splicing of side face figure in the first training, test and validation data set is schemed at one On；

Module S403: a GLD-GAN network model, GLD-GAN are trained for each class of the angular divisions according to side face The loss function of network are as follows:

Wherein,

G^*Indicate the loss function of GLD-GAN network

Indicate the loss of GAN network itself, i.e., it is steady to reach one by Min-max Fixed locally optimal solution；

λ₁Indicate the parameter of L1 loss；

λ₂Indicate the parameter of fuzzy loss；

λ₃Indicate that identity retains the parameter of loss；

Indicate L1 loss, i.e. norm loss；

Indicate fuzzy loss；

Indicate that identity retains loss；

G indicates to generate the loss function of network；

L2 indicates to generate the loss of sharpness function of image；

The module S5 includes:

Module S501: according to Inception V3 network, by the facial image P of input₂Assign to the GLD-GAN network mould of corresponding class In type；

Module S502: in GLD-GAN network model, by the facial image P of input₂It is converted into the picture of pre-set dimension, and is exported Corresponding face image P₃；

The module S6 includes:

The module S7 includes:

Module S701: by face image P₃、P₄Face fusion is carried out according to the first preset ratio, face fusion algorithm is based on open source Library dlib and OpenCV, wherein dlib is used to extract the characteristic model of face, reuses OpenCV proportionally for characteristic model It is merged, obtains fusion face；

Module S702: original face image P is replaced using obtained fusion face₄, obtain image P₅, it is same that face replaces algorithm Sample is based on dlib and OpenCV；

The fusion face that the utilization obtains replaces original face image P₄:

9. the irregular face correction system according to claim 8 based on GLD-GAN, which is characterized in that the module S8 includes:

Using two-sided filter, by image P₅It optimizes, obtains image P₆；

Wherein,

Indicate distance template coefficient；

K, l are the centre coordinate of template window；

I, j are the coordinate of other coefficients of template window；

σ_dFor the standard deviation of its Gaussian function；

The generation formula of codomain coefficients are as follows:

Wherein,

Indicate codomain coefficients；

Indicates coordinate is i, the pixel value of the point of j；

F (k, l) indicates coordinate is k, the pixel value of the point of l；

σ_rIndicate the standard deviation of its Gaussian function；

Wherein function f represents image to be processed；

To sum up, the template of two-sided filter are as follows:

Wherein,

Indicate the template of two-sided filter；

The module S9 includes:

The size of second training, the low-resolution image in test and validation data set and high-definition picture is second pre- If ratio；

The module S10 includes:

Wherein,

Indicate the loss function that network is generated in SRGAN network；

I^HRIndicate high-definition picture；

Indicate the corresponding characteristic pattern of image；

I^LRIndicate low-resolution image；

10. a kind of computer readable storage medium for being stored with computer program, which is characterized in that the computer program is located Reason device realizes the step of the irregular face antidote described in any one of claims 1 to 6 based on GLD-GAN when executing Suddenly.