CN109815893B - Color face image illumination domain normalization method based on cyclic generation countermeasure network - Google Patents
Color face image illumination domain normalization method based on cyclic generation countermeasure network Download PDFInfo
- Publication number
- CN109815893B CN109815893B CN201910061571.9A CN201910061571A CN109815893B CN 109815893 B CN109815893 B CN 109815893B CN 201910061571 A CN201910061571 A CN 201910061571A CN 109815893 B CN109815893 B CN 109815893B
- Authority
- CN
- China
- Prior art keywords
- illumination
- image
- generator
- discriminator
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000005286 illumination Methods 0.000 title claims abstract description 130
- 238000010606 normalization Methods 0.000 title claims abstract description 21
- 125000004122 cyclic group Chemical group 0.000 title claims description 7
- 238000000034 method Methods 0.000 claims abstract description 48
- 238000012549 training Methods 0.000 claims abstract description 43
- 238000012360 testing method Methods 0.000 claims abstract description 19
- 230000001815 facial effect Effects 0.000 claims abstract description 5
- 230000006870 function Effects 0.000 claims description 34
- 230000008569 process Effects 0.000 claims description 11
- 238000006243 chemical reaction Methods 0.000 claims description 10
- 230000000694 effects Effects 0.000 claims description 9
- 238000005070 sampling Methods 0.000 claims description 9
- 238000010276 construction Methods 0.000 claims description 6
- 230000017105 transposition Effects 0.000 claims description 4
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 3
- 238000000691 measurement method Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 claims 3
- 238000013508 migration Methods 0.000 abstract description 3
- 230000005012 migration Effects 0.000 abstract description 3
- 238000012545 processing Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 210000004209 hair Anatomy 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 210000005069 ears Anatomy 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 206010008909 Chronic Hepatitis Diseases 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 208000006454 hepatitis Diseases 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007935 neutral effect Effects 0.000 description 1
- 239000013612 plasmid Substances 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000011425 standardization method Methods 0.000 description 1
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a method for normalizing an illumination field of a color face image based on a loop generation countermeasure network, which comprises the following steps: s1, establishing a cycle generation confrontation network model for color face image illumination normalization; s2, establishing a loss function of the model; training of the model and testing on the test set is performed S3. The invention converts the color face image under various illumination into the designated target illumination domain, inputs the color face image with uneven illumination, uses the circularly generated countermeasure network as a model framework, and realizes the multi-illumination normalization of the face image by taking the target uniform illumination domain as the target, and the normalized image not only can better keep the facial attribute characteristics of the original face, but also can well realize the cross-data set migration.
Description
Technical Field
The invention relates to the field of computer vision face illumination, in particular to a color face image illumination domain normalization method based on a loop generation countermeasure network.
Background
In recent years, due to the rapid development of deep learning in computer vision, two-dimensional face-based analysis techniques, such as face recognition, face matching, face attribute recognition, and the like, have received great attention. At present, many algorithms in the face field can achieve nearly perfect performance and are widely used. The algorithms with good effect performance are based on strict control on non-human face identity information such as the posture, the shooting environment, the expression, the illumination and the like of a human face image, and certain defects exist in the application of the algorithms in a natural and unlimited real environment. Under natural conditions, the performance of the algorithm is easily influenced by various aspects such as illumination, shooting angle, shading and the like, wherein the change of the illumination is one of the most important factors influencing the performance of the algorithm. Because the human face is of a 3D structure, shadows cast by different angles of illumination can strengthen or weaken the original human face characteristics, and especially the performance of an analysis algorithm of the two-dimensional human faces is sharply reduced due to the facial shadows caused by insufficient illumination. Meanwhile, the existing theoretical experiments prove that the difference of one human face individual caused by illumination change is even larger than the difference of different individuals under the same illumination. Therefore, the method for preprocessing the face image under different illumination change environments by exploring a proper method can simulate the illumination imaging environment which may appear in the face image, and has great significance for improving the performance of the face analysis algorithm.
Early methods focused on human face gray level images, redistributed the intensity of original human face images by utilizing gray level adjustment, and made human face images not easy to generate illumination change, such as histogram equalization. The method proposed next focuses on extracting illumination invariant features of images of the same face under different illumination conditions, such as a self-service image method, an S & L illumination standardization method, and the like, and these methods are also not suitable for hard shadows, edge information, detail information, and the like are easily lost in processing generated face images, and meanwhile, the method is only applied to gray level images of face images, and is difficult to expand to a color space, and practical application has great limitations.
With the development of deep learning, the convolutional neural network is also used for deducing illumination parameters and reconstructing an illumination environment map; in addition, a three-dimensional model is directly fused to generate multiple views and samples under various illumination conditions, and a model capable of processing multiple illuminations simultaneously is constructed by utilizing the nonlinear transformation capability of a neural network. The methods are high in complexity and high in calculation cost, most methods are limited to processing well-segmented and strictly-aligned face regions, and the robustness of the complete face image is not high.
The generation of countermeasure networks, which are successfully applied to image inter-domain conversion and inter-domain migration, is a relatively popular field of computer vision research. These models learn how to convert images from one domain to another. The method provides an idea for conversion among multiple illumination domains of the human face image, simulates the imaging condition of the original human face in the illumination domains which may occur, and can be realized by generating an idea of domain conversion of a countermeasure network. The human face illumination processing method has the advantages that when human face illumination processing is carried out, human face parts do not need to be cut, strict human face alignment is not needed, illumination conversion can be carried out on the human face and the background well, and the method is more suitable for real scenes. Meanwhile, there is a limitation that, for example, when the illumination conversion is performed, the identity information of the original image is to be retained. More specifically, facial attribute features, hair, obstructions on the face, etc. The existing domain conversion generation countermeasure network model is easy to generate noise, lose detail information and have local fuzzy conditions in the generated image. The illumination normalization of the face gray-scale image based on the cycleGAN is only performed on the face gray-scale image, and is based on a strictly cut face region, and does not include hair, ears and other parts, so that the face gray-scale image is difficult to expand to a color space, and certain application limitation exists in a natural environment. The color face image normalization method based on the generation of the countermeasure network and the Triplet loss is also provided, but when the model is trained, a discriminator needs to discriminate the specific identity type of the face in the target illumination field, the model is trained by using specific face identity type information, the Triplet loss is added during reconstruction, the specific face identity information is used, and the method is equivalent to the method of using pairing data to convert the illumination field to a certain extent. The adaptability of the method to face images outside a training data set is limited, the generated illumination normalization effect is poor for cross-data set testing, and the application of the method is also greatly limited. Our illumination domain transformation generation employs cross-dataset unpaired data against the network model, and to overcome these problems described above, greater attention is paid to the quality of the transformation-generated face image and the effect of cross-dataset migration.
Disclosure of Invention
The invention mainly aims to overcome the defects of the prior art and provide a method for normalizing the illumination domain of a color face image based on a loop generation countermeasure network.
In order to achieve the purpose, the invention adopts the following technical scheme:
a color face image illumination domain normalization method based on a loop generation countermeasure network comprises the following steps:
s1: establishing a cyclic generation confrontation network model for color face image illumination normalization, wherein the cyclic generation confrontation network model comprises a generation network and an identification network, the generation network generates a face feature reconstruction picture through a construction generator, and the identification network identifies an illumination domain through a construction identifier;
s2: establishing a loss function of the model, so that the generated confrontation network training is stable, and the generator can better keep the facial features of the input image when learning illumination information of the target illumination field;
s3: and training the model, dividing the images of different illumination types into an uneven illumination field and a target illumination field, performing circular training in the generated countermeasure network, testing on the test set, and checking the generated face image effect graph.
Further, the specific process of step S1 is:
s11: building a Generation network, building two generators GAAnd GB. Wherein G isAThe generator illuminates the face image A in the domain set ArealAs an input, an image A is generated having the same illumination characteristics as the target illumination field BfakeIn the order of GBPicture B generated by the generatorfakeAs input, generate the sum ofrealReconstructed picture A with same illumination characteristics and human face characteristicsrec；GBGenerator with GAPicture A generated by the generatorfakeAs input, generate the sum ofrealReconstructed image B with same illumination characteristic and human face characteristicrecAnd illuminating image B in field BrealGenerating as input a picture B having the same illumination characteristics as the set of illumination fields Afake。GAGenerators and GBThe generator has the same network structure, the network of the generator is composed of four convolutional layers, six residual modules and two transposition convolutional layers, wherein the first convolutional layer is an input layer, the two convolutional layers used for down sampling and the two transposition convolutional layers used for up sampling are in one-to-one symmetry, and the six residual modules are embedded in the middle, so that the sizes of an output image and an input image are consistent. The last output layer is also a convolutional layer, and the Tanh activation function is used to ensure that the output value is between 0 and 255. The residue is leftThe remaining six convolutional layers and the six residual modules all use an example normalization strategy and a ReLU activation function, wherein the number of channels of the six residual modules is 256, the convolutions are all 3x3, and the step length and the padding are all 1; the number of channels of the first and last convolutional layer is 64, the convolutional kernel is 7x7, the padding is 3, and the step size is 1; the number of channels of the other convolutional layers is two times or half of that of the previous layer, the sizes of convolutional kernels are all 3x3, the step lengths are all 2, and the padding is all 1.
S12: constructing an authentication network, constructing two authenticators DAAnd DB，DAThe discriminator is used for discriminating the picture A in the face illumination domain set ArealAnd generator GBGenerated picture BfakeThe illumination field category of (1); dBThe discriminator is used for discriminating the picture B in the face illumination field BrealAnd generator GAGenerating Picture AfakeLighting inter-domain class. When the discriminator is used for discriminating the type of the illumination field, the discriminator adopts a multi-scale feature map for discrimination, six convolutional layers are used in total, the number of channels of the six convolutional layers is increased by two times of that of the previous layer, the convolutional cores are all 3x3, the step length and the padding are both 2 and 1, the activation function is LeakyReLU, and when discrimination is carried out, feature maps of the last three layers are used;
further, the specific process of step S2 is:
s21: construction of discrimination loss of discriminator D, let Dsrc(x) The probability that the input sample x is a positive sample obtained by the discriminator D through multi-scale feature fusion calculation is output to be close to 1 when the discriminator is real and close to 0 when the discriminator is false. The generator G is intended to generate an image xfakeWith the real image xrealAre consistent so that the discriminator D cannot distinguish. The discriminator D is intended to discriminate the generated image from the real image as much as possible, and therefore the discrimination loss of the discriminator D is defined as:
wherein,
for a real image xrealAnd generating an image xfakeRandomly carrying out interpolation sampling on the connecting line to obtain:
λgpas a gradient penalty term
α is a random number of 0 to 1.
Therefore, the discriminator DALoss function force of (1):
discriminator DBThe loss function of (d) is:
at this time, the discriminator DAAnd DBThe above loss function is minimized.
S24: and (3) constructing reconstruction loss of the generator G, wherein in order to ensure that the generator G can effectively learn reliable characteristics universally suitable for each illumination domain, the reconstruction learning is used for constraining mapping conversion between domains. The method specifically comprises the following steps: the human face image is mapped and converted from the original illumination domain to the target illumination domain through a generator G, then the generated image is used as input, the generated image is inversely mapped and converted into the human face image of the original illumination domain, and the distance error between the reconstructed image and the original image is calculated. Three distance measurement methods are combined here, specifically as follows:
L1the norm measures the distance between two face images before and after reconstruction:
wherein, x is the face image of the original illumination domain, l' is the class mark of the target illumination domain, G (x) is the generated image converted from the original image to the target illumination domain, and G (x)) represents the inverse conversion from the generated image to the face image of the original illumination domain, namely the reconstructed image.
L1The distance measurement is the absolute value of the pixel value difference of the channel positions corresponding to two images with the same size, the spatial relationship between the image pixel and the neighborhood is easy to ignore, for example, the loss of information such as edges and textures is easy to cause, and the consistency with human visual cognition is not high.
In order to enable the generated face image to keep more identity information of the original face image, reduce detail distortion of the generated image, reduce noise pollution and better display visual reality, two distance indexes for measuring similarity between images, namely MS-SSIM (MS structural similarity index) and PSNR (Peak to Peak similarity) are additionally used.
Wherein the loss of MS-SSIM is specifically as follows:
brightness difference:
contrast difference:
the structural difference is as follows:
the final MS-SSIM loss of the original image x and the reconstructed image y is as follows:
where μ is the mean, σ is the variance, β1＝(k1L)2，β2＝(k2L)2，k1＝0.01，k2When the value of the image pixel value is 0.03, L is the value range of the image pixel value, and M represents a scaling factor, which indicates that the width and the height of the image are 2M-1Scaling is performed, where M is 2, ω1＝μ1＝ρ1＝0.0448，ω2＝μ2＝ρ2The value range of the calculation result is [ -1, 1 [ -0.2856 ]]The larger the value is, the higher the similarity of the two images is.
PSNR penalty is specifically as follows:
the original image x and the reconstructed image y are both RGB color images, and their mean square deviations are defined as:
wherein w, h is the image width and height, (i, j)1，(i，j)2，(i，j)3Respectively representing RGB images of first and secondAnd the value of position (i, j) in the third channel.
PSNR penalty is as follows:
wherein,
the maximum pixel value in the original image x and the reconstructed image y.
Finally, the reconstruction loss is specifically:
Lrec(x，y)＝L1(x，y)+α1(1-LSSIM(x，y))-α2LPSNR(x，y)
wherein alpha is2，α3Are weight coefficients.
S25: the integral loss function of the generator G is beneficial to continuously iterating and perfecting the parameters of the generator when the true and false discrimination errors of the generated image by the discrimination network D are generated when the generator G is trained in the countermeasure network. The generator G in combination with the discriminator D and the overall loss function of the reconstruction loss are specifically:
LG＝Ladv+α3Lrec(x，y)
wherein L isadvFor multi-scale discrimination loss of discriminator D, LrecTo reconstruct the loss, α1，α2，α3Are weight coefficients. The generator G minimizes the above loss function.
Here, the generator GAThe specific generation loss is:
generator GBThe specific generation loss is:
the overall loss function of the generator is:
further, the specific process of step S3 is as follows:
s31: the face illumination image is divided into a type A and a type B according to illumination domains, wherein the type A is a plurality of illumination domain sets, and the type B is a single target illumination domain set. Inputting the face image A _ real in A to GAObtaining a generated picture AfakeThe face image B in BrealIs inputted into GBTo obtain a generated image BfakeThen A is addedfakeIs inputted into GBTo obtain a reconstructed picture BrecA 1 to BfakeIs inputted into GAObtaining a reconstructed picture ArecThen A is addedrealAnd BfakeInput to a discriminator DAWhether the intermediate is true or false of the illumination type in the set A or not is judged, and B is usedrealAnd AfakeInput to a discriminator DBAnd whether the judgment is true or false of the illumination type in the B set is made. In the training process, a distributed training method is used for generating a network and judging the network, when the generator is trained, the model parameters of the discriminator are fixed, and the parameters of the discriminator are not updated; when training the discriminator, the model parameters of the generator are fixed, and the model parameters of the generator G are not updated at the moment.
S32: for the above output, the loss function is calculated in the manner described in step S2, optimized using Adam optimizer, and the model parameters are updated in the distributed training process.
S33: and dividing the data set A into a training set and a test set according to the ratio of 9: 1, and randomly sampling color face images with uneven illumination on an additional data set to form the test set. The training set was iteratively trained 2000 times. Storing the trained model, and finally using a generator GAAnd testing on the test set to check the generated human face image effect.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. the invention can realize the processing of the illumination normalization of the color face images which are not strictly cut and aligned, including the background and the complete face parts in the color images, such as the ears, the hairs and other parts.
2. The invention can realize the normalization of the non-paired color face illumination images of the cross-data set, does not classify the specific face identity information during training as the limitation of a discriminator, and can be seen from the test set that the illumination normalization effect of the color face images outside the training set is better, which indicates that the model can be applied to the multi-illumination color face images under more non-limited conditions.
3. The generated picture can better keep the human face attribute detail part of the original image while changing the illumination condition of the input image, better keeps the face structure of the human face, displays good visual effect on hair, eyes, skin color and other detail parts, and can also better keep the sheltering objects such as glasses and the like, thereby being beneficial to keeping the identity information of the input person with uneven illumination.
Drawings
FIG. 1 is a flow chart of a method for illumination normalization of a face color image according to the present invention;
fig. 2 is a schematic structural view of the method in embodiment 1.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but the present invention is not limited thereto.
Examples
The invention aims at the conversion of human face color picture multi-illumination-domain normalization, a data set used by a human face data set of CMU-multi-PIE is collected by a visual research team of university of Kanai Melong USA, images of the data set comprise 337 individual face identities, each face identity comprises 20 pictures captured under 19 different illumination conditions, the first and the last pictures are not illuminated by any camera flash, and the rest pictures are respectively and independently captured by 18 annular flash lamps in a flash mode. The face image with 18420 neutral expressions is selected, wherein 1800 images are randomly selected as a test set, the rest 16620 images are selected as a training set, and the input and the output are all adjusted to be 128x 128. And selecting uniform illumination face pictures with illumination categories of 06, 07 and 08 from the training set to construct a target illumination domain set B, and constructing a multi-illumination domain set A by using the rest face pictures.
As shown in fig. 1 and fig. 2, the method for normalizing the illumination field of the color face image based on the loop generation confrontation network comprises the following steps: for the face illumination picture sets A and B, the face image A _ real in A is input into GAObtaining a generated picture AfakeThe face image B in BrealIs inputted into GBTo obtain a generated image BfakeThen A is addedfakeIs inputted into GBTo obtain a reconstructed picture BrecA 1 to BfakeIs inputted into GAObtaining a reconstructed picture ArecThen A is addedrealAnd BfakeInput to a discriminator DAWhether the intermediate is true or false of the illumination type in the set A or not is judged, and B is usedrealAnd AfakeInput to a discriminator DBAnd whether the judgment is true or false of the illumination type in the B set is made. In the training process, in order to ensure the stability of the training of the circularly generated confrontation network, a step-by-step training method is used for generating the network and distinguishing the network, when the generator is trained, the model parameters of the discriminator are fixed, at the moment, the parameters of the discriminator are not updated, and only the parameters of the generator are changed; when the discriminator is trained, the model parameters of the generator are fixed, and at the moment, the model parameters of the generator G are not updated, and only the parameters of the discriminator are updated. The details are as follows:
1. reading in a CMU-multi-PIE data set, randomly selecting and generating a training set and a test set according to the ratio of 9: 1, and additionally, randomly selecting 1000 pieces of data in an FRGC data set for multiple times to form a plurality of test sets. Then, a 06 th, 07 th and 08 th class illumination human face picture is selected in the training set to form a target illumination domain set B, and the rest of the training set forms a human face multi-illumination domain set A. The picture sizes are all set to 128x 128.
2. Construction of a plasmid containing two genesFinished device GA，GBAnd two discriminators DA、DBThe number of parameter updates of the generator and the discriminator at each training period is 1 and 3, respectively. Each training batch was set to 32 per training cycle, i.e. 32 pictures were randomly selected from a and B, respectively. The parameters of the generator are first fixed, and a discriminator D is trainedA、DB3 times, each time sampling a batch A from A and B respectivelyrealAnd BrealA isrealInput generator GAObtaining a generated picture AfakeA 1 to BrealInput to the generator GBTo obtain a generated picture BfakeThen A is addedrealAnd BfakeIs inputted into DAMaking a judgment of true or falserealAnd AfakeInput to a discriminator DBTo judge the truth and falseness and calculate the loss function
And
respectively to discriminator DAAnd DBUpdating the parameters; after the discriminator is trained, the parameters are fixed, the training generator is started for 1 time, ArealInput generator GAObtaining a generated picture AfakeA 1 to BrealInput to the generator GBTo obtain a generated picture BfakeThen A is addedrealAnd BfakeIs inputted into DAMaking a judgment of true or falserealAnd AfakeInput to a discriminator DBTo judge the truth and falseness and calculate the loss function
And
then B is mixedfakeIs inputted into GAObtaining a reconstructed generated picture Arec, and adding AfakeIs inputted into GBObtaining a reconstructed generated picture BrecCalculating the lossMedicine for treating chronic hepatitis B
And
the two generator parameters are then updated separately.
When the discriminator is trained, the weight coefficient of the loss function is set as follows:
LD＝-Ladv
discriminator DAAnd DBDiscrimination loss LadvThe gradient penalty term in
Is given by a weight coefficient ofgpAre all set to 10.
When training generator G, the parameters of its loss function are set as follows:
LG＝Ladv+α3Lrec(x，y)
Lrec(x，y)＝L1+α1(1-LMS-SSIM(x，y))-α2LPSNR(x，y)
generator GAAnd GBReconstruction loss L at the time of trainingrecWeight coefficient setting alpha3Is 10, the weight coefficient alpha1And alpha2Set to 0.5, 0.3, respectively.
3. In order to ensure the stability of the training and accelerate the convergence, the discriminator D is trainedAAnd DBAt that time, the historical values are used to supervise the training strategy. Setting a history buffer area, storing the generated images of the last batch judged by the discriminator, and randomly sampling half batch of the history generated images and half batch of the generated images of the current batch from the history buffer area to form a batch when the discriminator loss function is calculated in the current batchAn image is generated for discrimination by the discriminator, a loss function is calculated and discriminator parameters are updated.
4. The training period is 2000, all the network learning rates in the first 1000 periods are set to be 0.0002, the linear attenuation of the last 1000 periods is 0, and Adam optimization algorithm is used, wherein beta is1Is set to 0.5, beta2Set to 0.999.
5. Then generating model G in the trained generating network modelAAnd in the method, a cross-data set test set is used for testing, and the effect of generating the picture is checked.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.
Claims (7)
1. The method for normalizing the illumination domain of the color face image based on the loop generation confrontation network is characterized by comprising the following steps of:
s1: establishing a cyclic generation confrontation network model for color face image illumination normalization, wherein the cyclic generation confrontation network model comprises a generation network and an identification network, the generation network generates a face picture and a face feature reconstruction picture which are converted into a designated illumination domain through a construction generator, and the identification network identifies the illumination domain through a construction identifier; the method specifically comprises the following steps:
s11: building a Generation network, building two generators GAAnd GB(ii) a Wherein G isAThe generator uses the human face picture A in the multiple illumination domain set ArealAs an input, a picture a is generated having the same illumination characteristics as the target illumination field BfakeIn the order of GBPicture B generated by the generatorfakeAs input, generate the sum ofrealReconstructed picture A with same illumination characteristics and human face characteristicsrec；GBGenerator with GAPicture A generated by the generatorfakeAs input, generate the sum ofrealHaving the same illumination characteristics and face characteristicsReconstructed picture B ofrecAnd illuminating image B in field BrealGenerating as input a picture B having the same illumination characteristics as the set of illumination fields Afake；
S12: constructing an authentication network, constructing two authenticators DAAnd DB，DAThe discriminator is used for discriminating the picture A in the face illumination domain set ArealAnd generator GBGenerated picture BfakeThe illumination field category of (1); dBThe discriminator is used for discriminating the picture B in the face illumination field BrealAnd generator GAGenerating Picture AfakeWhen the discriminator is used for discriminating the illumination domain, the discriminator adopts a multi-scale characteristic diagram for discrimination;
the generator GAAnd generator GBThe network has the same network structure, and the network comprises four convolutional layers, six residual error modules and two transposition convolutional layers, wherein the first convolutional layer is an input layer, and the six residual error modules are embedded between the two convolutional layers used for down-sampling and the two transposition convolutional layers used for up-sampling, so that the sizes of an output image and an input image are consistent;
s2: establishing a loss function of the model, so that the generated confrontation network training is stable, and the generator can better keep the facial features of the input image when learning illumination information of the target illumination field; the method specifically comprises the following steps:
s21: for discriminator DALoss function thereof
Is to the face picture A in the multi-illumination-domain set ArealAnd illuminating image B in field BrealInput GBImage B generated by the generatorfakeThe identification loss of the illumination domain category is identified by using a multi-scale characteristic diagram, the output is close to 1 when the identification is true, and the output is close to 0 when the identification is false;
s22: for discriminator DBLoss function thereof
Is to face picture B in target illumination domain set BrealAnd image A in set ArealInput GAImage A generated by the generatorfakeThe identification loss of the illumination domain category is identified by using a multi-scale characteristic diagram, the output is close to 1 when the identification is true, and the output is close to 0 when the identification is false;
s23: for the discriminator GAAnd GBAll combined with WGAN gradient penalty strategy, discriminator GAAnd GBTaking a minimum loss function as an optimization target;
s24: reconstruction loss L of the generatorrecConstructing, calculating distance errors between a reconstructed image and an original image by combining three distance measurement methods, wherein the distance errors are respectively L1Norm error, MS-SSIM error, PSNR error;
s25: generator GAOverall loss function of
Which incorporates a discriminator DBInverse sum of loss function for image A in set ArealAnd GBReconstructed picture B generated by the generatorrecThe reconstruction error of (2);
s26: generator GBOverall loss function of
Which incorporates a discriminator DAInverse sum of loss function on image B in set BrealAnd GAReconstructed picture a generated by the generatorrecThe reconstruction error of (2);
α1is a reconstruction error weight parameter;
s26: the overall loss function of the generator is:
to minimize the loss function while optimizing the update generator GAAnd GBThe parameters of (1);
s3: and training the model, dividing the images of different illumination types into an uneven illumination field and a target even illumination field, performing circular training in the generation of the countermeasure network, testing on the test set, and checking the generated face image effect image.
2. The method for illumination domain normalization of color face images based on loop-generated confrontation network as claimed in claim 1, wherein said discriminator D is used for discriminating between color face imagesAAnd discriminator DBThe use of a multi-scale signature for identification is used for six convolutional layers in total, and the number of channels of the six convolutional layers is increased by twice of that of the previous layer.
3. The method for illumination domain normalization of color face images based on loop-generated confrontation network as claimed in claim 1, wherein in step S21, in order to ensure the training process is stable and generate high quality images, the WGAN gradient penalty strategy is used to redefine the loss function LadvThe method specifically comprises the following steps:
wherein,
for a real image xrealAnd generating an image xfakeRandomly carrying out interpolation sampling on the connecting line to obtain:
λgpas a gradient penalty term
A is a random number from 0 to 1;
therefore, the discriminator DAThe loss function of (d) is:
discriminator DBThe loss function of (d) is:
at this time, the discriminator DAAnd DBThe above loss function is minimized.
4. The method for normalizing the illumination field of the color face image based on the loop-generated confrontation network as claimed in claim 1, wherein in step S21, three distance measurement methods are as follows:
L1the norm measures the distance between two face images before and after reconstruction:
wherein, x is the face image of the original illumination domain, l' is the class mark of the target illumination domain, G (x) is the generated image converted from the original image to the target illumination domain, and G (x)) represents the inverse conversion from the generated image to the face image of the original illumination domain, namely the reconstructed image;
the loss measured by MS-SSIM is:
where μ is the mean, σ is the variance, β1＝(k1L)2，β2＝(k2L)2，k1＝0.01，k2When the value of the image pixel value is 0.03, L is the value range of the image pixel value, and M represents a scaling factor, which indicates that the width and the height of the image are 2M-1Zooming is carried out;
PSNR reconstruction loss is specifically:
Lrec(x，y)＝L1(x，y)+α1(1-LSSIM(x，y))-α2LPSNR(x，y)
wherein alpha is2，α3Are weight coefficients.
5. The method for normalizing the illumination field of the color face image based on the loop-generated confrontation network as claimed in claim 1, wherein the specific process of the step S3 is as follows:
s31: dividing the face image into A, B types according to the illumination domain, wherein the A type is a plurality of illumination domain sets, and the B type is a target illumination domain set; the face image A in ArealIs inputted into GAObtaining a generated picture AfakeThe face image B in BrealIs inputted into GBTo obtain a generated image BfakeThen A is addedfakeIs inputted into GBTo obtain a reconstructed picture BrecA 1 to BfakeIs inputted into GAObtaining a reconstructed picture ArecThen A is addedrealAnd BfakeInput to a discriminator DAWhether the intermediate is true or false of the illumination type in the set A or not is judged, and B is usedrealAnd AfakeInput to a discriminator DBJudging whether the illumination type in the set B is true or false, calculating the loss of a discriminator and the loss function of a generator, and updating model parameters;
s32: for the above outputs, the loss function is calculated in the manner described in step S2, the loss function is optimized using an Adam optimizer, and the model parameters are updated in a step training process;
s33: dividing the data set A into a training set and a testing set according to a set proportion, storing the trained model, and finally using a generator GAThe model of (2) is tested on the test set, and the generated human face image effect is checked.
6. The method for normalization of color face image illumination field based on cyclic generation of confrontation network as claimed in claim 5, wherein in step S32, in the training process, a step-by-step training method is used for generating network and discriminating network, and when training the generator, the model parameters of the discriminator are fixed; the model parameters of the generator are fixed while training the discriminator.
7. The method for generating color face image illumination domain normalization against network based on loop generation as claimed in claim 5, wherein in the step S32, the set ratio is 9: 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910061571.9A CN109815893B (en) | 2019-01-23 | 2019-01-23 | Color face image illumination domain normalization method based on cyclic generation countermeasure network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910061571.9A CN109815893B (en) | 2019-01-23 | 2019-01-23 | Color face image illumination domain normalization method based on cyclic generation countermeasure network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109815893A CN109815893A (en) | 2019-05-28 |
| CN109815893B true CN109815893B (en) | 2021-03-26 |
Family
ID=66604827
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910061571.9A Active CN109815893B (en) | 2019-01-23 | 2019-01-23 | Color face image illumination domain normalization method based on cyclic generation countermeasure network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109815893B (en) |
Families Citing this family (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110210556B (en) * | 2019-05-29 | 2022-09-06 | 中国科学技术大学 | Pedestrian re-identification data generation method |
| CN110197226B (en) * | 2019-05-30 | 2021-02-09 | 厦门大学 | Unsupervised image translation method and system |
| CN112102424A (en) * | 2019-06-18 | 2020-12-18 | 顺丰科技有限公司 | License plate image generation model construction method, generation method and device |
| CN110415297B (en) * | 2019-07-12 | 2021-11-05 | 北京三快在线科技有限公司 | Positioning method and device and unmanned equipment |
| CN110706152B (en) * | 2019-09-25 | 2021-09-17 | 中山大学 | Face illumination migration method based on generation of confrontation network |
| CN110633698A (en) * | 2019-09-30 | 2019-12-31 | 上海依图网络科技有限公司 | Infrared picture identification method, equipment and medium based on loop generation countermeasure network |
| CN111179228A (en) * | 2019-12-16 | 2020-05-19 | 浙江大学 | Single-energy CT energy spectrum imaging method based on deep learning |
| CN111047509B (en) * | 2019-12-17 | 2024-07-05 | 中国科学院深圳先进技术研究院 | Image special effect processing method, device and terminal |
| CN111275692B (en) * | 2020-01-26 | 2022-09-13 | 重庆邮电大学 | Infrared small target detection method based on generation countermeasure network |
| CN111325661B (en) * | 2020-02-21 | 2024-04-09 | 京工慧创(福州)科技有限公司 | Seasonal style conversion model and method for image named MSGAN |
| CN111275651B (en) * | 2020-02-25 | 2023-05-12 | 东南大学 | Face bright removal method based on antagonistic neural network |
| CN111429436B (en) * | 2020-03-29 | 2022-03-15 | 西北工业大学 | Intrinsic image analysis method based on multi-scale attention and label loss |
| CN111539897A (en) * | 2020-05-09 | 2020-08-14 | 北京百度网讯科技有限公司 | Method and apparatus for generating image conversion model |
| CN113689527B (en) * | 2020-05-15 | 2024-02-20 | 武汉Tcl集团工业研究院有限公司 | Training method of face conversion model and face image conversion method |
| CN111832229A (en) * | 2020-07-17 | 2020-10-27 | 中国人民解放军海军工程大学 | Vibration transmission system based on cycleGAN model and training method thereof |
| CN112131975B (en) * | 2020-09-08 | 2022-11-15 | 东南大学 | Face illumination processing method based on Retinex decomposition and generation of confrontation network |
| CN112259119B (en) * | 2020-10-19 | 2021-11-16 | 深圳市策慧科技有限公司 | Music source separation method based on stacked hourglass network |
| CN112287779B (en) * | 2020-10-19 | 2022-03-25 | 华南农业大学 | Low-illuminance image natural illuminance reinforcing method and application |
| CN112446873B (en) * | 2020-12-11 | 2024-09-24 | 深圳高性能医疗器械国家研究院有限公司 | Method for removing image artifacts |
| CN112634126B (en) * | 2020-12-22 | 2024-10-18 | 厦门美图之家科技有限公司 | Portrait age-reducing processing method, training method, device, equipment and storage medium |
| CN113378715B (en) * | 2021-06-10 | 2024-01-05 | 北京华捷艾米科技有限公司 | Living body detection method based on color face image and related equipment |
| CN113609893B (en) * | 2021-06-18 | 2024-04-16 | 大连民族大学 | Low-illuminance indoor human body target visible light characteristic reconstruction method and network based on infrared camera |
| CN114119356B (en) * | 2021-11-24 | 2024-06-11 | 北京理工大学 | CycleGAN-based method for converting thermal infrared image into visible light color image |
| CN115297263B (en) * | 2022-08-24 | 2023-04-07 | 广州方图科技有限公司 | Automatic photographing control method and system suitable for cube shooting and cube shooting |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107909640A (en) * | 2017-11-06 | 2018-04-13 | 清华大学 | Face weight illumination method and device based on deep learning |
| CN108334904A (en) * | 2018-02-07 | 2018-07-27 | 深圳市唯特视科技有限公司 | A kind of multiple domain image conversion techniques based on unified generation confrontation network |
| CN108334848A (en) * | 2018-02-06 | 2018-07-27 | 哈尔滨工业大学 | A kind of small face identification method based on generation confrontation network |
| CN108334847A (en) * | 2018-02-06 | 2018-07-27 | 哈尔滨工业大学 | A kind of face identification method based on deep learning under real scene |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018053340A1 (en) * | 2016-09-15 | 2018-03-22 | Twitter, Inc. | Super resolution using a generative adversarial network |
| CN108711138B (en) * | 2018-06-06 | 2022-02-11 | 北京印刷学院 | Gray level picture colorizing method based on generation countermeasure network |
-
2019
- 2019-01-23 CN CN201910061571.9A patent/CN109815893B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107909640A (en) * | 2017-11-06 | 2018-04-13 | 清华大学 | Face weight illumination method and device based on deep learning |
| CN108334848A (en) * | 2018-02-06 | 2018-07-27 | 哈尔滨工业大学 | A kind of small face identification method based on generation confrontation network |
| CN108334847A (en) * | 2018-02-06 | 2018-07-27 | 哈尔滨工业大学 | A kind of face identification method based on deep learning under real scene |
| CN108334904A (en) * | 2018-02-07 | 2018-07-27 | 深圳市唯特视科技有限公司 | A kind of multiple domain image conversion techniques based on unified generation confrontation network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109815893A (en) | 2019-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109815893B (en) | Color face image illumination domain normalization method based on cyclic generation countermeasure network | |
| CN108495110B (en) | Virtual viewpoint image generation method based on generation type countermeasure network | |
| CN110348330B (en) | Face pose virtual view generation method based on VAE-ACGAN | |
| CN108875935B (en) | Natural image target material visual characteristic mapping method based on generation countermeasure network | |
| CN109886881B (en) | Face makeup removal method | |
| CN111784821B (en) | Three-dimensional model generation method and device, computer equipment and storage medium | |
| CN116109798B (en) | Image data processing method, device, equipment and medium | |
| CN110175986B (en) | Stereo image visual saliency detection method based on convolutional neural network | |
| CN110909693A (en) | 3D face living body detection method and device, computer equipment and storage medium | |
| CN110097528A (en) | A kind of image interfusion method based on joint convolution autoencoder network | |
| CN110197229A (en) | Training method, device and the storage medium of image processing model | |
| CN111161191B (en) | Image enhancement method | |
| CN111145116A (en) | Sea surface rainy day image sample augmentation method based on generation of countermeasure network | |
| CN108764250B (en) | Method for extracting essential image by using convolutional neural network | |
| CN108932536A (en) | Human face posture method for reconstructing based on deep neural network | |
| CN108389192A (en) | Stereo-picture Comfort Evaluation method based on convolutional neural networks | |
| CN103971329A (en) | Cellular nerve network with genetic algorithm (GACNN)-based multisource image fusion method | |
| CN111242865A (en) | Fundus image enhancement method based on generation type countermeasure network | |
| CN112560710B (en) | Method for constructing finger vein recognition system and finger vein recognition system | |
| CN112002009A (en) | Unsupervised three-dimensional face reconstruction method based on generation of confrontation network | |
| Zhang et al. | MSGAN: generative adversarial networks for image seasonal style transfer | |
| CN116229528A (en) | Living body palm vein detection method, device, equipment and storage medium | |
| Liu et al. | Modern architecture style transfer for ruin or old buildings | |
| CN114119356A (en) | Method for converting thermal infrared image into visible light color image based on cycleGAN | |
| CN117593238A (en) | Low-illumination image enhancement method and system based on improved generation type countermeasure network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |