CN110084121A - Implementation method based on the human face expression migration for composing normalized circulation production confrontation network - Google Patents
Implementation method based on the human face expression migration for composing normalized circulation production confrontation network Download PDFInfo
- Publication number
- CN110084121A CN110084121A CN201910240461.9A CN201910240461A CN110084121A CN 110084121 A CN110084121 A CN 110084121A CN 201910240461 A CN201910240461 A CN 201910240461A CN 110084121 A CN110084121 A CN 110084121A
- Authority
- CN
- China
- Prior art keywords
- human face
- face expression
- picture
- generator
- arbiter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 230000008921 facial expression Effects 0.000 title claims abstract description 155
- 238000004519 manufacturing process Methods 0.000 title claims abstract description 46
- 238000000034 method Methods 0.000 title claims abstract description 32
- 238000013508 migration Methods 0.000 title claims abstract description 22
- 230000005012 migration Effects 0.000 title claims abstract description 22
- 238000001514 detection method Methods 0.000 claims abstract description 6
- 239000000203 mixture Substances 0.000 claims abstract description 6
- 238000005259 measurement Methods 0.000 claims abstract description 3
- 239000010410 layer Substances 0.000 claims description 13
- 230000014509 gene expression Effects 0.000 claims description 11
- 238000012549 training Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 8
- 238000001228 spectrum Methods 0.000 claims description 7
- 230000008859 change Effects 0.000 claims description 6
- 238000009826 distribution Methods 0.000 claims description 6
- 239000000284 extract Substances 0.000 claims description 6
- 238000003475 lamination Methods 0.000 claims description 5
- 239000003795 chemical substances by application Substances 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 239000002365 multiple layer Substances 0.000 claims description 3
- 230000003595 spectral effect Effects 0.000 claims description 3
- 238000013461 design Methods 0.000 claims description 2
- 230000002708 enhancing effect Effects 0.000 claims description 2
- 238000013507 mapping Methods 0.000 claims description 2
- 238000013480 data collection Methods 0.000 claims 1
- 230000002194 synthesizing effect Effects 0.000 claims 1
- 230000006870 function Effects 0.000 description 18
- 230000015572 biosynthetic process Effects 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 230000001815 facial effect Effects 0.000 description 3
- 238000003786 synthesis reaction Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 239000011521 glass Substances 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G06T3/04—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
Present invention discloses a kind of implementation methods based on the human face expression migration for composing normalized circulation production confrontation network, method includes the following steps: S1: acquiring various human face expression pictures, and classified one by one according to human face expression；S2: picture pretreatment removes fuzzy photo, then employment face detection algorithm obtains five key points of face, and uniformly cuts face picture according to key point；S3: the circulation production of building generator and arbiter composition fights network, and the pretreated picture of two classes is inputted respectively in network and calculates loss function and is trained；S4: the tool that trained generator is migrated as human face expression is obtained, and is applied to actual measurement.A generator is enabled to realize a variety of human face expression migrations based on normalized circulation production confrontation network is composed, and the human face expression generated can be more naturally, there is preferably robustness.
Description
Technical field
The present invention relates to a kind of realization sides based on the human face expression migration for composing normalized circulation production confrontation network
Method can be used for image processing technology in computer vision.
Background technique
In recent years, with the high speed development of artificial intelligence, deep learning also becomes popular research field, and production
The it is proposed of confrontation network accelerates the process of deep learning.The scholars such as University of Montreal Ian Goodfellow mentioned in 2014
Production confrontation network is gone out, the network of production confrontation in recent years has become one of research hotspot in deep learning.
It is a kind of generation model that production, which fights network, its structure is inspired in zero-sum two-person game.Production pair
Anti- network contains a generator and an arbiter.Generator can be learnt by the potential distribution of truthful data,
Then the data distribution that generator can generate vacation carrys out approach truthful data.Arbiter is a classifier, can be judged
Whether data distribution is true.By the continuous competition learning of two networks, generator can generate false number more and more true to nature
According to distribution, it is finally reached the effect mixed the spurious with the genuine.
Circulation production confrontation network is then the combination that production is fought to network and the conversion of figure to figure, recycles production pair
Anti- network is substantially the production confrontation network of two mirror symmetries, constitutes a loop network.Two production confrontation
Two generators of network share and two arbiters, i.e., there are two arbiter and two generators altogether.It recycles production and fights net
Network is suitble to handle picture Style Transfer, but applies and hold during the local feature migration of face removal this picture of glasses
Easily there is the change of picture integral color, i.e., is likely to occur the variation in other regions of face after face removal glasses.
Summary of the invention
The object of the invention is to propose a kind of based on spectrum normalization to solve the above-mentioned problems in the prior art
Circulation production confrontation network human face expression migration implementation method.
The purpose of the invention will be achieved through the following technical solutions: fights net based on the normalized circulation production of spectrum
The implementation method of the human face expression migration of network, method includes the following steps:
S1: various human face expression pictures are acquired, and are classified one by one according to human face expression, sorted human face expression number is obtained
According to collection；
S2: pre-processing the sorted human face expression data set picture that the S1 step obtains, and removes fuzzy photo, then
Employment face detection algorithm obtains five key points of face, and uniformly cuts face picture according to key point, obtains pretreated
Human face expression data set；
S3: the circulation production of building generator and arbiter composition fights network, and the pretreatment that S2 step is obtained
Loss function is calculated in human face expression data set input network afterwards and is trained, and trained Maker model is obtained；
S4: the tool that the trained generator of acquisition of the S3 step is migrated as human face expression, and be applied to real
It surveys.
Preferably, in the S1 step, acquisition human face expression picture needs classification balanced, and all kinds of human face expression pictures need
Will more than two thousand sheets, and need face is clear, posture rectify.
Preferably, in the S2 step, fuzzy photo is removed, then employment face detection algorithm obtains five keys of face
Point, and uniformly cut face picture according to key point, such as has that certain class human face expression picture is less, then to the progress of this kind of pictures
Data enhancing.
Preferably, in the S2 step, it includes generator G that circulation, which generates confrontation network, and arbiter D, generator G are negative
Only hot vector of human face expression picture and human face expression is generated its color and pays attention to trying hard to pay attention to trying hard to mask by duty, arbiter D
Be responsible for calculating the only hot vector of human face expression of probability and input picture that input picture is real human face expression picture, arbiter and
The human face expression that generator all only needs a model that multiple domain can be completed is converted.
Preferably, the generator is made of four parts: convolutional layer, warp lamination, residual error network block and convolution pay attention to
Power module；Convolutional layer extracts the characteristic information of picture by convolution operation, and with the multiple-layer stacked of convolution operation, the letter of extraction
Breath is more and more abstract；Residual error network block in network structure is used in order to which low-level image feature is passed into high level, and inhibits ladder
The case where degree disappears；The convolutional layer superposition of multilayer is equivalent to an encoder, extracts high latitude information；By the warp lamination of multilayer
Superposition is equivalent to a decoder；Decoding process restores low-level features from feature vector；The arbiter is by two parts
Composition: convolutional layer, convolution pay attention to power module, and the input of arbiter is a picture, and output is that input picture is true picture
The only hot vector of the human face expression of probability and input picture.
Preferably, the convolution notices that power module includes that channel pays attention to power module and space transforms power module, and channel pays attention to
Power module enhances important channel information, space transforms power module is used for the space of weighted feature figure for weighting channel characteristics
Feature, important spatial information in Enhanced feature figure.
Preferably, it is realized in generator and arbiter by constraining the spectral norm of each convolution in the S3 step entire
It is continuous that generator and the nonlinear mapping function of arbiter fitting meet Lipschitz, is conducive to generator fitting truthful data
Distribution；Pay attention to trying hard to pay attention to mask to try hard to the synthesis process with true picture by the color that generator generates in the S3 step
In, as shown in Figure 2: first color is paid attention to trying hard to pay attention to trying hard to carry out being multiplied to obtain face change regional value point by point with mask,
Mask is noticed that try hard to reversion is multiplied to obtain face constant region thresholding with true picture point by point, finally carries out point-by-point phase for the two again
The human face expression picture that add operation is synthesized.
Preferably, both the arbiter and generator group are combined into arbiter loss function and classification loss function, circulation
Loss function is that desired picture image after generator converts human face expression twice is consistent, and the expression of last time is solely hot
Solely hot vector is consistent for the expression of vector sum original image, constitutes a circulation.
Preferably, it is trained in the S3 step to based on the normalized circulation production confrontation network of spectrum, including step
It is rapid:
S31: by solely hot vector input is given birth to by true human face expression a picture and human face expression b described in the S2 step
The G that grows up to be a useful person obtains its color and pays attention to trying hard to pay attention to trying hard to mask, then the color is paid attention to trying hard to paying attention to trying hard to mask and really
The human face expression b picture that is synthetically generated of human face expression a picture, the human face expression b picture input arbiter D of generation is obtained
The human face expression b picture of generation is the human face expression of genuine probability and predictionOnly hot vector, minimizes the human face expression of prediction
The only hot vector of only hot vector sum human face expression b；
S32: by the human face expression b picture of generation described in the S31 step and human face expression a, solely hot vector inputs generator
G obtains its color and pays attention to trying hard to pay attention to trying hard to mask, then the color is paid attention to trying hard to the people for paying attention to trying hard to and generating with mask
The human face expression a picture that face expression b picture is synthetically generated minimizes the human face expression a picture and real human face expression a figure of generation
Piece；
S33: being trained the generator G and arbiter D, so that the loss function of network minimizes.
Preferably, loss function designs in the S33 step are as follows:
L (G, D)=LGAN(G, D)+λ Lcyc(G)+βLTV(A)+χLlabel(D)
Wherein,
In the formula, G is generator, and D is arbiter, DIIt is that arbiter prediction inputs picture as the general of human face expression figure
Rate, DlabelIt is the only hot vector of human face expression of arbiter prediction input picture, A is that mask pays attention to trying hard to, AI, jFor mask attention
I-th row j of figure is arranged, xaIt is the human face expression a picture in training sample, a xaSolely hot vector, b are training to corresponding human face expression
Solely hot vector, x ' are the generations obtained by human face expression a picture and only hot vector b by generator to the human face expression of Shi Zhiding
Human face expression b picture, x " is by the people of the human face expression b picture that generates and only hot vector a generation obtained by generator
Face expression a picture,It is xaPicture and only hot vector b pay attention to trying hard to by the mask that generator obtains,It is xaFigure
Piece and only hot vector b pay attention to trying hard to by the color that generator obtains, AG (x ', a)It is x ' picture and solely hot vector a passes through generation
The mask that device obtains pays attention to trying hard to, CG (x ', a)It is that x ' picture and only hot vector a pay attention to trying hard to by the color that generator obtains, λ,
β, χ are hyper parameter,To be multiplied point by point, LGANIt is arbiter loss, LcycIt is circulation loss, LTVIt is full variational regularization loss,
LlabelIt is Tag Estimation loss
The invention adopts the above technical scheme compared with prior art, has following technical effect that this method can solve to scheme
The problems such as piece local feature cannot focus on local feature and striated noise during migrating.Based on the normalized circulation of spectrum
Production confrontation network enables to a generator to realize a variety of human face expression migrations, and the human face expression that mesh generates can be more
Add naturally, there is preferable robustness.
Detailed description of the invention
Fig. 1 is that the present invention is based on the implementation methods for the human face expression migration for composing normalized circulation production confrontation network
Flow chart.
Fig. 2 is that the present invention is based on the structure charts for composing generator in normalized circulation production confrontation network.
Fig. 3 is that the present invention is based on the structure charts for composing residual error network block in normalized circulation production confrontation network.
Fig. 4 is that the present invention is based on compose normalized circulation production to fight the structure chart that convolution in network pays attention to power module.
Fig. 5 is that the present invention is based on compose convolution in normalized circulation production confrontation network to notice that the channel of power module pays attention to
The structure chart of power module.
Fig. 6 is that the present invention is based on compose normalized circulation production to fight the space transforms that convolution in network pays attention to power module
The structure chart of power module.
Fig. 7 is that the present invention is based on the structure charts for composing arbiter in normalized circulation production confrontation network.
Fig. 8 is that the present invention is based on compose normalized circulation production to fight the process that generator in network is synthetically generated picture
Figure.
Specific embodiment
The purpose of the present invention, advantage and feature, by by the non-limitative illustration of preferred embodiment below carry out diagram and
It explains.These embodiments are only the prominent examples using technical solution of the present invention, it is all take equivalent replacement or equivalent transformation and
The technical solution of formation, all falls within the scope of protection of present invention.
Present invention discloses a kind of realizations based on the human face expression migration for composing normalized circulation production confrontation network
Method, specific process is as shown in Fig. 1, method includes the following steps:
S1: various human face expression pictures are acquired, and are classified one by one according to human face expression, sorted human face expression number is obtained
According to collection；
It is specific as follows: to find picture website, find human face expression picture and guarantee that picture is relatively clear.Utilize crawler skill
Art crawls all kinds of human face expression pictures from website respectively, and guarantees that the human face expression picture of every one kind is greater than 1,000.
S2: pre-processing sorted human face expression data set picture, removes fuzzy photo, then employment face detection algorithm obtains
To five key points of face, and face picture is uniformly cut according to key point, obtains pretreated human face expression data set；
Screening picture one by one, the picture that removal obscures and image content is not inconsistent.By the unified cutting of the picture screened
For 128*128 size, saved respectively according to all kinds of expressions of facial image.
S3: the circulation production of building generator and arbiter composition fights network, and by pretreated human face expression
Loss function is calculated in data set input network and is trained, and trained Maker model is obtained；
The network structure of generator is as shown in Fig. 2.Generator is made of four parts: convolutional layer, warp lamination, residual error
Network block and convolution pay attention to power module.Convolutional layer extracts the characteristic information of picture by convolution operation, and with convolution operation
Multiple-layer stacked, the information of extraction is more and more abstract.Residual error network block in network structure is as shown in Fig. 3, residual error network block
It is and to inhibit the phenomenon that gradient disappears in order to which low-level image feature is passed into high level.The convolutional layer superposition of multilayer is equivalent to
One encoder extracts high latitude information, and the deconvolution layer superposition of multilayer is equivalent to a decoder.
It joined convolution attention mechanism module in the S3 step in the network structure of generator and arbiter.Convolution
Notice that power module is as shown in Fig. 4: paying attention to power module and space transforms power module comprising channel.Channel pays attention to power module such as attached drawing
Shown in 5, space transforms power module is as shown in Fig. 6.Channel notices that power module is to enhance important lead to weight channel characteristics
Road information.Space transforms power module is the important spatial information in Enhanced feature figure for the space characteristics of weighted feature figure.It is raw
The input grown up to be a useful person is a kind of picture, and output is that color pays attention to trying hard to pay attention to trying hard to mask.
The continuity of whole network is constrained in the network structure of generator and arbiter in the S3 step.Its
In realize that it is continuous that whole network meets Lipschitz by constraining the spectral norm of each convolution in arbiter and generator.
It includes generator G, arbiter D that circulation, which generates confrontation network, in the S3 step.Generator G is responsible for human face expression
Only hot vector of picture and human face expression generates its color and pays attention to trying hard to pay attention to trying hard to mask, and arbiter D is responsible for calculating input
Picture is the probability of real human face expression picture and the only hot vector of human face expression of input picture.Arbiter and generator all only need
Want a model that the human face expression conversion of multiple domain can be completed.
The network structure of arbiter is as shown in Fig. 7, and arbiter is made of two parts: convolutional layer, convolution attention mould
Block.The input of arbiter is a picture, and output is the human face expression of the probability that input picture is true picture and input picture
Only hot vector.
Loss function is to lose letter by arbiter loss function, circulation loss function, full variation difference loss function and classification
Array synthesis.Arbiter be desirable to accurately judge picture whether be true picture and input picture expression classification,
Generator is then desirable to the facial image according to input and the human face expression specified the solely corresponding human face expression of hot vector generation
Image, i.e. expectation arbiter judge that the picture generated can accurately identify the human face expression of generation for true picture and arbiter
The classification of image, the two group are combined into arbiter loss function and classification loss function.
Circulation loss function is then that expectation picture image after generator conversion twice human face expression is consistent and last
Solely hot vector is consistent for the expression of the only hot vector sum original image of primary expression, constitutes a circulation.Full variation difference loss function
The mask for being to be able to that generator is generated notices that trying hard to A has preferable continuity, and mask is paid attention to
Try hard to change face as small as possible during i.e. human face expression small as far as possible converts.
The training process, specifically:
S31: by solely hot vector inputs generator G by true human face expression a picture and human face expression b described in S2 step
It obtains its color to pay attention to trying hard to pay attention to trying hard to mask, then this color is paid attention to trying hard to paying attention to trying hard to and true face with mask
The human face expression b picture that expression a picture is synthetically generated.
The human face expression b picture that the human face expression b picture input arbiter D of generation is generated is for genuine probability and in advance
The human face expression of surveyOnly hot vector.Minimize the human face expression of predictionThe only hot vector of only hot vector sum human face expression b.
S32: by the human face expression b picture of generation described in S31 step and human face expression a, solely hot vector input generator G is obtained
Pay attention to trying hard to pay attention to trying hard to mask to its color, then this color is paid attention to trying hard to the face table for paying attention to trying hard to and generating with mask
The human face expression a picture that feelings b picture is synthetically generated.Synthesis process is as shown in Fig. 8, minimizes the human face expression a picture of generation
With real human face expression a picture.
S33: being trained the generator G and arbiter D, so that the loss function of network minimizes.
Loss function are as follows:
L (G, D)=LGAN(G, D)+λ Lcyc(G)+βLTV(A)+χLlabel(D)
Wherein,
G is generator in the formula, and D is arbiter, DIIt is that arbiter prediction inputs picture as the general of human face expression figure
Rate, DlabelIt is the only hot vector of human face expression of arbiter prediction input picture, A is that mask pays attention to trying hard to, AI, jFor mask attention
I-th row j of figure is arranged, xaIt is the human face expression a picture in training sample, a xaSolely hot vector, b are training to corresponding human face expression
Solely hot vector (different and a), x ' are to be obtained by human face expression a picture and only hot vector b by generator to the human face expression of Shi Zhiding
The human face expression b picture for the generation arrived, x " are that the human face expression b picture and only hot vector a by generating are obtained by generator
The human face expression a picture of generation,It is xaPicture and only hot vector b pay attention to trying hard to by the mask that generator obtains,It is xaPicture and only hot vector b pay attention to trying hard to by the color that generator obtains, AG (x ', a)It is x ' picture and only hot vector
A pays attention to trying hard to by the mask that generator obtains, CG (x ', a)It is the color note that x ' picture and only hot vector a are obtained by generator
Meaning is tried hard to, and λ, β, χ are hyper parameter,To be multiplied point by point. LGANIt is arbiter loss, LcycIt is circulation loss, LTVBe full variation just
Then change loss, LlabelIt is Tag Estimation loss.
S4: the tool that trained generator is migrated as human face expression is obtained, and is applied to actual measurement.
To sum up, the present invention is by the way that by human face expression picture and corresponding human face expression, solely hot vector is input to based on spectrum normalizing
In the circulation production confrontation network of change, training pattern obtains training perfect generator G.The first generator G can be incited somebody to action at this time
According to the facial image of input and specified human face expression, solely hot vector generates the human face expression picture of correspondence image.It will be based on spectrum
Normalized circulation production confrontation role of network has accomplished the good transformation of human face expression in human face expression migration application,
Optimize the problem of circulation generates a shortcomings that model can only train a kind of expression to migrate in confrontation network and easy over-fitting.
Still there are many embodiment, all technical sides formed using equivalents or equivalent transformation by the present invention
Case is within the scope of the present invention.
Claims (10)
1. the implementation method based on the human face expression migration for composing normalized circulation production confrontation network, it is characterised in that: should
Method the following steps are included:
S1: various human face expression pictures are acquired, and are classified one by one according to human face expression, sorted human face expression data are obtained
Collection；
S2: the sorted human face expression data set picture that the S1 step obtains is pre-processed, fuzzy photo, then employment are removed
Face detection algorithm obtains five key points of face, and uniformly cuts face picture according to key point, obtains pretreated face
Expression data collection；
S3: the circulation production of building generator and arbiter composition fights network, and S2 step is obtained pretreated
Loss function is calculated in human face expression data set input network and is trained, and trained Maker model is obtained；
S4: the tool that the trained generator of acquisition of the S3 step is migrated as human face expression, and it is applied to actual measurement.
2. the realization according to claim 1 based on the human face expression migration for composing normalized circulation production confrontation network
Method, it is characterised in that: in the S1 step, acquisition human face expression picture needs classification balanced, all kinds of human face expression pictures
Need more than two thousand sheets, and need face is clear, posture rectify.
3. the realization according to claim 1 based on the human face expression migration for composing normalized circulation production confrontation network
Method, it is characterised in that: in the S2 step, remove fuzzy photo, then employment face detection algorithm obtains five keys of face
Point, and uniformly cut face picture according to key point, such as has that certain class human face expression picture is less, then to the progress of this kind of pictures
Data enhancing.
4. the realization according to claim 1 based on the human face expression migration for composing normalized circulation production confrontation network
Method, it is characterised in that: in the S2 step, it includes generator G that circulation, which generates confrontation network, and arbiter D, generator G are negative
Only hot vector of human face expression picture and human face expression is generated its color and pays attention to trying hard to pay attention to trying hard to mask by duty, arbiter D
Be responsible for calculating the only hot vector of human face expression of probability and input picture that input picture is real human face expression picture, arbiter and
The human face expression that generator all only needs a model that multiple domain can be completed is converted.
5. the realization according to claim 4 based on the human face expression migration for composing normalized circulation production confrontation network
Method, it is characterised in that: the generator is made of four parts: convolutional layer, warp lamination, residual error network block and convolution pay attention to
Power module；Convolutional layer extracts the characteristic information of picture by convolution operation, and with the multiple-layer stacked of convolution operation, the letter of extraction
Breath is more and more abstract；Residual error network block in network structure is used in order to which low-level image feature is passed into high level, and inhibits ladder
The case where degree disappears；The convolutional layer superposition of multilayer is equivalent to an encoder, extracts high latitude information；By the warp lamination of multilayer
Superposition is equivalent to a decoder；Decoding process restores low-level features from feature vector；The arbiter is by two parts
Composition: convolutional layer, convolution pay attention to power module, and the input of arbiter is a picture, and output is that input picture is true picture
The only hot vector of the human face expression of probability and input picture.
6. the realization according to claim 5 based on the human face expression migration for composing normalized circulation production confrontation network
Method, it is characterised in that: the convolution notices that power module includes that channel pays attention to power module and space transforms power module, and channel pays attention to
Power module enhances important channel information, space transforms power module is used for the space of weighted feature figure for weighting channel characteristics
Feature, important spatial information in Enhanced feature figure.
7. the realization according to claim 4 based on the human face expression migration for composing normalized circulation production confrontation network
Method, it is characterised in that: whole to realize by the spectral norm for constraining each convolution in generator and arbiter in the S3 step
It is continuous that a generator and the nonlinear mapping function of arbiter fitting meet Lipschitz, is conducive to the true number of generator fitting
According to distribution；Pay attention to trying hard to paying attention to mask trying hard to synthesizing with true picture by the color that generator generates in the S3 step
Cheng Zhong, as shown in Figure 2: paying attention to trying hard to pay attention to trying hard to carry out being multiplied to obtain face change region point by point with mask by color first
Value, then mask is paid attention to trying hard to reversion and is multiplied to obtain face constant region thresholding point by point with true picture, finally by the two progress by
The human face expression picture that point phase add operation is synthesized.
8. the realization according to claim 4 based on the human face expression migration for composing normalized circulation production confrontation network
Method, it is characterised in that: both the arbiter and generator group are combined into arbiter loss function and classification loss function, circulation
Loss function is that desired picture image after generator converts human face expression twice is consistent, and the expression of last time is solely hot
Solely hot vector is consistent for the expression of vector sum original image, constitutes a circulation.
9. the realization according to claim 1 based on the human face expression migration for composing normalized circulation production confrontation network
Method, it is characterised in that: be trained in the S3 step to based on the normalized circulation production confrontation network of spectrum, including step
It is rapid:
S31: by solely hot vector inputs generator G by true human face expression a picture and human face expression b described in the S2 step
It obtains its color to pay attention to trying hard to pay attention to trying hard to mask, then the color is paid attention to trying hard to paying attention to trying hard to and true face with mask
The human face expression b picture that expression a picture is synthetically generated, the people that the human face expression b picture input arbiter D of generation is generated
Face expression b picture is the human face expression of genuine probability and predictionOnly hot vector, minimizes the human face expression of predictionOnly hot vector
With the only hot vector of human face expression b；
S32: by the human face expression b picture of generation described in the S31 step and human face expression a, solely hot vector input generator G is obtained
Pay attention to trying hard to pay attention to trying hard to mask to its color, then the color is paid attention to trying hard to the face table for paying attention to trying hard to and generating with mask
The human face expression a picture that feelings b picture is synthetically generated minimizes the human face expression a picture and real human face expression a picture of generation；
S33: being trained the generator G and arbiter D, so that the loss function of network minimizes.
10. the reality according to claim 1 based on the human face expression migration for composing normalized circulation production confrontation network
Existing method, it is characterised in that:
Loss function designs in the S33 step are as follows:
L (G, D)=LGAN(G, D)+λ Lcyc(G)+βLTV(A)+χLlabel(D)
Wherein,
In the formula, G is generator, and D is arbiter, DIIt is the probability that arbiter prediction input picture is human face expression figure,
DlabelIt is the only hot vector of human face expression of arbiter prediction input picture, A is that mask pays attention to trying hard to, AI, jPay attention to trying hard to for mask
The i-th row j column, xaIt is the human face expression a picture in training sample, a xaSolely hot vector, b are when training to corresponding human face expression
Solely hot vector, x ' are by the people of human face expression a picture and only hot vector b generation obtained by generator to specified human face expression
Face expression b picture, x " are the human face expression b picture and the face table of only hot vector a generation obtained by generator by generating
Feelings a picture,It is xaPicture and only hot vector b pay attention to trying hard to by the mask that generator obtains,It is xaPicture and
Only hot vector b pays attention to trying hard to by the color that generator obtains, AG (x ', a)It is that x ' picture and only hot vector a are obtained by generator
Mask pay attention to trying hard to, CG (x ', a)It is that x ' picture and only hot vector a pay attention to trying hard to by the color that generator obtains, λ, β, χ are
Hyper parameter,To be multiplied point by point, LGANIt is arbiter loss, LcycIt is circulation loss, LTVIt is full variational regularization loss, LlabelIt is
Tag Estimation loss.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910240461.9A CN110084121A (en) | 2019-03-27 | 2019-03-27 | Implementation method based on the human face expression migration for composing normalized circulation production confrontation network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910240461.9A CN110084121A (en) | 2019-03-27 | 2019-03-27 | Implementation method based on the human face expression migration for composing normalized circulation production confrontation network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110084121A true CN110084121A (en) | 2019-08-02 |
Family
ID=67413824
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910240461.9A Withdrawn CN110084121A (en) | 2019-03-27 | 2019-03-27 | Implementation method based on the human face expression migration for composing normalized circulation production confrontation network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110084121A (en) |
Cited By (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110503601A (en) * | 2019-08-28 | 2019-11-26 | 上海交通大学 | Face based on confrontation network generates picture replacement method and system |
| CN110633665A (en) * | 2019-09-05 | 2019-12-31 | 卓尔智联(武汉)研究院有限公司 | Recognition method, device and storage medium |
| CN110705406A (en) * | 2019-09-20 | 2020-01-17 | 五邑大学 | Face beauty prediction method and device based on transfer learning resistance |
| CN110718232A (en) * | 2019-09-23 | 2020-01-21 | 东南大学 | Speech enhancement method for generating countermeasure network based on two-dimensional spectrogram and condition |
| CN110728319A (en) * | 2019-09-30 | 2020-01-24 | 中国科学院深圳先进技术研究院 | Image generation method and device and computer storage medium |
| CN110852292A (en) * | 2019-11-18 | 2020-02-28 | 南京邮电大学 | Sketch face recognition method based on cross-modal multi-task depth measurement learning |
| CN111091059A (en) * | 2019-11-19 | 2020-05-01 | 佛山市南海区广工大数控装备协同创新研究院 | Data equalization method in household garbage plastic bottle classification |
| CN111192343A (en) * | 2019-12-17 | 2020-05-22 | 南京理工大学 | Face expression synthesis method based on generative confrontation network |
| CN111275779A (en) * | 2020-01-08 | 2020-06-12 | 网易（杭州）网络有限公司 | Expression migration method, training method and device of image generator and electronic equipment |
| CN111369646A (en) * | 2020-03-09 | 2020-07-03 | 南京理工大学 | Expression synthesis method integrating attention mechanism |
| CN111445484A (en) * | 2020-04-01 | 2020-07-24 | 华中科技大学 | Image-level labeling-based industrial image abnormal area pixel level segmentation method |
| CN111640057A (en) * | 2020-05-25 | 2020-09-08 | 武汉理工大学 | Hidden variable decoupling-based human face image local feature migration network and method |
| CN111652121A (en) * | 2020-06-01 | 2020-09-11 | 腾讯科技（深圳）有限公司 | Training method of expression migration model, and expression migration method and device |
| CN111667559A (en) * | 2020-05-27 | 2020-09-15 | 西北工业大学 | Polymorphic human face emotion generation method based on deep migration network |
| CN111738230A (en) * | 2020-08-05 | 2020-10-02 | 深圳市优必选科技股份有限公司 | Face recognition method, face recognition device and electronic equipment |
| CN111798546A (en) * | 2020-06-03 | 2020-10-20 | 浙江大学 | Multi-user expression migration method based on generation countermeasure mechanism |
| CN111860380A (en) * | 2020-07-27 | 2020-10-30 | 平安科技（深圳）有限公司 | Face image generation method, device, server and storage medium |
| CN111899160A (en) * | 2020-08-03 | 2020-11-06 | 上海交通大学 | Multi-instance image conversion method, system, computer-readable storage medium and device |
| CN112614070A (en) * | 2020-12-28 | 2021-04-06 | 南京信息工程大学 | DefogNet-based single image defogging method |
| WO2021073417A1 (en) * | 2019-10-18 | 2021-04-22 | 平安科技（深圳）有限公司 | Expression generation method and apparatus, device and storage medium |
| CN112733954A (en) * | 2021-01-20 | 2021-04-30 | 湖南大学 | Abnormal traffic detection method based on generation countermeasure network |
| CN112767519A (en) * | 2020-12-30 | 2021-05-07 | 电子科技大学 | Controllable expression generation method combined with style migration |
| CN113343924A (en) * | 2021-07-01 | 2021-09-03 | 齐鲁工业大学 | Modulation signal identification method based on multi-scale cyclic spectrum feature and self-attention generation countermeasure network |
| CN113689527A (en) * | 2020-05-15 | 2021-11-23 | 武汉Tcl集团工业研究院有限公司 | Training method of face conversion model and face image conversion method |
| CN113807265A (en) * | 2021-09-18 | 2021-12-17 | 山东财经大学 | Diversified human face image synthesis method and system |
| CN113918716A (en) * | 2021-10-14 | 2022-01-11 | 中山大学 | Method and device for constructing generation confrontation topic model based on spectrum norm normalization |
| CN114140320A (en) * | 2021-12-09 | 2022-03-04 | 北京百度网讯科技有限公司 | Image migration method and training method and device of image migration model |
| CN114399829A (en) * | 2022-03-25 | 2022-04-26 | 浙江壹体科技有限公司 | Posture migration method based on generative countermeasure network, electronic device and medium |
| CN116958468A (en) * | 2023-07-05 | 2023-10-27 | 中国科学院地理科学与资源研究所 | Mountain snow environment simulation method and system based on SCycleGAN |
-
2019
- 2019-03-27 CN CN201910240461.9A patent/CN110084121A/en not_active Withdrawn
Cited By (46)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110503601A (en) * | 2019-08-28 | 2019-11-26 | 上海交通大学 | Face based on confrontation network generates picture replacement method and system |
| CN110633665A (en) * | 2019-09-05 | 2019-12-31 | 卓尔智联(武汉)研究院有限公司 | Recognition method, device and storage medium |
| CN110633665B (en) * | 2019-09-05 | 2023-01-10 | 卓尔智联(武汉)研究院有限公司 | Identification method, device and storage medium |
| CN110705406B (en) * | 2019-09-20 | 2022-11-15 | 五邑大学 | Face beauty prediction method and device based on transfer learning resistance |
| CN110705406A (en) * | 2019-09-20 | 2020-01-17 | 五邑大学 | Face beauty prediction method and device based on transfer learning resistance |
| CN110718232A (en) * | 2019-09-23 | 2020-01-21 | 东南大学 | Speech enhancement method for generating countermeasure network based on two-dimensional spectrogram and condition |
| CN110728319A (en) * | 2019-09-30 | 2020-01-24 | 中国科学院深圳先进技术研究院 | Image generation method and device and computer storage medium |
| CN110728319B (en) * | 2019-09-30 | 2023-09-29 | 中国科学院深圳先进技术研究院 | Image generation method and device and computer storage medium |
| WO2021073417A1 (en) * | 2019-10-18 | 2021-04-22 | 平安科技（深圳）有限公司 | Expression generation method and apparatus, device and storage medium |
| CN110852292A (en) * | 2019-11-18 | 2020-02-28 | 南京邮电大学 | Sketch face recognition method based on cross-modal multi-task depth measurement learning |
| CN111091059A (en) * | 2019-11-19 | 2020-05-01 | 佛山市南海区广工大数控装备协同创新研究院 | Data equalization method in household garbage plastic bottle classification |
| CN111192343A (en) * | 2019-12-17 | 2020-05-22 | 南京理工大学 | Face expression synthesis method based on generative confrontation network |
| CN111192343B (en) * | 2019-12-17 | 2023-03-24 | 南京理工大学 | Face expression synthesis method based on generative confrontation network |
| CN111275779A (en) * | 2020-01-08 | 2020-06-12 | 网易（杭州）网络有限公司 | Expression migration method, training method and device of image generator and electronic equipment |
| CN111369646A (en) * | 2020-03-09 | 2020-07-03 | 南京理工大学 | Expression synthesis method integrating attention mechanism |
| CN111369646B (en) * | 2020-03-09 | 2023-03-24 | 南京理工大学 | Expression synthesis method integrating attention mechanism |
| CN111445484A (en) * | 2020-04-01 | 2020-07-24 | 华中科技大学 | Image-level labeling-based industrial image abnormal area pixel level segmentation method |
| CN113689527A (en) * | 2020-05-15 | 2021-11-23 | 武汉Tcl集团工业研究院有限公司 | Training method of face conversion model and face image conversion method |
| CN111640057A (en) * | 2020-05-25 | 2020-09-08 | 武汉理工大学 | Hidden variable decoupling-based human face image local feature migration network and method |
| CN111640057B (en) * | 2020-05-25 | 2022-04-15 | 武汉理工大学 | Hidden variable decoupling-based human face image local feature migration network and method |
| CN111667559A (en) * | 2020-05-27 | 2020-09-15 | 西北工业大学 | Polymorphic human face emotion generation method based on deep migration network |
| WO2021244217A1 (en) * | 2020-06-01 | 2021-12-09 | 腾讯科技（深圳）有限公司 | Method for training expression transfer model, and expression transfer method and apparatus |
| CN111652121A (en) * | 2020-06-01 | 2020-09-11 | 腾讯科技（深圳）有限公司 | Training method of expression migration model, and expression migration method and device |
| CN111652121B (en) * | 2020-06-01 | 2023-11-03 | 腾讯科技（深圳）有限公司 | Training method of expression migration model, and method and device for expression migration |
| CN111798546B (en) * | 2020-06-03 | 2022-04-26 | 浙江大学 | Multi-user expression migration method based on generation countermeasure mechanism |
| CN111798546A (en) * | 2020-06-03 | 2020-10-20 | 浙江大学 | Multi-user expression migration method based on generation countermeasure mechanism |
| CN111860380A (en) * | 2020-07-27 | 2020-10-30 | 平安科技（深圳）有限公司 | Face image generation method, device, server and storage medium |
| CN111899160A (en) * | 2020-08-03 | 2020-11-06 | 上海交通大学 | Multi-instance image conversion method, system, computer-readable storage medium and device |
| CN111738230B (en) * | 2020-08-05 | 2020-12-15 | 深圳市优必选科技股份有限公司 | Face recognition method, face recognition device and electronic equipment |
| CN111738230A (en) * | 2020-08-05 | 2020-10-02 | 深圳市优必选科技股份有限公司 | Face recognition method, face recognition device and electronic equipment |
| CN112614070B (en) * | 2020-12-28 | 2023-05-30 | 南京信息工程大学 | defogNet-based single image defogging method |
| CN112614070A (en) * | 2020-12-28 | 2021-04-06 | 南京信息工程大学 | DefogNet-based single image defogging method |
| CN112767519A (en) * | 2020-12-30 | 2021-05-07 | 电子科技大学 | Controllable expression generation method combined with style migration |
| CN112767519B (en) * | 2020-12-30 | 2022-04-19 | 电子科技大学 | Controllable expression generation method combined with style migration |
| CN112733954A (en) * | 2021-01-20 | 2021-04-30 | 湖南大学 | Abnormal traffic detection method based on generation countermeasure network |
| CN113343924B (en) * | 2021-07-01 | 2022-05-17 | 齐鲁工业大学 | Modulation signal identification method based on cyclic spectrum characteristics and generation countermeasure network |
| CN113343924A (en) * | 2021-07-01 | 2021-09-03 | 齐鲁工业大学 | Modulation signal identification method based on multi-scale cyclic spectrum feature and self-attention generation countermeasure network |
| CN113807265B (en) * | 2021-09-18 | 2022-05-06 | 山东财经大学 | Diversified human face image synthesis method and system |
| CN113807265A (en) * | 2021-09-18 | 2021-12-17 | 山东财经大学 | Diversified human face image synthesis method and system |
| CN113918716B (en) * | 2021-10-14 | 2023-06-02 | 中山大学 | Method and device for constructing generated countermeasure topic model based on spectrum norm normalization |
| CN113918716A (en) * | 2021-10-14 | 2022-01-11 | 中山大学 | Method and device for constructing generation confrontation topic model based on spectrum norm normalization |
| CN114140320B (en) * | 2021-12-09 | 2023-09-01 | 北京百度网讯科技有限公司 | Image migration method and training method and device of image migration model |
| CN114140320A (en) * | 2021-12-09 | 2022-03-04 | 北京百度网讯科技有限公司 | Image migration method and training method and device of image migration model |
| CN114399829A (en) * | 2022-03-25 | 2022-04-26 | 浙江壹体科技有限公司 | Posture migration method based on generative countermeasure network, electronic device and medium |
| CN114399829B (en) * | 2022-03-25 | 2022-07-05 | 浙江壹体科技有限公司 | Posture migration method based on generative countermeasure network, electronic device and medium |
| CN116958468A (en) * | 2023-07-05 | 2023-10-27 | 中国科学院地理科学与资源研究所 | Mountain snow environment simulation method and system based on SCycleGAN |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110084121A (en) | Implementation method based on the human face expression migration for composing normalized circulation production confrontation network | |
| Ren et al. | Single image dehazing via multi-scale convolutional neural networks with holistic edges | |
| CN109493303B (en) | Image defogging method based on generation countermeasure network | |
| CN109815826B (en) | Method and device for generating face attribute model | |
| CN108986058B (en) | Image fusion method for brightness consistency learning | |
| CN110084734A (en) | A kind of big data ownership guard method being locally generated confrontation network based on object | |
| CN108549893A (en) | A kind of end-to-end recognition methods of the scene text of arbitrary shape | |
| CN108921123A (en) | A kind of face identification method based on double data enhancing | |
| CN110427875A (en) | Infrared image object detection method based on depth migration study and extreme learning machine | |
| CN112347888B (en) | Remote sensing image scene classification method based on bi-directional feature iterative fusion | |
| CN106599800A (en) | Face micro-expression recognition method based on deep learning | |
| CN110276402A (en) | A kind of salt body recognition methods based on the enhancing of deep learning semanteme boundary | |
| CN110008846B (en) | Image processing method | |
| CN110853119B (en) | Reference picture-based makeup transfer method with robustness | |
| CN111832573B (en) | Image emotion classification method based on class activation mapping and visual saliency | |
| CN109410135A (en) | It is a kind of to fight learning-oriented image defogging plus mist method | |
| CN111126464A (en) | Image classification method based on unsupervised domain confrontation field adaptation | |
| Frants et al. | QCNN-H: Single-image dehazing using quaternion neural networks | |
| CN112580636A (en) | Image aesthetic quality evaluation method based on cross-modal collaborative reasoning | |
| Sun et al. | Underwater image enhancement with encoding-decoding deep CNN networks | |
| CN112633234A (en) | Method, device, equipment and medium for training and applying face glasses-removing model | |
| CN111489304A (en) | Image deblurring method based on attention mechanism | |
| CN108009512A (en) | A kind of recognition methods again of the personage based on convolutional neural networks feature learning | |
| CN111126155A (en) | Pedestrian re-identification method for generating confrontation network based on semantic constraint | |
| CN111489405B (en) | Face sketch synthesis system for generating confrontation network based on condition enhancement |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20190802 |