CN111144563A - Method for training generation countermeasure network based on dual distance loss - Google Patents
Method for training generation countermeasure network based on dual distance loss Download PDFInfo
- Publication number
- CN111144563A CN111144563A CN201911345589.8A CN201911345589A CN111144563A CN 111144563 A CN111144563 A CN 111144563A CN 201911345589 A CN201911345589 A CN 201911345589A CN 111144563 A CN111144563 A CN 111144563A
- Authority
- CN
- China
- Prior art keywords
- training
- dual distance
- distance loss
- generator
- distribution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000012549 training Methods 0.000 title claims abstract description 43
- 230000009977 dual effect Effects 0.000 title claims abstract description 39
- 238000000034 method Methods 0.000 title claims abstract description 34
- 230000006870 function Effects 0.000 claims abstract description 26
- 238000013528 artificial neural network Methods 0.000 claims abstract description 16
- 238000011478 gradient descent method Methods 0.000 claims abstract description 7
- 238000007781 pre-processing Methods 0.000 claims abstract description 6
- 238000005457 optimization Methods 0.000 claims description 8
- 230000008901 benefit Effects 0.000 abstract description 5
- 238000013135 deep learning Methods 0.000 abstract description 2
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical group C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 description 1
- 230000003213 activating effect Effects 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method for training a generation confrontation network based on dual distance loss, and relates to the technical field of deep learning neural networks. The invention comprises the following steps: step S1: acquiring a data set of target distribution, and preprocessing the data set; step S2: setting the structures and parameters of a generator and a discriminator neural network and the learning rate in the training process; step S3: and (3) calculating a dual distance loss function according to parameters of the neural network, and training a generator to generate real distribution by adopting a random gradient descent method based on the dual distance loss function. According to the invention, more accurate results are obtained under the condition of the same iteration steps, the training quality can be improved, and the cost benefit is better.
Description
Technical Field
The invention belongs to the technical field of deep learning neural networks, and particularly relates to a method for training a generation confrontation network based on dual distance loss.
Background
The generation of the confrontation network is a kind of neural network, and the discriminators and the generators are trained in turn to confront each other to sample from complex probability distribution, such as generating pictures, characters, voice and the like.
If the original generators and discriminators are random, it is difficult to determine whether the generators and discriminators can converge to an ideal conclusion through training of given data. While it can be shown that under some strong assumptions, generators and discriminators can converge to local nash equilibrium, many generation-confrontation network algorithms do not converge globally.
Disclosure of Invention
The invention aims to provide a method for training a generation countermeasure network based on dual distance loss, which comprises the steps of preprocessing a data set obtained by target distribution, setting structures, parameters and learning rate in a training process of a generator and a discriminator network, calculating a dual distance loss function according to parameters of an applied neural network, training the generator to generate real distribution based on the random gradient descent method adopted by the dual distance loss function, and solving the problems that the existing generation network countermeasure algorithm can not achieve global convergence and the training result is not accurate enough.
In order to solve the technical problems, the invention is realized by the following technical scheme:
the invention relates to a method for training a generation confrontation network based on dual distance loss, which comprises the following steps:
step S1: acquiring a data set of target distribution, and preprocessing the data set;
step S2: setting the structures and parameters of a generator and a discriminator neural network and the learning rate in the training process;
step S3: and (3) calculating a dual distance loss function according to parameters of the neural network, and training a generator to generate real distribution by adopting a random gradient descent method based on the dual distance loss function. Preferably, after the step S3, an empirical dual distance between the target real distribution and the generated distribution needs to be calculated:
wherein x isiIs a sample point in the true distribution, ziIs a sample point in the Gaussian distribution, m and n are positive integers, f is a discriminator, g is a generator,andrespectively, the space where the arbiter and the generator are located.
Preferably, the method for generating the confrontation network training calculates perturbation points, and then determines a dual distance loss function and an optimization direction by using the perturbation points, including the following steps:
an initialization step: the target data set is processed. Given the arbiter f in the initial state0Sum generator g0Setting parameters gamma epsilon (0, 2) and k as 0, and giving two positive integers m and n;
random selection of data points: select m points in the target data set, and record as { x1,...,xmSelecting n points in a specified Gaussian noise, and recording the points as z1,...,zn};
And a step of calculating the shooting point: for a given generator gkSum discriminator fkConsider thatCalculating perturbation pointsAnd
and (3) calculating an optimization direction: consideration functionThe sub-gradient of (A) is recorded asAndthen, considering the optimization direction, respectively
Preferably, the generative confrontation network training method satisfies at least with a probability of 1-3 δ:
the invention has the following beneficial effects:
the invention carries out preprocessing by acquiring a data set of target distribution, sets the structures, parameters and learning rate in the training process of a generator and a discriminator network, calculates a dual distance loss function according to the parameters of an applied neural network, trains the generator to generate real distribution by adopting a random gradient descent method based on the dual distance loss function, obtains more accurate results under the condition of the same iteration step number, can improve the training quality and has better cost benefit.
Of course, it is not necessary for any product in which the invention is practiced to achieve all of the above-described advantages at the same time.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart illustrating a method for generating a confrontation network training according to the present invention.
Fig. 2 is a comparison graph of the generated countermeasure network training method provided by the embodiment of the present invention with the generated result of the WGAN-GP method after 20000 iterations on the CIFAR10 data set.
Fig. 3 is a comparison graph of the generated confrontation network training method provided by the embodiment of the invention on a CIFAR10 data set and the inclusion Score obtained by the WGAN-GP method.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1-3, the present invention is a method for training a generation countermeasure network based on dual distance loss, comprising:
step S1: acquiring a data set of target distribution, and preprocessing the data set;
step S2: setting the structures and parameters of a generator and a discriminator neural network and the learning rate in the training process;
step S3: calculating a dual distance loss function according to parameters of the neural network, and training a generator to generate real distribution by adopting a random gradient descent method based on the dual distance loss function;
the method for generating the confrontation training based on the dual distance loss function can obtain more accurate results under the condition of the same iteration steps, can improve the training quality, and has better cost benefit and generalization performance.
After step S3, it is necessary to calculate the empirical dual distance between the target real distribution and the generated distribution:
wherein,xi is the sample point in the true distribution, ziIs a sample point in the Gaussian distribution, m and n are positive integers, f is a discriminator, g is a generator,andrespectively, the space where the arbiter and the generator are located.
Specifically, 2 fields are givenAndarbiter and generator satisfyA convex function phi, a true data distribution pdataAnd a Gaussian distribution pz,(f*,g*) Upper dual distance loss DG (f)*,g*) Is composed ofHere, the loss function for generating the countermeasure network isEmpirical dual distance lossSatisfy the requirement ofWherein the empirical loss function that generates the countermeasure network is:
if the true sample X and the Gaussian distribution sample Z are bounded and the boundary is bounded by BxAnd BzRepresents;so thatAndis provided withWherein L isfIs the Lipschitz constant, L, of the arbiter network fgA liphoz constant for the generator network; then with a probability of at least 1-3 δ there is equation (1):
at this time, a specific process of obtaining formula (1) is given, which may include:
the equation of formula (1) is simplified to the left as:
the McDiarmid inequality condition is:
wherein X ═ { X ═ X1,x2,...,xi,...,xn},X′={x1,x2,...,x′i,...,xn},ρφIs the Liphoz constant of φ.
where e is ∈1,∈2,...,∈n) And P (∈ C)i=1)=P(∈i-1) 0.5. Therefore, the probability of at least 1- δ is given by equation (4):
similarly, the probability of at least 1- δ has formula (5) and formula (6):
thus, the probability of at least 1-3 δ has equation (7):
since both the discriminator f and the generator g are neural networks, they can be written in the form of equations (8) and (9):
f=aH(MH(aH-1(MH-1(...a1(M1(·))...)))) (8);
g=bH′(NH′(bH′-1(NH′-1)...b1(N1(·))...)))) (9);
wherein, aiAnd biFor activating functions, MiAnd NiFor the matrix, a is the activation function Relu in the experimentiAnd biHas a lipschitz constant of less than 1; and assume | Mi||≤BiAnd Ni||≤B′i;dfAnd dgThe width of the neural network of the arbiter and generator.
According to the above assumptions, there is formula (10):
according to the relationship between the Ladamard Mach complexity and the coverage number, obtaining a formula (13):
similarly, formula (14) and formula (15) are obtained:
and (3) assuming that m > is greater than n, and combining the formula (7), the formula (13), the formula (14) and the formula (15), obtaining a generalization error bound based on the dual loss distance, namely the formula (1).
Specifically, in the method for training a generative confrontation based on dual distance loss according to the embodiment of the present invention, after setting the structure and parameters of the neural network and providing a data set conforming to the target distribution, solving the dual distance loss function by using a gradient descent method may include:
an initialization step: processing the target data set; given the arbiter f in the initial state0Sum generator g0Setting parameters gamma epsilon (0, 2) and k as 0, and giving two positive integers m and n;
random selection of data points: select m points in the target data set, and record as { x1,...,xmSelecting n points in a specified Gaussian noise, and recording the points as z1,...,zn};
Calculating the shooting point: for a given generator gkSum discriminator fkConsider thatCalculating perturbation pointsAnd
dual distance calculation: calculating a dual distance loss functionIf E iskIf 0, then the algorithm stops and the generator g is outputk;
And (3) calculating an optimization direction: consideration functionThe sub-gradient of (A) is recorded asAndthen, considering the optimization direction, respectively
The method for generating the confrontation training based on the dual distance loss function provided by the embodiment of the invention successfully realizes data distribution generation on MNIST and CIFAR 10; the result shows that, compared with the traditional method for generating the confrontation training, the method for generating the confrontation training based on the dual distance loss function provided by the embodiment of the invention can obtain a more accurate result under the condition of the same iteration step number, can improve the training quality and has better cost benefit.
Since each training requires generation of gaussian noise during the training process, the method for generating a confrontation training based on a dual distance loss function provided by this embodiment seems to be more complicated in terms of setting of the loss function, but does not increase a generalization error, and thus has the same generalization performance as the conventional method for generating a confrontation training.
It should be noted that, in the above system embodiment, each included unit is only divided according to functional logic, but is not limited to the above division as long as the corresponding function can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
Claims (4)
1. A method for generating a confrontation network training based on dual distance loss is characterized by comprising the following steps:
step S1: acquiring a data set of target distribution, and preprocessing the data set;
step S2: setting the structures and parameters of a generator and a discriminator neural network and the learning rate in the training process;
step S3: and (3) calculating a dual distance loss function according to parameters of the neural network, and training a generator to generate real distribution by adopting a random gradient descent method based on the dual distance loss function.
2. The method for training a generative confrontation network based on dual distance loss as claimed in claim 1, wherein after step S3, an empirical dual distance between the target real distribution and the generative distribution is calculated:
3. The method for generating confrontation network training based on dual distance loss as claimed in claim 1, wherein the method for generating confrontation network training calculates perturbation points, and then determines dual distance loss function and optimization direction by using the perturbation points, comprising the following steps:
an initialization step: the target data set is processed. Given the arbiter f in the initial state0Sum generator g0Setting parameters gamma epsilon (0, 2) and k as 0, and giving two positive integers m and n;
random selection of data points: select m points in the target data set, and record as { x1,...,xmSelecting n points in a specified Gaussian noise, and recording the points as z1,...,zn};
And a step of calculating the shooting point: for a given generator gkSum discriminator fkConsider thatCalculating perturbation pointsAnd
and (3) calculating an optimization direction: consideration functionThe sub-gradient of (A) is recorded asAndthen, considering the optimization direction, respectively
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911345589.8A CN111144563A (en) | 2019-12-24 | 2019-12-24 | Method for training generation countermeasure network based on dual distance loss |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911345589.8A CN111144563A (en) | 2019-12-24 | 2019-12-24 | Method for training generation countermeasure network based on dual distance loss |
Publications (1)
Publication Number | Publication Date |
---|---|
CN111144563A true CN111144563A (en) | 2020-05-12 |
Family
ID=70519634
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911345589.8A Withdrawn CN111144563A (en) | 2019-12-24 | 2019-12-24 | Method for training generation countermeasure network based on dual distance loss |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111144563A (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112488309A (en) * | 2020-12-21 | 2021-03-12 | 清华大学深圳国际研究生院 | Training method and system of deep neural network based on critical damping momentum |
CN112668239A (en) * | 2020-12-30 | 2021-04-16 | 山东交通学院 | Hybrid power truck fleet experience teaching method based on counterstudy |
CN112766489A (en) * | 2021-01-12 | 2021-05-07 | 合肥黎曼信息科技有限公司 | Method for training generation countermeasure network based on dual distance loss |
CN115205738A (en) * | 2022-07-05 | 2022-10-18 | 广州和达水务科技股份有限公司 | Emergency drainage method and system applied to urban inland inundation |
-
2019
- 2019-12-24 CN CN201911345589.8A patent/CN111144563A/en not_active Withdrawn
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112488309A (en) * | 2020-12-21 | 2021-03-12 | 清华大学深圳国际研究生院 | Training method and system of deep neural network based on critical damping momentum |
CN112488309B (en) * | 2020-12-21 | 2023-10-20 | 清华大学深圳国际研究生院 | Training method and system of deep neural network based on critical damping momentum |
CN112668239A (en) * | 2020-12-30 | 2021-04-16 | 山东交通学院 | Hybrid power truck fleet experience teaching method based on counterstudy |
CN112668239B (en) * | 2020-12-30 | 2022-11-15 | 山东交通学院 | Hybrid power truck fleet experience teaching method based on counterstudy |
CN112766489A (en) * | 2021-01-12 | 2021-05-07 | 合肥黎曼信息科技有限公司 | Method for training generation countermeasure network based on dual distance loss |
CN115205738A (en) * | 2022-07-05 | 2022-10-18 | 广州和达水务科技股份有限公司 | Emergency drainage method and system applied to urban inland inundation |
CN115205738B (en) * | 2022-07-05 | 2023-08-01 | 广州和达水务科技股份有限公司 | Emergency drainage method and system applied to urban inland inundation |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111144563A (en) | Method for training generation countermeasure network based on dual distance loss | |
Thapa et al. | Splitfed: When federated learning meets split learning | |
Zhang et al. | Deep learning with elastic averaging SGD | |
Yang et al. | Condensenet v2: Sparse feature reactivation for deep networks | |
US20230036702A1 (en) | Federated mixture models | |
CN109165735B (en) | Method for generating sample picture based on generation of confrontation network and adaptive proportion | |
Huang et al. | Optic flow field segmentation and motion estimation using a robust genetic partitioning algorithm | |
CN112799708B (en) | Method and system for jointly updating business model | |
CN111612872B (en) | Face age change image countermeasure generation method and system | |
CN107403466A (en) | Ultra-large unstrctured grid generation method based on overall situation encryption | |
Bonettini et al. | A variable metric forward-backward method with extrapolation | |
Zhou et al. | Hybrid social spider optimization algorithm with differential mutation operator for the job-shop scheduling problem. | |
Alabdulatif et al. | Fully homomorphic based privacy-preserving distributed expectation maximization on cloud | |
CN115733569A (en) | Rapid retraining of fully fused neural transceiver components | |
CN115204416A (en) | Heterogeneous client-oriented joint learning method based on hierarchical sampling optimization | |
CN113807330B (en) | Three-dimensional sight estimation method and device for resource-constrained scene | |
KR20230073751A (en) | System and method for generating images of the same style based on layout | |
CN116227578A (en) | Unsupervised domain adaptation method for passive domain data | |
CN116629376A (en) | Federal learning aggregation method and system based on no data distillation | |
Kang et al. | Time efficient offloading optimization in automotive multi-access edge computing networks using mean-field games | |
Gauthier et al. | Personalized graph federated learning with differential privacy | |
Valle et al. | Parallel approach for ensemble learning with locally coupled neural networks | |
US11080365B2 (en) | Solving lattice problems using annealing | |
CN112766489A (en) | Method for training generation countermeasure network based on dual distance loss | |
KR20210107261A (en) | Method and apparatus for clustering data by using latent vector |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
WW01 | Invention patent application withdrawn after publication | ||
WW01 | Invention patent application withdrawn after publication |
Application publication date: 20200512 |