CN111144563A - Method for training generation countermeasure network based on dual distance loss - Google Patents

Method for training generation countermeasure network based on dual distance loss Download PDF

Info

Publication number
CN111144563A
CN111144563A CN201911345589.8A CN201911345589A CN111144563A CN 111144563 A CN111144563 A CN 111144563A CN 201911345589 A CN201911345589 A CN 201911345589A CN 111144563 A CN111144563 A CN 111144563A
Authority
CN
China
Prior art keywords
training
dual distance
distance loss
generator
distribution
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
CN201911345589.8A
Other languages
Chinese (zh)
Inventor
宋艳枝
彭程
王昊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hefei Liman Information Technology Co ltd
Original Assignee
Hefei Liman Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hefei Liman Information Technology Co ltd filed Critical Hefei Liman Information Technology Co ltd
Priority to CN201911345589.8A priority Critical patent/CN111144563A/en
Publication of CN111144563A publication Critical patent/CN111144563A/en
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a method for training a generation confrontation network based on dual distance loss, and relates to the technical field of deep learning neural networks. The invention comprises the following steps: step S1: acquiring a data set of target distribution, and preprocessing the data set; step S2: setting the structures and parameters of a generator and a discriminator neural network and the learning rate in the training process; step S3: and (3) calculating a dual distance loss function according to parameters of the neural network, and training a generator to generate real distribution by adopting a random gradient descent method based on the dual distance loss function. According to the invention, more accurate results are obtained under the condition of the same iteration steps, the training quality can be improved, and the cost benefit is better.

Description

Method for training generation countermeasure network based on dual distance loss
Technical Field
The invention belongs to the technical field of deep learning neural networks, and particularly relates to a method for training a generation confrontation network based on dual distance loss.
Background
The generation of the confrontation network is a kind of neural network, and the discriminators and the generators are trained in turn to confront each other to sample from complex probability distribution, such as generating pictures, characters, voice and the like.
If the original generators and discriminators are random, it is difficult to determine whether the generators and discriminators can converge to an ideal conclusion through training of given data. While it can be shown that under some strong assumptions, generators and discriminators can converge to local nash equilibrium, many generation-confrontation network algorithms do not converge globally.
Disclosure of Invention
The invention aims to provide a method for training a generation countermeasure network based on dual distance loss, which comprises the steps of preprocessing a data set obtained by target distribution, setting structures, parameters and learning rate in a training process of a generator and a discriminator network, calculating a dual distance loss function according to parameters of an applied neural network, training the generator to generate real distribution based on the random gradient descent method adopted by the dual distance loss function, and solving the problems that the existing generation network countermeasure algorithm can not achieve global convergence and the training result is not accurate enough.
In order to solve the technical problems, the invention is realized by the following technical scheme:
the invention relates to a method for training a generation confrontation network based on dual distance loss, which comprises the following steps:
step S1: acquiring a data set of target distribution, and preprocessing the data set;
step S2: setting the structures and parameters of a generator and a discriminator neural network and the learning rate in the training process;
step S3: and (3) calculating a dual distance loss function according to parameters of the neural network, and training a generator to generate real distribution by adopting a random gradient descent method based on the dual distance loss function. Preferably, after the step S3, an empirical dual distance between the target real distribution and the generated distribution needs to be calculated:
Figure BDA0002333221190000021
in the formula (I), the compound is shown in the specification,
Figure BDA0002333221190000022
wherein x isiIs a sample point in the true distribution, ziIs a sample point in the Gaussian distribution, m and n are positive integers, f is a discriminator, g is a generator,
Figure BDA0002333221190000023
and
Figure BDA0002333221190000024
respectively, the space where the arbiter and the generator are located.
Preferably, the method for generating the confrontation network training calculates perturbation points, and then determines a dual distance loss function and an optimization direction by using the perturbation points, including the following steps:
an initialization step: the target data set is processed. Given the arbiter f in the initial state0Sum generator g0Setting parameters gamma epsilon (0, 2) and k as 0, and giving two positive integers m and n;
random selection of data points: select m points in the target data set, and record as { x1，...，xmSelecting n points in a specified Gaussian noise, and recording the points as z1，...，zn}；
And a step of calculating the shooting point: for a given generator gkSum discriminator fkConsider that
Figure BDA0002333221190000031
Calculating perturbation points
Figure BDA0002333221190000032
And
Figure BDA0002333221190000033
and (3) calculating an optimization direction: consideration function
Figure BDA0002333221190000034
The sub-gradient of (A) is recorded as
Figure BDA0002333221190000035
And
Figure BDA0002333221190000036
then, considering the optimization direction, respectively
Figure BDA0002333221190000037
An updating step: computing
Figure BDA0002333221190000038
And
Figure BDA0002333221190000039
and τk＝γEk/||dk||2。。
Preferably, the generative confrontation network training method satisfies at least with a probability of 1-3 δ:
Figure BDA00023332211900000310
the invention has the following beneficial effects:
the invention carries out preprocessing by acquiring a data set of target distribution, sets the structures, parameters and learning rate in the training process of a generator and a discriminator network, calculates a dual distance loss function according to the parameters of an applied neural network, trains the generator to generate real distribution by adopting a random gradient descent method based on the dual distance loss function, obtains more accurate results under the condition of the same iteration step number, can improve the training quality and has better cost benefit.
Of course, it is not necessary for any product in which the invention is practiced to achieve all of the above-described advantages at the same time.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart illustrating a method for generating a confrontation network training according to the present invention.
Fig. 2 is a comparison graph of the generated countermeasure network training method provided by the embodiment of the present invention with the generated result of the WGAN-GP method after 20000 iterations on the CIFAR10 data set.
Fig. 3 is a comparison graph of the generated confrontation network training method provided by the embodiment of the invention on a CIFAR10 data set and the inclusion Score obtained by the WGAN-GP method.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1-3, the present invention is a method for training a generation countermeasure network based on dual distance loss, comprising:
step S1: acquiring a data set of target distribution, and preprocessing the data set;
step S2: setting the structures and parameters of a generator and a discriminator neural network and the learning rate in the training process;
step S3: calculating a dual distance loss function according to parameters of the neural network, and training a generator to generate real distribution by adopting a random gradient descent method based on the dual distance loss function;
the method for generating the confrontation training based on the dual distance loss function can obtain more accurate results under the condition of the same iteration steps, can improve the training quality, and has better cost benefit and generalization performance.
After step S3, it is necessary to calculate the empirical dual distance between the target real distribution and the generated distribution:
Figure BDA0002333221190000051
in the formula (I), the compound is shown in the specification,
Figure BDA0002333221190000052
wherein the content of the first and second substances,xi is the sample point in the true distribution, ziIs a sample point in the Gaussian distribution, m and n are positive integers, f is a discriminator, g is a generator,
Figure BDA0002333221190000053
and
Figure BDA0002333221190000054
respectively, the space where the arbiter and the generator are located.
Specifically, 2 fields are given
Figure BDA0002333221190000055
And
Figure BDA0002333221190000056
arbiter and generator satisfy
Figure BDA0002333221190000057
A convex function phi, a true data distribution pdataAnd a Gaussian distribution pz，(f*，g*) Upper dual distance loss DG (f)*，g*) Is composed of
Figure BDA0002333221190000058
Here, the loss function for generating the countermeasure network is
Figure BDA0002333221190000059
Empirical dual distance loss
Figure BDA00023332211900000510
Satisfy the requirement of
Figure BDA00023332211900000511
Wherein the empirical loss function that generates the countermeasure network is:
Figure BDA00023332211900000512
if the true sample X and the Gaussian distribution sample Z are bounded and the boundary is bounded by BxAnd BzRepresents;
Figure BDA00023332211900000513
so that
Figure BDA00023332211900000514
And
Figure BDA00023332211900000515
is provided with
Figure BDA00023332211900000516
Wherein L isfIs the Lipschitz constant, L, of the arbiter network fgA liphoz constant for the generator network; then with a probability of at least 1-3 δ there is equation (1):
Figure BDA0002333221190000061
at this time, a specific process of obtaining formula (1) is given, which may include:
the equation of formula (1) is simplified to the left as:
Figure BDA0002333221190000062
the McDiarmid inequality condition is:
Figure BDA0002333221190000063
wherein X ═ { X ═ X1，x2，...，xi，...，xn}，X′＝{x1，x2，...，x′i，...，xn}，ρφIs the Liphoz constant of φ.
Using the McDiarmid inequality, at least
Figure BDA0002333221190000064
The probability of (d) is given by equation (2):
Figure BDA0002333221190000065
again using the McDiarmid inequality, at least
Figure BDA0002333221190000071
The probability of (c) is given by equation (3):
Figure BDA0002333221190000072
where e is ∈1，∈2，...，∈n) And P (∈ C)i＝1)＝P(∈i-1) 0.5. Therefore, the probability of at least 1- δ is given by equation (4):
Figure BDA0002333221190000073
similarly, the probability of at least 1- δ has formula (5) and formula (6):
Figure BDA0002333221190000074
Figure BDA0002333221190000075
thus, the probability of at least 1-3 δ has equation (7):
Figure BDA0002333221190000081
since both the discriminator f and the generator g are neural networks, they can be written in the form of equations (8) and (9):
f＝aH(MH(aH-1(MH-1(...a1(M1(·))...)))) (8)；
g＝bH′(NH′(bH′-1(NH′-1)...b1(N1(·))...)))) (9)；
wherein, aiAnd biFor activating functions, MiAnd NiFor the matrix, a is the activation function Relu in the experimentiAnd biHas a lipschitz constant of less than 1; and assume | Mi||≤BiAnd Ni||≤B′i；dfAnd dgThe width of the neural network of the arbiter and generator.
According to the above assumptions, there is formula (10):
Figure BDA0002333221190000082
order to
Figure BDA0002333221190000083
Its coverage number
Figure BDA0002333221190000084
Satisfies formula (11):
Figure BDA0002333221190000085
due to the fact that
Figure BDA0002333221190000086
Therefore, the formula (12):
Figure BDA0002333221190000091
according to the relationship between the Ladamard Mach complexity and the coverage number, obtaining a formula (13):
Figure BDA0002333221190000092
similarly, formula (14) and formula (15) are obtained:
Figure BDA0002333221190000093
Figure BDA0002333221190000094
and (3) assuming that m > is greater than n, and combining the formula (7), the formula (13), the formula (14) and the formula (15), obtaining a generalization error bound based on the dual loss distance, namely the formula (1).
Specifically, in the method for training a generative confrontation based on dual distance loss according to the embodiment of the present invention, after setting the structure and parameters of the neural network and providing a data set conforming to the target distribution, solving the dual distance loss function by using a gradient descent method may include:
an initialization step: processing the target data set; given the arbiter f in the initial state0Sum generator g0Setting parameters gamma epsilon (0, 2) and k as 0, and giving two positive integers m and n;
random selection of data points: select m points in the target data set, and record as { x1，...，xmSelecting n points in a specified Gaussian noise, and recording the points as z1，...，zn}；
Calculating the shooting point: for a given generator gkSum discriminator fkConsider that
Figure BDA0002333221190000101
Calculating perturbation points
Figure BDA0002333221190000102
And
Figure BDA0002333221190000103
dual distance calculation: calculating a dual distance loss function
Figure BDA0002333221190000104
If E iskIf 0, then the algorithm stops and the generator g is outputk；
And (3) calculating an optimization direction: consideration function
Figure BDA0002333221190000105
The sub-gradient of (A) is recorded as
Figure BDA0002333221190000106
And
Figure BDA0002333221190000107
then, considering the optimization direction, respectively
Figure BDA0002333221190000108
An updating step: computing
Figure BDA0002333221190000109
And
Figure BDA00023332211900001010
and τk＝γEk/||dk||2。
The method for generating the confrontation training based on the dual distance loss function provided by the embodiment of the invention successfully realizes data distribution generation on MNIST and CIFAR 10; the result shows that, compared with the traditional method for generating the confrontation training, the method for generating the confrontation training based on the dual distance loss function provided by the embodiment of the invention can obtain a more accurate result under the condition of the same iteration step number, can improve the training quality and has better cost benefit.
Since each training requires generation of gaussian noise during the training process, the method for generating a confrontation training based on a dual distance loss function provided by this embodiment seems to be more complicated in terms of setting of the loss function, but does not increase a generalization error, and thus has the same generalization performance as the conventional method for generating a confrontation training.
It should be noted that, in the above system embodiment, each included unit is only divided according to functional logic, but is not limited to the above division as long as the corresponding function can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.

Claims (4)

1. A method for generating a confrontation network training based on dual distance loss is characterized by comprising the following steps:
step S1: acquiring a data set of target distribution, and preprocessing the data set;
step S2: setting the structures and parameters of a generator and a discriminator neural network and the learning rate in the training process;
step S3: and (3) calculating a dual distance loss function according to parameters of the neural network, and training a generator to generate real distribution by adopting a random gradient descent method based on the dual distance loss function.
2. The method for training a generative confrontation network based on dual distance loss as claimed in claim 1, wherein after step S3, an empirical dual distance between the target real distribution and the generative distribution is calculated:
Figure FDA0002333221180000011
in the formula (I), the compound is shown in the specification,
Figure FDA0002333221180000012
wherein x isiIs a sample point in the true distribution, ziIs a sample point in a Gaussian distribution, and m and n arePositive integer, f is a discriminator, g is a generator,
Figure FDA0002333221180000013
and
Figure FDA0002333221180000014
respectively, the space where the arbiter and the generator are located.
3. The method for generating confrontation network training based on dual distance loss as claimed in claim 1, wherein the method for generating confrontation network training calculates perturbation points, and then determines dual distance loss function and optimization direction by using the perturbation points, comprising the following steps:
an initialization step: the target data set is processed. Given the arbiter f in the initial state0Sum generator g0Setting parameters gamma epsilon (0, 2) and k as 0, and giving two positive integers m and n;
random selection of data points: select m points in the target data set, and record as { x1，...，xmSelecting n points in a specified Gaussian noise, and recording the points as z1，...，zn}；
And a step of calculating the shooting point: for a given generator gkSum discriminator fkConsider that
Figure FDA0002333221180000021
Calculating perturbation points
Figure FDA0002333221180000022
And
Figure FDA0002333221180000023
and (3) calculating an optimization direction: consideration function
Figure FDA0002333221180000024
The sub-gradient of (A) is recorded as
Figure FDA0002333221180000025
And
Figure FDA0002333221180000026
then, considering the optimization direction, respectively
Figure FDA0002333221180000027
An updating step: computing
Figure FDA0002333221180000028
And
Figure FDA0002333221180000029
and τk＝γEk/||dk||2。
4. The method for training a generative warfare network based on dual distance loss as claimed in claim 1, wherein the method satisfies at least with a probability of 1-3 δ:
Figure FDA00023332211800000210
CN201911345589.8A 2019-12-24 2019-12-24 Method for training generation countermeasure network based on dual distance loss Withdrawn CN111144563A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911345589.8A CN111144563A (en) 2019-12-24 2019-12-24 Method for training generation countermeasure network based on dual distance loss

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911345589.8A CN111144563A (en) 2019-12-24 2019-12-24 Method for training generation countermeasure network based on dual distance loss

Publications (1)

Publication Number Publication Date
CN111144563A true CN111144563A (en) 2020-05-12

Family

ID=70519634

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911345589.8A Withdrawn CN111144563A (en) 2019-12-24 2019-12-24 Method for training generation countermeasure network based on dual distance loss

Country Status (1)

Country Link
CN (1) CN111144563A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112488309A (en) * 2020-12-21 2021-03-12 清华大学深圳国际研究生院 Training method and system of deep neural network based on critical damping momentum
CN112668239A (en) * 2020-12-30 2021-04-16 山东交通学院 Hybrid power truck fleet experience teaching method based on counterstudy
CN112766489A (en) * 2021-01-12 2021-05-07 合肥黎曼信息科技有限公司 Method for training generation countermeasure network based on dual distance loss
CN115205738A (en) * 2022-07-05 2022-10-18 广州和达水务科技股份有限公司 Emergency drainage method and system applied to urban inland inundation

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112488309A (en) * 2020-12-21 2021-03-12 清华大学深圳国际研究生院 Training method and system of deep neural network based on critical damping momentum
CN112488309B (en) * 2020-12-21 2023-10-20 清华大学深圳国际研究生院 Training method and system of deep neural network based on critical damping momentum
CN112668239A (en) * 2020-12-30 2021-04-16 山东交通学院 Hybrid power truck fleet experience teaching method based on counterstudy
CN112668239B (en) * 2020-12-30 2022-11-15 山东交通学院 Hybrid power truck fleet experience teaching method based on counterstudy
CN112766489A (en) * 2021-01-12 2021-05-07 合肥黎曼信息科技有限公司 Method for training generation countermeasure network based on dual distance loss
CN115205738A (en) * 2022-07-05 2022-10-18 广州和达水务科技股份有限公司 Emergency drainage method and system applied to urban inland inundation
CN115205738B (en) * 2022-07-05 2023-08-01 广州和达水务科技股份有限公司 Emergency drainage method and system applied to urban inland inundation

Similar Documents

Publication Publication Date Title
CN111144563A (en) Method for training generation countermeasure network based on dual distance loss
Thapa et al. Splitfed: When federated learning meets split learning
Zhang et al. Deep learning with elastic averaging SGD
CN107330516B (en) Model parameter training method, device and system
US20230036702A1 (en) Federated mixture models
Yang et al. Condensenet v2: Sparse feature reactivation for deep networks
Xie et al. BAYHENN: Combining Bayesian deep learning and homomorphic encryption for secure DNN inference
CN109165735B (en) Method for generating sample picture based on generation of confrontation network and adaptive proportion
CN111737743A (en) Deep learning differential privacy protection method
Bonettini et al. A variable metric forward-backward method with extrapolation
CN107403466A (en) Ultra-large unstrctured grid generation method based on overall situation encryption
Tang et al. Skfac: Training neural networks with faster kronecker-factored approximate curvature
Zhou et al. Hybrid social spider optimization algorithm with differential mutation operator for the job-shop scheduling problem.
Alabdulatif et al. Fully homomorphic based privacy-preserving distributed expectation maximization on cloud
CN115204416A (en) Heterogeneous client-oriented joint learning method based on hierarchical sampling optimization
Kang et al. Time efficient offloading optimization in automotive multi-access edge computing networks using mean-field games
Gauthier et al. Personalized graph federated learning with differential privacy
CN113807330B (en) Three-dimensional sight estimation method and device for resource-constrained scene
US11080365B2 (en) Solving lattice problems using annealing
CN116629376A (en) Federal learning aggregation method and system based on no data distillation
Nakano Hybrid algorithm of ensemble transform and importance sampling for assimilation of non-Gaussian observations
Valle et al. Parallel approach for ensemble learning with locally coupled neural networks
CN112766489A (en) Method for training generation countermeasure network based on dual distance loss
Schaurecker et al. Super-resolving Dark Matter Halos using Generative Deep Learning
CN116227578A (en) Unsupervised domain adaptation method for passive domain data

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WW01 Invention patent application withdrawn after publication
WW01 Invention patent application withdrawn after publication

Application publication date: 20200512