CN108921281A - A kind of field adaptation method based on depth network and countermeasure techniques - Google Patents

Abstract

A kind of field adaptation method based on depth network and countermeasure techniques of the present invention is related to deep learning, transfer learning, field adaptation, convolutional neural networks, the technologies such as confrontation network.We increase by two confrontation subnets on the basis of finely tuning Alexnet, and for the difference between the amendment different field sample of confrontation type, arriving in high-level layers of study can sharing feature.The cost of handmarking in big data environment can be effectively reduced in the method, there is certain practical significance.The algorithm proposes innovation on the basis of new target risk upper error, and algorithm mainly includes initial phase and network training stage.In initial phase, according to new upper error and new neuronal layers are constructed, while increasing corresponding loss and regularization term, and initial work is carried out to network and data set.Training stage replaces original hyper parameter by probability threshold value, runs several iteration cycles according to the SGD algorithm of probability iteration, until meeting condition, training terminates.Final trained network can effectively substitute handmarking's process, obtain more, more accurate marked sample.

Description

A kind of field adaptation method based on depth network and countermeasure techniques

Technical field

A kind of field adaptation algorithm based on depth network and countermeasure techniques of the present invention, algorithm are related to including that convolution is refreshing Through network, deep learning, machine learning optimization, belong to artificial intelligence field.In particular to the new error based on the derivation of equation Previous increase multiple-branching construction and loss function finely tune the good AlexNet of pre-training after fighting in subnet, are related to out one The new combination of kind fights network, field adaptation (transfer learning) task in artificial intelligence can be effectively completed, in different necks Expressing for knowledge migration is carried out between domain.

Background technique

Data set offset is can not be ignored the problem of in machine learning field.Data set is description real world object The one-sided of body is stated, and the mutually isostructural model of training on the data set for describing the same collection of objects, generalization ability is often There are deviation, effect is not ideal enough.Data set offset reduces generalization ability of the model on same type objects.For true For the world, data set offset can be understood as model and overfitting problem have occurred on data set.Domain adaptation is attempted to solve number According to collection offset problem, model is improved in target domain data based on similitude, the otherness between source domain and target domain Performance.The development of deep learning is so that profound learning model also implies that needs are big with the parameter for more needing to learn The sample training model of amount.On the other hand, searching out quantity, enough to have the training of exemplar bolster model be extremely difficult 's.

Summary of the invention

To solve the above-mentioned problems, a kind of field adaptation method based on depth network and countermeasure techniques of the present invention, not only The offset problem between FIELD Data can be effectively corrected, and may learn the representation space of neck inter-domain sharing, algorithm It can be adapted for unsupervised field adaptation task, i.e. target domain needs not exist for any label and can run.The present invention Field adaptation method can save a large amount of manpower markers work, for solving the problems, such as under big data that flag data is rare There is wide applicability.

A kind of field adaptation method based on depth network and countermeasure techniques of the present invention, mainly includes the following steps that：

Step 1:Obtain the sample of source domain and the sample of label and target domain.Source domain and target domain difference Be derived from two approximate fields, wherein the sample of source domain has a label, and the sample of target domain does not have a label, two fields it Between to have certain similitude, i.e., the knowledge that can be multiplexed can be acquired between field.

Step 2:Adaptation network is constructed, it is random initial using the good parameter initialization AlexNet of ImageNet pre-training Change network others parameter.The purpose of algorithm is that a proper subspace is established between field, the feature in this proper subspace It can be shared by two fields.AlexNet is a kind of depth convolutional network, we construct this neck using depth network Domain knowledge is indicated, while being lost by increasing confrontation come the difference degree between amendment field.The communication process of loss passes through anti- It is realized to gradient descent method.

Step 3：According to task, network correlation hyper parameter, such as learning rate, probability threshold value, dropout rate etc. are initialized.Net Network hyper parameter needs manually determine that the convergence and performance to algorithm have important influence.In general, the learning rate of algorithm can To give the value of a very little, make network convergence using more frequently batch gradient decline, probability threshold value is poor according to specific field Off course degree determines, dropout rate then over-fitting in order to prevent.

Step 4：Using the alternate gradient back-propagation algorithm such as SGD training network parameter of probability, training terminates to obtain most Final cast and result.The alternate gradient back-propagation algorithm of probability can be by hyper parameter " probability threshold value " it controls in very little In range, so that hyper parameter is easier to search for.Training generallys use several periods, each period, we were traversed using SGD Primary all data samples.After a few cycles, depth network training is completed, and algorithm terminates.

The present invention has the advantages that and effect：

1) it is proposed that new aiming field upper error establishes new network structure for domain based on new upper error Adaptation, model are made of two son confrontation networks, have stronger confrontation dynamics and public characteristic ability to express.

2) compared to the balance that hyper parameter is used to control loss, we have proposed a kind of more flexible replacement methods, greatly Amplitude reduce hyper parameter search and control it is difficult, it is easier to the different degrees of distributional difference of adaptation.

3) our algorithm obtains performance best at present on the domain adaptation data collection of some classics.

Detailed description of the invention

Fig. 1 is a kind of schematic network structure of the field adaptation method based on depth network and countermeasure techniques of the present invention.

Fig. 2 is a kind of algorithm flow schematic diagram of the field adaptation method based on depth network and countermeasure techniques of the present invention.

Specific embodiment

A kind of realization of the field adaptation method based on depth network and countermeasure techniques of the present invention is by initial phase, instruction Practice stage and service stage composition.Initial phase includes data initialization and model initialization, and the training stage includes several times Gradient back-propagation process, according to the size of data set, the number of iterations is differed from several hundred times to tens of thousands of times, until the number of iterations Meet a certain condition or network convergence, then training is completed.

Initial phase：

Step 1, data initialization.The input of network is a tensor, usually has the color image of RGB triple channel, Firstly, for all pictures, we pass through grayscale image the tensor of its size scaling to 227 × 227 × 3 simple Image is repeated 3 times, cromogram is become.Secondly, R, G, B three-dimensional need to subtract the mean value of data set first, then divided by standard Difference, so that data meet standard and are just distributed very much, this operation is also referred to as z-score standardization.Such data prediction makes Network parameter is easier to restrain when must training.

Step 2, model initialization：

Model initialization is first loaded into the good parameter of the i.e. related pre-training of AlexNet, secondly, increasing some layers and correspondence Loss, regularization term, structure as shown in Figure 1 initializes, just every layer parameter except the original layer of AlexNet It is 0 that beginning, which turns to and obeys mean value, and the truncation that standard deviation is 1 just too distribution random numbers are set to when initializing for offset parameter 0。

Training stage：

Step 1, the training of network：

Training method is based on following formula：Enabling H is the hypothesis collection space that VC-dimension is d, and m have label sample Originally from D_SMiddle sampling obtains, m' unlabeled exemplars respectively fromWithMiddle sampling obtains, for all η ∈ H at least with Probability 1- δ inequality

We try to optimize following target：

Wherein hyper parameter λ > 0, β > 0 controls the influence that two confrontation subnets accordingly lose item.

We need learning parameter θ={ θ_c,θ_l,θ_m}.We disassemble out 3 loss functions that two are fought subnet Come, and replaces optimization process with following training objective in an iteration：

Wherein p₁,p₂∈ [0,1] is probability threshold value, for controlling the training of each loss function in iteration, in net When network starts to train, probability threshold value is provided, each time before iteration, one 0 to 1 random number is generated, if random number is arrived 0 p₁Between, then select L_MAs the loss function of current iteration, similarly, if random number is in p₁And p₂Between, selectionMake For the target of current iteration, if random number is in p₂To between 1, L is selected_CAs loss function.

Step 2, the termination of network：

The termination when reaching any one following condition of the training process of network：

1) setting value maxIter indicates that maximum number of iterations is trained if training process reaches maximum number of iterations It terminates.

2) before network reaches maxIter the number of iterations, given threshold lossChangeThreshold, if continuously The loss absolute value of the difference of network is less than lossChangeThreshold twice, then it is assumed that network parameter has been restrained, training It terminates.

According to above-mentioned initialization and training operation, network will converge to the sufficiently low point of loss, in this point Solution be exactly the final solution of model, finally, training complete network can be used for predicting the unmarked sample of target domain, generation For manually with higher accuracy rate label unknown data.

Claims (7)

1. a kind of field adaptation method based on depth network and countermeasure techniques of the present invention, in the use pair of depth convolutional neural networks Damage-retardation is become estranged distributional difference between parallel organization diminution field.

2. the algorithm mainly includes three phases：Network structure construction, confrontation loss function construction, parallelization backpropagation.

3. network have it is multiple input and multiple outputs, synonym training in complete one have monitor task and one it is unsupervised Task.

4. hyper parameter needs best to achieve the effect that by fine adjustment in input phase.

5. SGD algorithm is had modified in the training process of network, can be in an iteration, use is kind with random number The multiple loss functions of probability alternating gradient descent algorithm training of son, the current best loss of selection each time carry out the instruction of network Practice.

6. network can jump back and forth between different loss functions according to the selection of probability value.

7. network is end-to-end depth adaptation network, parallelization confrontation loss migrates knowledge between field.