CN108197697A - A kind of dynamic method for resampling of trained deep neural network - Google Patents
A kind of dynamic method for resampling of trained deep neural network Download PDFInfo
- Publication number
- CN108197697A CN108197697A CN201711477740.4A CN201711477740A CN108197697A CN 108197697 A CN108197697 A CN 108197697A CN 201711477740 A CN201711477740 A CN 201711477740A CN 108197697 A CN108197697 A CN 108197697A
- Authority
- CN
- China
- Prior art keywords
- training
- training set
- neural network
- resampling
- class
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention discloses a kind of dynamic method for resampling of trained deep neural network, into Mobile state resampling during the algorithm of algorithm, dynamic generation training set including calculating sampled probability and training neural network.Using the present invention, the different classes of sample number relative equilibrium included in causing training set when training starts, neural network is enabled to extract group another characteristic, cause the category distribution of training set close to the true distribution of original training set at the end of training, so as to reduce over-fitting, and keep training accuracy rate and verify the association of accuracy rate.
Description
Technical field
The present invention relates to a kind of artificial intelligence depth learning technology field more particularly to a kind of trained deep neural networks
Dynamic method for resampling.
Background technology
Since AlexNet in 2012 wins an immediate success in ImageNet contests, artificial intelligent depth neural network is used（It refers in particular to
Convolutional neural networks）It carries out image classification and achieves rapid development, accuracy rate has reached the water for being even more than human expert
It is flat.But deep neural network carries out image classification and has a problem that, when sample set class is distributed not perfectly flat weighing apparatus, if
Training stage is not pocessed, and prediction when can be tended to comprising the class more than sample number, and ignore the few class of sample number, so as to
Even if classification accuracy is caused to seem well, but virtually completely unavailable.
There are two ways to currently solving the problems, such as this：Using the loss function comprising class distribution of weights and training
When to sample set carry out resampling.
The unbalanced problem of class, such as Kappa, F1 Score, Recall are inherently considered there are many evaluation index
Deng, but they are not suitable as loss function a bit, some can be used only in two classification or regression problem, it is impossible to be used in more
Classification.Therefore existing loss function can only be utilized by solving class imbalance（It is such as with cross entropy cross_entropy or square
Error mean_squared_error etc.）, handle class imbalance problem by increasing the weight factor of class, such as using
Keras deep learnings frame can set the class weight Class_weight of model.But this method has the disadvantage that：Class is set
Weight is not very intuitive, and is difficult the suitable value of setting, cumbersome in training process dynamic change class weight.
It is to carry out resampling to training set before training to solve the existing method of class imbalance problem using resampling,
Lack sampling is carried out to big classification（undersample）, over-sampling is carried out to small classification（oversample）, pass through resampling static state
Generate the training set of a relative equilibrium.This method has the disadvantage that：Some preciousnesses can be abandoned to big classification lack sampling first
Sample data, additionally due to the class regularity of distribution of training set is static in entire training process, it is difficult to find resampling
Equalization point.If on the one hand ignoring class imbalance, neural network can tend to the classification more than sample and ignore the few class of sample
Not, if on the other hand each classification is forced to include same sample number by resampling, over-fitting can be led to, because each
One sample of the medium and small classification of wheel training can repeat many times, and since resampling changes the data point of training set
Cloth so as to isolate trained accuracy rate and verify the contact between accuracy rate, considerably increases analyzing and training process and adjustment is instructed
Practice the difficulty of parameter.
Invention content
Technical problem to be solved of the embodiment of the present invention is, provides a kind of dynamic of trained deep neural network and adopts again
Quadrat method.The accuracy rate of deep neural network classification can be improved, prediction process is avoided to tend to big classification and ignore small classification, together
When avoid over-fitting, sample set data can be made full use of, can keep train between accuracy rate and predictablity rate
Contact.
In order to solve the above-mentioned technical problem, an embodiment of the present invention provides a kind of dynamics of trained deep neural network to adopt again
Quadrat method, including calculating the algorithm of the algorithm of sampled probability, dynamic generation training set and training neural network during into
Mobile state resampling；
The algorithm for calculating sampled probability includes the use of an array and preserves class sampled probability, by each in the array
Element corresponds to the sampled probability of each class, and defines initial value, end value and the changing rule function of the array；
The algorithm of the dynamic generation training set includes each element adopting with the element generic to original training set
Sample probability carries out stochastical sampling, one interim training set of dynamic generation；
Into Mobile state resampling by being used described in call back function calling before every wheel training during the trained neural network
The method of dynamic generation training set generates a training set, then the input which is trained as epicycle carries out epicycle
Training.
Further, the initial value of the array includes falling for the n times power of sample number for each class in training set sample
Number, the end value of the array include the inverse of the m power of sample number for each class in training set sample, and wherein n and m's takes
It is worth range (0-1), n>m.
Further, the input parameter of the changing rule function is trained round, and output is class sampled probability.Training
The output of the smaller changing rule function of round more connects closer to array initial value, the output of the bigger changing rule function of training round
The end value of nearly array.Implement the embodiment of the present invention, have the advantages that：The present invention can improve deep neural network point
The accuracy rate of class avoids prediction process from tending to big classification and ignore small classification, while avoids over-fitting, can be to sample set number
According to being made full use of, can keep training the contact between accuracy rate and predictablity rate.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, the present invention will be made below further detailed
Description.
The dynamic method for resampling of a kind of trained deep neural network of the embodiment of the present invention, which is characterized in that including meter
Into Mobile state resampling during calculating the algorithm of sampled probability, the algorithm of dynamic generation training set and training neural network
Method.
1. calculate the algorithm of dynamic class sampled probability
The algorithm for calculating sampled probability includes the use of array and preserves class sampled probability, each element in array corresponds to every
The sampled probability of one class.Define initial value, end value and the changing rule function of the array, the input ginseng of changing rule function
Number is trained round, and output is class sampled probability.The output of training round smaller changing rule function closer to array initial value,
The output of the bigger changing rule function of training round is closer to the end value of array.
It is as follows with Python puppet representations：
prob_start_i =1/ (xi ** power_start)
prob_end_i =1/ (xi ** power_end)
alpha = balance_ratio ** epochs
prob_i = prob_start_i * alpha+ prob_end_i * (1 - alpha)
Xi represents the sample number that i-th of class includes, and prob_start_i represents the initial samples probability of i-th of class, prob_end_
I represents the end sampled probability of i-th of class, and epochs represents currently trained wheel number, and power_start and power_end are
Hyper parameter, and value range (0-1], if taking 1 to cause class complete equipilibrium, if taking the minimum close to 0 so that keeping original
Class is distributed.Balance_ratio is also a hyper parameter, is represented with trained progress, the speed degree of resampling variation.This
A little hyper parameters are adjusted according to actual sample collection class imbalance degree and convergence speed, in practical training process
Using following hyper parameter, power_start=0.7, power_end=0.2, balance_ratio=0.95 can obtain very
Good effect, and the setting can be suitable for most of application scenarios.
By the algorithm packaging into function, using array and Numpy, Python code is as follows：
weight_class_start =np.array（prob_start）
weight_class_end =np.array（prob_end）
def resampling_dynamic(epoch):
alpha=balance_ratio**epoch
class_weights = weight_class_start * alpha \
+weight_class_end*(1-alpha)
returnclass_weights
2. the algorithm of dynamic generation training set
The algorithm of dynamic generation training set includes general with the sampling of the element generic to each element of original training set
Rate carries out stochastical sampling, one interim training set of dynamic generation, some sample can repeat, can also be ignored, this be faced
When input data of the training set as deep neural network this time training.
Resampling function get_balance_class is write using Python, input is training set（Image file and mark
Note）And sampled probability, output are dynamic samplings as a result, code is as follows：
def get_balance_class(files, labels,
weights
):
y=np.array(labels)
weights=np.array(weights,dtype=float)
p=np.zeros(len(y))
fori,weightinenumerate(weights):
p[y==i]=weight
#randomsampling
random_sampling = np.random.choice(np.arange(len(y)), size=len(y),
replace=True, p = (np.array(p) / p.sum()))
random_sampling_list = random_sampling.tolist()
r_files = []
r_labels = []
for i in random_sampling_list:
r_files.append(files[i])
r_labels.append(labels[i])
return r_files, r_labels
3. into Mobile state resampling during training neural network
Method for resampling dynamic generation training set is called by call back function before often wheel training, then carries out the training of epicycle.
In the present embodiment, using keras deep learning frames, it is as follows that call back function code is defined when model defines：
callbacks=[ resampling]
Resampling=keras.callbacks.LearningRateScheduler (scheduler), in the call back function
Interior progress resampling code is as follows：
def resampling (epoch):
BALANCE_WEIGHTS = resampling_dynamic(epoch)
train_files_balanced, train_labels_balanced = my_data.get_balance_class
(train_files, train_labels, weights=BALANCE_WEIGHTS)
Each round training can call the call back function and pass over current training round epoch, and call back function carries out weight
One training set of dynamic generation is sampled, is then trained.
Implement the embodiment of the present invention and train the different classes of sample number relative equilibrium included in making training set during beginning,
Neural network is enabled to extract group another characteristic, the category distribution of training set is made at the end of training close to original training set
True distribution so as to reduce over-fitting, and keeps training accuracy rate and verifies the association of accuracy rate, as trained adopt
Sample probability is from initial value to the gradual transition of end value.The present invention can improve the accuracy rate of deep neural network classification, avoid pre-
Survey process tends to big classification and ignores small classification, while avoids over-fitting, sample set data can be made full use of, energy
Enough contacts kept between training accuracy rate and predictablity rate.
Above disclosed is only a kind of preferred embodiment of the present invention, cannot limit the power of the present invention with this certainly
Sharp range, therefore equivalent variations made according to the claims of the present invention, are still within the scope of the present invention.
Claims (3)
1. a kind of dynamic method for resampling of trained deep neural network, which is characterized in that the algorithm including calculating sampled probability,
Into Mobile state resampling during the algorithm of dynamic generation training set and training neural network；
The algorithm for calculating sampled probability includes the use of an array and preserves class sampled probability, by each in the array
Element corresponds to the sampled probability of each class, and defines initial value, end value and the changing rule function of the array；
The algorithm of the dynamic generation training set includes each element adopting with the element generic to original training set
Sample probability carries out stochastical sampling, one interim training set of dynamic generation；
Into Mobile state resampling by being used described in call back function calling before every wheel training during the trained neural network
The method of dynamic generation training set generates a training set, then the input which is trained as epicycle carries out epicycle
Training.
2. the dynamic method for resampling of trained deep neural network according to claim 1, which is characterized in that the array
Initial value included for each class in training set sample sample number n times power inverse, the end value of the array is training
Collect the value range (0-1) of the inverse for the m power that each class in sample includes sample number, wherein n and m, n>m.
3. the dynamic method for resampling of trained deep neural network according to claim 1 or 2, which is characterized in that described
The input parameter of changing rule function is trained round, and output is class sampled probability.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711477740.4A CN108197697A (en) | 2017-12-29 | 2017-12-29 | A kind of dynamic method for resampling of trained deep neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711477740.4A CN108197697A (en) | 2017-12-29 | 2017-12-29 | A kind of dynamic method for resampling of trained deep neural network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108197697A true CN108197697A (en) | 2018-06-22 |
Family
ID=62586514
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711477740.4A Pending CN108197697A (en) | 2017-12-29 | 2017-12-29 | A kind of dynamic method for resampling of trained deep neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108197697A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109598349A (en) * | 2018-11-23 | 2019-04-09 | 华南理工大学 | Overhead transmission line fault detection data sample batch processing training method based on classification stochastical sampling |
| CN109782231A (en) * | 2019-01-17 | 2019-05-21 | 北京大学 | A kind of end-to-end sound localization method and system based on multi-task learning |
| CN110688888A (en) * | 2019-08-02 | 2020-01-14 | 浙江省北大信息技术高等研究院 | Pedestrian attribute identification method and system based on deep learning |
| CN114399669A (en) * | 2022-03-25 | 2022-04-26 | 江苏智云天工科技有限公司 | Target detection method and device |
| CN115512391A (en) * | 2022-09-29 | 2022-12-23 | 珠海视熙科技有限公司 | Target detection model training method, device and equipment for data adaptive resampling |
-
2017
- 2017-12-29 CN CN201711477740.4A patent/CN108197697A/en active Pending
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109598349A (en) * | 2018-11-23 | 2019-04-09 | 华南理工大学 | Overhead transmission line fault detection data sample batch processing training method based on classification stochastical sampling |
| CN109782231A (en) * | 2019-01-17 | 2019-05-21 | 北京大学 | A kind of end-to-end sound localization method and system based on multi-task learning |
| CN109782231B (en) * | 2019-01-17 | 2020-11-20 | 北京大学 | End-to-end sound source positioning method and system based on multi-task learning |
| CN110688888A (en) * | 2019-08-02 | 2020-01-14 | 浙江省北大信息技术高等研究院 | Pedestrian attribute identification method and system based on deep learning |
| CN110688888B (en) * | 2019-08-02 | 2022-08-05 | 杭州未名信科科技有限公司 | Pedestrian attribute identification method and system based on deep learning |
| CN114399669A (en) * | 2022-03-25 | 2022-04-26 | 江苏智云天工科技有限公司 | Target detection method and device |
| CN115512391A (en) * | 2022-09-29 | 2022-12-23 | 珠海视熙科技有限公司 | Target detection model training method, device and equipment for data adaptive resampling |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108197697A (en) | A kind of dynamic method for resampling of trained deep neural network | |
| US11030523B2 (en) | Neural architecture search | |
| Rasmussen et al. | Occam's razor | |
| CN105046277B (en) | Robust mechanism study method of the feature significance in image quality evaluation | |
| CN106777402B (en) | A kind of image retrieval text method based on sparse neural network | |
| GB2588523A (en) | Epilepsy seizure detection and prediction using techniques such as deep learning methods | |
| CN102075352A (en) | Method and device for predicting network user behavior | |
| CN108566627A (en) | A kind of method and system identifying fraud text message using deep learning | |
| CN111078866B (en) | Chinese text abstract generation method based on sequence-to-sequence model | |
| CN110300075B (en) | Wireless channel estimation method | |
| CN107220180A (en) | A kind of code classification method based on neutral net language model | |
| CN111242287A (en) | Neural network compression method based on channel L1 norm pruning | |
| CN108573399A (en) | Method and its system are recommended by trade company based on transition probability network | |
| CN110147745A (en) | A kind of key frame of video detection method and device | |
| CN111680109A (en) | Knowledge graph representation learning model training method and device and electronic equipment | |
| CN104573000A (en) | Sequential learning based automatic questions and answers device and method | |
| CN112153616A (en) | Power control method in millimeter wave communication system based on deep learning | |
| CN109583561A (en) | A kind of the activation amount quantization method and device of deep neural network | |
| CN110362683A (en) | A kind of information steganography method based on recurrent neural network, device and storage medium | |
| CN107087160A (en) | A kind of Forecasting Methodology of the user experience quality based on BP Adaboost neutral nets | |
| CN110472255A (en) | Neural network machine interpretation method, model, electric terminal and storage medium | |
| CN109033169A (en) | Mobile traffic classification method based on multistage weight conversion and convolutional neural networks | |
| CN101447995B (en) | Method for identifying P2P data stream, device and system thereof | |
| CN108280207A (en) | A method of the perfect Hash of construction | |
| CN112564107A (en) | Transient stability assessment method for power system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180622 |