KR20220049709A - System and Method of Adaptive Bach Selection for Accelerating Deep Neural Network Learning based on Data Uncertainty - Google Patents

Abstract

Disclosed are a system and method for an adaptive batch selection based on data uncertainty for accelerating a deep neural network learning. The system comprises: a data storage module; an adaptive batch selection module; a network learning module; a model prediction history module; a sample uncertainty evaluation module; a quantization module; and a sample selection probability calculation module.

Description

System and Method of Adaptive Bach Selection for Accelerating Deep Neural Network Learning based on Data Uncertainty

With the recent availability of large-scale data, it has become possible to build a high-performance deep neural network model, and it is showing very good performance in fields closely related to real life, such as autonomous driving. However, the training time of the model, which increases linearly with the increase in data size, is becoming an important challenge to put a high-performance model into practice.

In an attempt to reduce the model training time, transfer learning using a pre-trained model and model compression method to reduce the size of the model have been proposed, but they degrade the final generalization performance of the trained model. a problem occurred.

Accordingly, as another method for accelerating the training time of a deep neural network or improving the generalization performance of a model, Online Batch Selection, which uses only difficult samples included in the training data, for training, and Active Bias etc. have been proposed.

Online batch selection is a more efficient way to learn machine learning methodologies. Based on the fact that most of the training data are easy samples, learning using the loss of training samples It is a batch selection method that evaluates the difficulty of learning and increases the probability that difficult samples are selected as mini-batch samples for the next learning based on the assessed difficulty for learning. However, because only difficult samples were emphasized, the model overfitting occurred on such training samples, which deteriorated the generalization performance of the model and ultimately caused the model to make erroneous predictions.

Active bias is a method to improve the final generalization performance of a model obtained by deep neural network learning, and the prediction accuracy of the selected mini-batch samples is used to evaluate the prediction uncertainty of the training sample. The variance of the prediction accuracy values obtained during the “whole” learning period was assumed as uncertainty, and the generalization performance of the finally trained model was greatly improved by selecting the sample with the largest variance as the mini-batch sample for the next training. . However, the “growing window”-based method that uses time-series data for all periods in evaluating uncertainty leads to erroneous estimation of learning uncertainty due to outdated values when learning continues for a long time. , the learning speed of the model was rather slowed by selecting too easy or too difficult samples, which are not very helpful for learning.

The present disclosure is to provide an effective batch selection method and system for deep neural network learning.

The present disclosure improves the learning speed and final generalization performance of the trained model by configuring a batch by searching for the most effective training samples for model learning, unlike the method in which all existing training samples are considered with the same importance. It relates to a data uncertainty-based adaptive batch selection system and method for increasing .

As one of the previously proposed adaptive batch selection methods for deep neural network learning, the present disclosure solves the “incorrect sample uncertainty evaluation problem” that emerged as a problem in the previously proposed method for improving generalization performance of deep neural networks. Data for accelerating learning of new deep neural networks, which can train models by using large-scale data more efficiently and effectively by achieving “acceleration of deep learning learning” and finally achieving “high generalization to the test data of the model” To provide an uncertainty-based adaptive placement selection system and method.

A system according to an embodiment includes a data storage module for storing entire learning data; an adaptive batch selection module for adaptively selecting a learning mini-batch, which is a minimum unit for learning, from the data storage module; a network learning module that performs deep neural network learning on the received uncertainty-based mini-batch; a model prediction recording module for storing and managing only recent prediction results for mini-batch samples for the trained model; a sample uncertainty evaluation module for calculating sample uncertainty based on the results accumulated in the prediction recording module; a quantization module that converts the obtained uncertainty value into a quantitative exponent; and a sample selection probability calculation module for determining the sample selection probability by using the quantized index.

1 is a block diagram of a data uncertainty-based adaptive batch selection system for accelerating deep neural network learning.

The online batch selection method, which is known to accelerate deep learning learning, accelerates learning, but has a problem in that it reduces the generalization performance of the model for test samples used in real applications due to overfitting to difficult samples. On the other hand, the active bias method, which is known to improve the generalization performance of deep learning learning, improved the final generalization performance of the model on the test data, but it is a fundamental limitation of the Growing Window technique, “It depends on old values rather than recent ones”. This causes a problem that slows down the learning rate.

The present disclosure provides an adaptive batch selection system and method for learning a new deep neural network that simultaneously achieves “fast learning” and “improving generalization performance on test samples”.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present disclosure will be described in detail so that those of ordinary skill in the art to which the present disclosure pertains can easily implement them. However, the present disclosure may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In the description, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

In the description, “transmitting or providing” may include not only direct transmission or provision, but also transmission or provision indirectly through another device or using a detour path.

In the description, an expression written in the singular may be construed in the singular or plural unless an explicit expression such as “a” or “a” is used.

In the description, the order of operations described in the flowchart may be changed, several operations may be merged, some operations may be divided, and specific operations may not be performed.

In the description, terms such as “… unit”, “… group”, “… module” mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. .

1 is a block diagram of a data uncertainty-based adaptive batch selection system for accelerating deep neural network learning.

Referring to Figure 1, the system, the data storage module for storing the entire learning data; an adaptive batch selection module for adaptively selecting a learning mini-batch, which is a minimum unit for learning, from the data storage module; a network learning module that performs deep neural network learning on the received uncertainty-based mini-batch; a model prediction recording module for storing and managing only recent prediction results for mini-batch samples for the trained model; a sample uncertainty evaluation module for calculating sample uncertainty based on the results accumulated in the prediction recording module; a quantization module that converts the obtained uncertainty value into a quantitative exponent; and a sample selection probability calculation module for determining the sample selection probability by using the quantized index.

The data storage module 11 stores data for general supervised learning. Data is a set of multiple samples, and the i-th sample in the data includes both the multidimensional vector (feature) x _i indicating the characteristics of the sample and the actual label y _i information to which the sample belongs. For example, if the task is an image classification task, the R, G, and B values for the cat picture are the features, and the cat information is the label. Numerically, the data consisting of N samples is expressed as D={(x ₁ ,y ₁ ), (x ₂ ,y ₂ ), ..., (x _N ,y _N )} . The data storage module is a module that stores and manages the data, and data access is possible in the adaptive batch selection module.

The adaptive batch selection module 120 receives training samples from the data storage module and divides them into mini-batch, which is a minimum unit of a data sample set for deep neural network training. Typically 128 or 256 samples are included to make up one mini-batch. For example, if the number of samples in the total data is N and the number of samples in each mini-batch is the batch size, the total number of mini-batch is the same as Equation 1.

[Equation 1]

B=N/Batch Size

를 지속적으로 관리 및 업데이트하며 학습을 위한 미니 배치 u는 수학식 2와 같이 해당 분포로부터 선택된다. In the past, samples are randomly selected to construct one mini-batch, but in the present invention, the mini-batch is configured by adaptively selecting samples most suitable for learning the current model. At this time, the sample with a high probability that the model fits or is wrong with a high probability is selected with a low probability because it is too easy or difficult samples, and in the case of a sample with an uncertain model prediction, the next mini batch is chosen with a high probability for Thus, for the adaptive batch selection module, the uncertainty probability distribution for the samples in the data

is continuously managed and updated, and a mini-batch u for learning is selected from the corresponding distribution as in Equation (2).

[Equation 2]

The process of calculating the uncertainty probability distribution will be described later in modules (14) to (16).

The neural network learning module 13 operates as follows.

Learning of the deep neural network in the network learning module 131 is performed in units of mini-batch provided by the adaptive batch selection module. This is called mini-batch stochastic gradient descent, and in general, deep neural network training is performed on B mini-batches given in the batch selection module, which is called 1 Epcoh. do.

으로 구성되며, 표본 x_i에 대한 t시점 모델에 대한 손실을

라 했을 때, 모델의 매개변수를 수학식 3과 같이 업데이트하여 학습이 진행된다. 여기서 미니 배치 표본들은 앞서 언급한 불확실도 확률 분포를 활용하여 선택된 미니 배치 u에 속한 표본들이다.More specifically, in each learning step, the model performs label prediction on samples of a given mini-batch, that is, batch size, and calculates the loss of how inaccurate the model's prediction is. A neural network model is usually a set of parameters for constructing a large number of complex functions.

is composed of , and the loss for the time t model for sample x _i is

In this case, learning proceeds by updating the parameters of the model as in Equation (3). Here, the mini-batch samples are samples belonging to the mini-batch u selected using the aforementioned uncertainty probability distribution.

[Equation 3]

The model prediction recording module 132 outputs the predicted label for the sample x _i at each time t, similar to the loss value for the label prediction obtained in the neural network learning module. The model prediction recording module stores the predicted label value at time t for each sample. However, instead of maintaining the label values at all time points, only the prediction records for the most recent q epochs are maintained. In the case of the active bias described above, the growing window technique that utilizes model prediction for the entire period is used, whereas the present invention uses the sliding window technique that utilizes prediction only for the recent q epoch. Through this, the present invention succeeds in improving generalization performance while improving learning speed by overcoming the problem of erroneous uncertainty estimation due to old prediction values.

For the sample uncertainty evaluation and sample selection probability distribution calculation, which will be described later, the sliding window-based model prediction record for the entire sample is called H, and the prediction record for a specific sample x _i is called H _i . Collected model prediction records are updated per training epoch, and prediction records older than q epochs are deleted from the module.

The sample uncertainty evaluation module 14 calculates the sample uncertainty based on the model label prediction record during the most recent q epochs.

라고 할 때, 특정 라벨 j가 최근에 모델에 의해 표본의 라벨로 예측될 확률은 수학식 4와 같다. []는 Iverson bracket이다.First, record the last q predictions of sample x _i .

, the probability that the specific label j is recently predicted as the label of the sample by the model is the same as in Equation 4. [] is an Iverson bracket.

[Equation 4]

를 곱합으로써 항상

이 보장된다.Next, generally widely used empirical entropy is used, and the sample prediction uncertainty u(x _i ) for the final sample x _i is expressed by Equation (5). where k is the number of labels you want to predict and the prediction uncertainty is

always by multiplying

this is guaranteed

[Equation 5]

The quantization module 15 receives the uncertainty values for all the samples calculated in the sample uncertainty evaluation module, and quantizes each uncertainty u(x _i ) to convert it to a sample selection probability distribution to convert the quantization index Q(u(x _i ) )) is calculated as in Equation 5.

[Equation 5]

얻어진 양자화 지수는 각 표본이 다음 미니 배치로 선택될 확률인 표본 선택 확률

를 계산하는데 사용된다.The higher the prediction uncertainty, the lower the quantization index is obtained, and the quantization index ranges from 1 to N, the number of data samples.

The resulting quantization index is the sample selection probability, which is the probability that each sample will be selected in the next mini-batch.

is used to calculate

로 만들고 해당 확률 본포를 (12) 적응적 배치 선택 모듈로 전송한다.The sample selection probability module 16 calculates the quantization index Q(u(x _i )) of each sample obtained from the sample selection probability distribution.

, and transmits the corresponding probability distribution to the (12) adaptive batch selection module.

As mentioned above, a sample with a certain prediction from the model decreases the probability of being selected, while a sample with an uncertain prediction increases the probability of being selected. Therefore, each sample x _i selection probability is calculated according to Equation 6, which exponentially reduces the probability that the sample is selected as the next mini-batch by utilizing the quantization coefficient. Here, the selection pressure s _e is a parameter for controlling the difference in the selection probability between the sample with the most uncertainty and the sample with the most prediction. When a _high value of s is used, the uncertain sample is chosen with a higher probability, whereas when a _low value of s is used, even a certain sample can be selected as a mini-batch sample. In the extreme, if s _e =1, it operates in the same way as the existing random sampling method.

[Equation 6]

In general, excessive emphasis on specific samples in the late part of training causes the problem of overfitting the model to those samples. Therefore, the initial selection pressure s _e0 =100 exponentially attenuates the value using Equation 7 as learning proceeds to finally make s _e-end = 1 to avoid the problem of overfitting the model.

[Equation 7]

As such, the present disclosure solves the problem of lowering the learning speed due to the existing Growing Window technique through data uncertainty-based adaptive batch selection for accelerating deep neural network learning, and generalization of models for test data and model learning Speed can be improved at the same time.

According to the present disclosure, it is possible to obtain a final model showing higher accuracy on test data while accelerating model learning. Therefore, acceleration of deep neural network learning can be achieved without degradation of generalization performance, which has been a problem until now. As a result, it can be applied to various real-life applications such as autonomous driving and object search that require accelerated learning on large-scale data, and its performance can be greatly improved.

Claims (2)

data storage module to store the entire training data;
an adaptive batch selection module for adaptively selecting a learning mini-batch that is a minimum unit for learning from the data storage module;
A network learning module that performs deep neural network training on the received uncertainty-based mini-batch;
A model prediction recording module that stores and manages only the latest prediction results for mini-batch samples for the trained model;
a sample uncertainty evaluation module that calculates sample uncertainty based on the results accumulated in the prediction recording module;
a quantization module for converting the obtained uncertainty value into a quantitative exponent; and
A system, comprising: a sample selection probability calculation module that utilizes the quantized index to determine a sample selection probability. In claim 1,
The adaptive placement selection module is
A system that continuously manages and updates the uncertainty probability distribution for samples in the data and selects mini-batches from that distribution for training.

Images