KR20220151257A - Hyper-parameter optimization based on reinforcement learning and gaussian process regression - Google Patents

Abstract

Disclosed is a hyperparameter optimization technique based on a reinforcement learning and a Gaussian process regression. A hyperparameter optimization method performed by a hyperparameter optimization system according to one embodiment may comprise: a step of updating a machine learning model using a training set generated through a combination of hyperparameters selected from an agent based on reinforcement learning; and a step of predicting a compensation for the updated machine learning model using the Gaussian process regression. Therefore, the present invention is capable of reducing a cost for evaluation.

Description

Reinforcement Learning and Gaussian Process Regression Based Hyperparameter Optimization {HYPER-PARAMETER OPTIMIZATION BASED ON REINFORCEMENT LEARNING AND GAUSSIAN PROCESS REGRESSION}

The description below relates to techniques for optimizing hyperparameters.

Commonly used hyperparameter optimization methods include grid search (GS), random search (RS), Bayesian optimization (BO), gradient-based optimization (GO), and friendly optimization. (evolutionary optimization) and population-based optimization.

가 설정되면,

와 같이 몇 개의 "경험치"를 선택할 수 있다. 연속적인 오버래핑의 경우, 동일한 간격으로 이산화를 할 수 없으며, 오버헤드를 자신의 특성에 따라 분산시켜야 한다. 그리드 탐색은 하이퍼 파라미터의 서로 다른 조합 방식에 따라 각각 하나의 모델을 훈련시킨 후, 성능을 측정하고, 가장 성능이 좋은 사양을 선택한다. 그러나, 그리드 탐색은 필요한 함수 평가 수가 구성 공간의 차원성에 따라 기하급수적으로 증가하기 때문에 차원성의 저주에 시달린다. 그리드 검색의 또 다른 문제는 이산화의 분해능을 높이려면 필요한 함수 평가 수가 상당히 증가한다는 것이다. Grid search is a method of trying all hyperparameter combinations to find the appropriate block hyperparameter settings. Assuming that there are a total of K (K is a natural number), the kth number takes the mk number. The total setting combinations are m1*m1, ..., mK. If the overlapping is continuous, the overlapping can be dispersed. For example, the learning rate

is set,

You can choose a few "Experience Points" like . In the case of continuous overlapping, it is impossible to discretize at equal intervals, and the overhead must be distributed according to its own characteristics. Grid search trains one model each according to different combinations of hyperparameters, measures its performance, and selects the best-performing specification. However, grid search suffers from the curse of dimensionality because the number of required function evaluations grows exponentially with the dimensionality of the construction space. Another problem with grid search is that increasing the resolution of the discretization significantly increases the number of required function evaluations.

If there is a very large difference in the effect that affects model performance depending on the hyperparameter, some hyperparameters (eg, regularization coefficients) have a limited effect on the model performance, but some hyperparameters (eg, regularization coefficients) have a limited effect on the performance of the model. , learning rate) has a relatively large effect on the performance of the model. In this case, the grid search may attempt an insignificant overlap. For this reason, random search is more suitable. Random search selects a random combination of parameters and the best-performing configuration. In this way, random search selects the most suitable point from a randomly drawn set of points. Although both empirically and theoretically show that random search is more practical and efficient than grid search, random search does not promise an optimal search. This means that the longer the search, the more likely it is to find the optimal hyperparameters, but will consume more resources.

Bayesian optimization is an efficient method for global optimization of expensive blackbox functions. Bayesian optimization is a surrogate model based optimization (SMBO) method, which builds a stochastic model mapping from hyperparameters to target metrics evaluated on a validation set. It strikes a good balance between exploration (evaluating as many hyperparameter sets as possible) and attacking (allocating more resources to promising hyperparameters). The most important thing to note about Bayesian optimization is that once you find a local maxima or minima, it is easy to fall into a local maxima because it constantly samples in this region. Bayesian optimization needs to consume a large amount of resources and time. For high-dimensional, non-convex functions where unknown smoothness is noisy, Bayesian optimization algorithms often find it difficult to agree and optimize. In general, Bayesian optimization algorithms have very strong hypothesis conditions, and these requirements are generally difficult to meet. In addition, there are papers that say that Bayesian optimization algorithms are not significantly better than random searches.

Unlike black box HOP methods (e.g., grid search, random search, and Bayesian optimization), gradient-based optimization uses gradient information to optimize hyperparameters and greatly improve the efficiency of HPO. The hyperparameter space is usually not continuous or differentiable because it consists of many ideal decisions. Because of this, gradient descent cannot be made in normal hyperparameter space.

A method and system for optimizing hyperparameters based on reinforcement learning and Gaussian process regression can be provided.

In order to sequentially select hyperparameters using an agent and accelerate the training process, a combination of hyperparameters and rewards are presented as training data, and a method and system for predicting rewards using Gaussian process regression can be provided. have.

The hyperparameter optimization method performed by the hyperparameter optimization system includes updating a machine learning model using a training set generated through a combination of hyperparameters selected from an agent based on reinforcement learning; and predicting a reward for the updated machine learning model using Gaussian process regression.

The updating may include initializing a plurality of policies including a first policy and a second policy of a machine learning model, sampling data using the first policy, and training the second policy with the sampled data. and receiving a selection of a plurality of hyperparameters for hyperparameter optimization from the agent in the machine learning model.

The updating may include training a machine learning model using a training set generated according to a combination of the selected plurality of hyperparameters, obtaining accuracy of the trained machine learning model using a validation set, and obtaining the A step of using the obtained accuracy as a reward may be included.

The updating may include updating a first policy through a PPO algorithm for a machine learning model using the selected hyperparameter combination and the obtained reward, and Gaussian prediction of the selected hyperparameter combination and the obtained reward It may include using as a training set for.

In the predicting step, when the KL distances of the plurality of policies configured in the machine learning are less than or equal to the threshold value, Gaussian process regression is used, and a reward is predicted by using a combination of the selected hyperparameters as an input of the Gaussian process regression and updating a first policy through a PPO algorithm using the selected hyperparameter combination and the predicted reward.

In the predicting step, the reward is predicted using the Gaussian process regression, and if the KL distances of the plurality of policies configured in the machine learning are greater than a threshold value, the machine using a combination of other hyperparameters selected from a training set. It may include repeating the process of training the learning model.

The hyperparameter optimization system includes a reinforcement learning unit that updates a machine learning model using a training set generated through a combination of hyperparameters selected from an agent based on reinforcement learning; and a reward prediction unit that predicts a reward for the updated machine learning model using Gaussian process regression.

In order to predict the reward function of the hyperparameter selection process based on reinforcement learning, Gaussian process regression can be used to reduce the evaluation cost according to the optimization of the hyperparameter.

After the accuracy of the validation set is verified by training the machine learning model using the training set generated according to the selected hyperparameters, the Gaussian process is performed according to a specific control method using the hyperparameters and accuracy as a training set for Gaussian process regression. Regression can be used to predict accuracy.

Rewards can be accurately predicted through Gaussian process regression, and data that does not satisfy a normal distribution can also be predicted by Gaussian process regression.

1 is a flowchart illustrating the operation of a system for optimizing hyperparameters according to an exemplary embodiment.
2 is a block diagram for explaining the configuration of a system for optimizing hyperparameters according to an embodiment.
3 is a diagram for explaining a hyperparameter optimization method in a hyperparameter optimization system according to an embodiment.

Hereinafter, an embodiment will be described in detail with reference to the accompanying drawings.

In the embodiment, an operation of optimizing hyperparameters based on reinforcement learning and Gaussian process regression will be described. In order to sequentially select hyperparameters using an agent and accelerate the training process, hyperparameter combinations and rewards are presented as training data, and rewards can be predicted using Gaussian process regression.

1 is a flowchart illustrating the operation of a system for optimizing hyperparameters according to an exemplary embodiment.

In the embodiment, the selection of hyperparameters is abstracted into a sequence determination process, and since the next hyperparameter is selected according to the previous selection, a Markov Decision Process (MDP) can be modeled. Thus, reinforcement learning may be used to select hyperparameters. An agent can be modeled with LSTM (Long-Short-Memory). Agents can select hyperparameters sequentially. A next policy may be selected through Proximal Policy Optimization (PPO) using the accuracy obtained from the validation set as a reward. At this time, in an intermediate step, Gaussian Process Regression (GPR) can be used to predict the reward. It should be controlled when true rewards are used and when rewards are predicted by Gaussian process regression.

The hyperparameter system may update the machine learning model using a training set generated through a combination of hyperparameters selected from the agent based on reinforcement learning.

)과 제2 정책(

)을 포함하는 복수 개의 정책을 초기화할 수 있다. 하이퍼 파라미터 시스템은 제1 정책을 이용하여 데이터를 샘플링하고, 샘플링된 데이터로 제2 정책을 훈련함에 따라 머신러닝 모델을 업데이트할 수 있다. In detail, the hyperparameter system is a first policy (

) and the second policy (

) can initialize a plurality of policies including. The hyperparameter system may update the machine learning model by sampling data using the first policy and training a second policy with the sampled data.

)과 재2 정책(

)은 데이터를 샘플링하고 샘플링된 데이터로 제2 정책(

)을 훈련할 수 있다. 최적화할 복수 개(n개)의 하이퍼 파라미터의 수가 선택될 수 있다. LSTM으로 에이전트를 트레이닝시킬 수 있다. Initialization first policy (

) and second policy (

) Samples the data and uses the sampled data as the second policy (

) can be trained. The number of multiple (n) hyperparameters to be optimized can be selected. You can train an agent with LSTM.

가 업데이트될 수 있다. 하이퍼 파라미터의 조합과 보상이 가우시안 예측을 위한 트레이닝 세트로 사용될 수 있다. 이때, 가우시안 예측을 위한 트레이닝 세트(하이퍼 파라미터의 조합과 보상)가 저장될 수 있다. The hyperparameter system may receive selection of a plurality of hyperparameters from the agent. In other words, the agent can select a plurality of hyperparameters one by one. A machine learning model may be trained through a training set generated using selected hyperparameters. The accuracy (performance result evaluation) of the machine learning model trained using the validation set may be obtained. The accuracy of the validation set can be used as a reward. Through the PPO algorithm using a combination of hyperparameters and compensation

can be updated. A combination of hyperparameters and compensation can be used as a training set for Gaussian prediction. In this case, a training set (hyper-parameter combination and compensation) for Gaussian prediction may be stored.

The PPO method can be defined as follows.

L can be defined as:

를 업데이트할 수 있다. 그런 다음, 에이전트가 하이퍼 파라미터를 다시 선택할 수 있다. 액션(Action) 공간에서 모델을 사용하기 전후에 정책 간의 거리 측정이 정의될 수 있다.The hyperparameter system can use Gaussian process regression if the KL distance of the last two policies is less than or equal to the threshold. The agent still selects multiple hyperparameters one by one, uses the combination of selected hyperparameters as input to a Gaussian process regression to predict a reward, and then uses the combination of hyperparameters and the predicted reward to generate a reward through the PPO algorithm.

can be updated. The agent can then choose the hyperparameter again. Distance measures between policies can be defined before and after using the model in the action space.

Hyper-parameter systems can predict rewards by Gaussian process regression. If it is assumed that the agent selects an m-dimensional hyperparameter at this time, the training data predicted by Gaussian can be expressed as follows.

Since observations are usually noisy, each observation y can be modeled as Gaussian noise in the luminous function f(x).

가 된다. in other words,

becomes

f(x) can be assumed to give a Gaussian process a priori.

That is, f(x) to GP(0, K).

The covariance function can choose squared exponential.

이다. in other words,

to be.

Accordingly, k(x, x') with noise can be expressed as follows.

이다. in other words,

to be.

In this way, basic modeling can be completed regardless of other calculations. For each new x* (the new combination of hyperparameters chosen by the agent), we need to find the corresponding y*.

in those,

If there is a joint distribution, then we can easily obtain the conditional distribution p(y*|y) of y*. The condition distribution can also be obtained by deriving a Gaussian distribution as follows.

For the estimate of y*, the mean of the distribution can be used as an estimate.

Through this process, rewards can be predicted.

The error starts to increase when the KL distance of the last two policies for the machine learning model is greater than the threshold. The hyperparameter system can then train a machine learning model using the next selected hyperparameter combination from the training set. In other words, the process mentioned above can be repeated.

According to an embodiment, evaluation cost according to optimization of a hyperparameter may be reduced by using Gaussian process regression to predict a reward function of a hyperparameter selection process based on reinforcement learning.

According to an embodiment, after training a machine learning model using a training set generated according to selected hyperparameters, verifying the accuracy of the validation set, and then using the hyperparameters and accuracy as a training set for Gaussian process regression, a specific Depending on the control method (e.g., the distance between two policies in a machine learning model), Gaussian process regression can be used to predict accuracy.

According to an embodiment, Gaussian process regression has a great advantage in prediction, and even data that does not satisfy a normal distribution can be predicted by Gaussian process regression.

2 is a block diagram for explaining the configuration of a hyper parameter optimization system according to an embodiment, and FIG. 3 is a diagram for explaining a hyper parameter optimization method in the hyper parameter optimization system according to an embodiment.

The processor of the hyperparameter optimization system 100 may include a reinforcement learning unit 210 and a reward prediction unit 220 . Components of the processor may be representations of different functions performed by the processor according to control instructions provided by program codes stored in the hyperparameter optimization system. The processor and components of the processor may control the hyper parameter optimization system to perform steps 310 to 320 included in the hyper parameter optimization method of FIG. 3 . In this case, the processor and components of the processor may be implemented to execute instructions according to the code of an operating system included in the memory and the code of at least one program.

The processor may load a program code stored in a file of a program for a hyperparameter optimization method into a memory. For example, when a program is executed in the hyperparameter optimization system, the processor may control the hyperparameter optimization system to load a program code from a program file into a memory under the control of an operating system. At this time, each of the reinforcement learning unit 210 and the reward prediction unit 220 executes a command of a corresponding part of the program code loaded into the memory to perform the subsequent steps 310 to 320 with different functional representations of the processor. can be picked up

In step 310, the reinforcement learning unit 210 may update the machine learning model using a training set generated through a combination of hyperparameters selected from the agent based on reinforcement learning. The reinforcement learning unit 210 initializes a plurality of policies including the first policy and the second policy of the machine learning model, samples data using the first policy, trains the second policy with the sampled data, The machine learning model may receive a selection of a plurality of hyperparameters for hyperparameter optimization from an agent. The reinforcement learning unit 210 trains a machine learning model using a training set generated according to a combination of a plurality of selected hyperparameters, obtains accuracy of the trained machine learning model using a validation set, and obtains accuracy. can be used as a reward. The reinforcement learning unit 210 updates the first policy through the PPO algorithm for the machine learning model using the selected hyperparameter combination and the obtained reward, and uses the selected hyperparameter combination and the obtained reward for training for Gaussian prediction Can be used as a set.

In step 320, the reward predictor 220 may predict the reward for the updated machine learning model using Gaussian process regression. The reward prediction unit 220 uses Gaussian process regression when the KL distances of a plurality of policies configured in machine learning are less than or equal to the threshold value, predicts compensation by using a combination of selected hyperparameters as an input of Gaussian process regression, The first policy may be updated through the PPO algorithm using the selected combination of hyperparameters and the predicted reward. The reward prediction unit 220 predicts a reward using Gaussian process regression, and if the KL distances of a plurality of policies configured in machine learning are greater than a threshold value, a machine learning model using a combination of other hyperparameters selected from the training set The training process can be repeated.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. can be embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (7)

In the hyperparameter optimization method performed by the hyperparameter optimization system,
Updating a machine learning model using a training set generated through a combination of hyperparameters selected from an agent based on reinforcement learning; and
Predicting a reward for the updated machine learning model using Gaussian process regression.
Hyperparameter optimization method comprising a. According to claim 1,
The updating step is
A plurality of policies including a first policy and a second policy of a machine learning model are initialized, data is sampled using the first policy, the second policy is trained with the sampled data, and the machine learning model Receiving that a plurality of hyperparameters for hyperparameter optimization are selected from the agent in
Hyperparameter optimization method comprising a. According to claim 2,
The updating step is
A machine learning model is trained using a training set generated according to the combination of the selected plurality of hyperparameters, accuracy of the trained machine learning model is obtained using a validation set, and the obtained accuracy is used as a reward. step to do
Hyperparameter optimization method comprising a. According to claim 3,
The updating step is
Update a first policy through a PPO algorithm for a machine learning model using the selected combination of hyperparameters and the obtained reward, and use the selected combination of hyperparameters and the obtained reward as a training set for Gaussian prediction step to do
Hyperparameter optimization method comprising a. According to claim 1,
The predicting step is
If the KL distances of the plurality of policies configured in the machine learning are less than or equal to the threshold, Gaussian process regression is used, and a reward is predicted using a combination of the selected hyperparameters as an input of the Gaussian process regression, and the selected hyperparameters Updating a first policy through a PPO algorithm using a combination of and the predicted reward.
Hyperparameter optimization method comprising a. According to claim 5,
The predicting step is
Predicting a reward using the Gaussian process regression, and training the machine learning model using a combination of other hyperparameters selected from a training set if the KL distances of the plurality of policies configured in the machine learning are greater than a threshold value. steps to repeat
Hyperparameter optimization method comprising a. In the hyperparameter optimization system,
a reinforcement learning unit that updates a machine learning model using a training set generated through a combination of hyperparameters selected from an agent based on reinforcement learning; and
A reward prediction unit that predicts a reward for the updated machine learning model using Gaussian process regression
A hyperparameter optimization system comprising a.

Images