KR102605220B1 - System and method for optimization of hyperparameter - Google Patents

Abstract

A hyperparameter optimization system and method are provided. The hyperparameter optimization system according to an embodiment of the present invention extracts feature vectors for each of a plurality of previously prepared training data, and uses the feature vector and the K-NN (K-Nearest Neighbor) algorithm to a data selection unit that selects one or more learning data to be used to search for optimal hyperparameters among the learning data; Based on the selected learning data, a basis hopping algorithm is repeatedly performed to limit the search range of the hyperparameters, and a Bayesian optimization algorithm is performed within the limited search range to select one hyperparameter. a parameter search unit that recommends parameters; and a model creation unit that generates a new model by performing evaluation of learning and performance in the target model based on the recommended hyperparameters.

Description

Hyperparameter optimization system and method {SYSTEM AND METHOD FOR OPTIMIZATION OF HYPERPARAMETER}

Embodiments of the present invention relate to technology for optimizing hyperparameters that have a significant impact on the learning speed and performance of a target model, such as a deep learning model.

Deep Learning model's learning speed and performance vary depending on hyperparameters. Hyperparameters are not peripheral numbers that need to be tuned or optimized through learning, but rather variables that people set with a priori knowledge (priori) or are set automatically through an external model mechanism, such as learning progress rate or generalization variables. The hyperparameter may be, for example, a learning rate, a regularization parameter, the number of repetitions of training, the number of hidden units, etc.

Conventionally, learning was performed after selecting hyperparameters using methods such as grid search or random search, but in this case, depending on the amount of learning data used to select hyperparameters and the depth of the network architecture, etc. There was a problem that it took time. In addition, the Bayesian Optimization algorithm has recently been used as a technique for optimizing hyperparameters. However, in the case of the Bayesian optimization algorithm, it is not practical due to the search time that takes several hours to several days and the possibility of searching for a local optimum. There are limitations to its use in business.

Korean Patent Publication No. 10-1075824 (2011.10.25)

Embodiments of the present invention are intended to reduce the time required for constructing and learning a target model and to provide a means of finding the global optimum of hyperparameters.

According to an exemplary embodiment, feature vectors for a plurality of previously prepared training data are extracted, and an optimal hyperparameter among the plurality of training data is determined using the feature vector and a K-Nearest Neighbor (K-NN) algorithm. a data selection unit that selects one or more learning data to be used to explore (hyperparameter); Based on the selected learning data, a basis hopping algorithm is repeatedly performed to limit the search range of the hyperparameters, and a Bayesian optimization algorithm is performed within the limited search range to select one hyperparameter. a parameter search unit that recommends parameters; and a model generator that generates a new model by performing evaluation of learning and performance in a target model based on the recommended hyperparameters. A hyperparameter optimization system is provided.

The data selection unit applies the K-NN algorithm to the feature vector of each training data to determine a class of target data, which is one of the plurality of training data, and a representative class of the area to which the target data belongs, If the class of the target data is different from the representative class, the target data may be selected as learning data to be used to search for the optimal hyperparameter.

The data selection unit may select the one or more learning data in consideration of the number of learning data in the region for each class and the distance between the target data and the learning data in the region.

The parameter search unit stores a loss index and a performance evaluation index of the target model for a plurality of candidate hyperparameters derived from the Basin hopping algorithm or the Bayesian optimization algorithm, respectively, and stores the loss index and performance evaluation index for the kth candidate hyperparameter. When evaluating the performance of the target model, a moving average of n loss indicators corresponding to a performance evaluation indicator greater than a set value among the stored loss indicators is calculated, and the loss indicator of the target model for the kth candidate hyperparameter is calculated. You can compare the moving average with .

The parameter search unit, when the loss index of the target model for the k-th candidate hyperparameter is less than the moving average, determines whether to early stop learning. Learning is terminated, and if the loss index for the kth candidate hyperparameter is greater than or equal to the moving average, learning of the target model for the kth candidate hyperparameter may be continuously performed.

If the loss index for the kth candidate hyperparameter is greater than or equal to the moving average and the performance evaluation index for the kth candidate hyperparameter is greater than one of the stored performance evaluation indexes, the parameter search unit selects one of the n loss metrics. The moving average can be recalculated by replacing the loss index corresponding to the lowest performance evaluation index with the loss index for the kth candidate hyperparameter.

The loss indicator may be a Nat (natural unit of information) value of loss output during the learning process of the target model.

The performance evaluation index may include one or more of accuracy, error rate, sensitivity, precision, specificity, and false positive rate of the target model. You can.

The parameter search unit repeatedly performs the basis hopping algorithm based on the selected learning data to derive a plurality of first candidate hyperparameters, and sets the minimum and maximum values of each first candidate hyperparameter to the hyperparameters. The search range can be limited to .

The parameter search unit repeatedly performs the Bayesian optimization algorithm within the limited search range to derive a plurality of second candidate hyperparameters, and the best candidate among the plurality of first candidate Piper parameters and the plurality of second candidate hyperparameters. One hyperparameter that outputs the optimal performance evaluation index can be recommended.

According to another exemplary embodiment, in a data selection unit, extracting feature vectors for each of a plurality of already provided training data; In the data selection unit, selecting one or more learning data to be used to search for an optimal hyperparameter among the plurality of learning data using the feature vector and a K-Nearest Neighbor (K-NN) algorithm; In a parameter search unit, limiting the search range of the hyperparameter by repeatedly performing a basis hopping algorithm based on the selected learning data; Recommending one hyperparameter by performing a Bayesian optimization algorithm within the limited search range, in the parameter search unit; And in the model creation unit, a step of generating a new model by learning from a target model and evaluating the performance of the target model based on the recommended hyperparameters. Hyperparameters An optimization method is provided.

The step of selecting the one or more learning data includes applying the K-NN algorithm to the feature vector of each learning data to determine the class of the target data, which is one of the plurality of learning data, and the area to which the target data belongs. A representative class may be determined, and if the class of the target data is different from the representative class, the target data may be selected as learning data to be used to search for the optimal hyperparameter.

In the step of selecting the one or more learning data, the one or more learning data may be selected in consideration of the number of learning data in the region for each class and the distance between the target data and the learning data in the region.

The hyperparameter optimization method stores, in the parameter search unit, a loss index and a performance evaluation index of the target model for a plurality of candidate hyperparameters derived from the Basin hopping algorithm or the Bayesian optimization algorithm, respectively. step; Calculating, in the parameter search unit, a moving average of n loss indicators corresponding to performance evaluation indicators greater than a set value among the stored loss indicators when evaluating the performance of the target model for the kth candidate hyperparameter; and comparing, in the parameter search unit, a loss index of the target model for the k-th candidate hyperparameter with the moving average.

In the hyperparameter optimization method, after the comparison step, the parameter search unit determines whether to early stop learning when the loss index of the target model for the kth candidate hyperparameter is less than the moving average. terminating learning of the target model for the kth candidate hyperparameter before starting; Alternatively, the parameter search unit may further include continuously performing learning of the target model for the kth candidate hyperparameter when the loss index for the kth candidate hyperparameter is greater than or equal to the moving average.

In the hyperparameter optimization method, after the comparing step, in the parameter search unit, a loss index for the kth candidate hyperparameter is greater than or equal to the moving average, and a performance evaluation index for the kth candidate hyperparameter is calculated as the stored performance. If it is greater than one of the evaluation indicators, it may further include recalculating the moving average by replacing the loss indicator corresponding to the lowest performance evaluation indicator among the n loss indicators with the loss indicator for the kth candidate hyperparameter. You can.

The loss indicator may be a Nat (natural unit of information) value of loss output during the learning process of the target model.

The performance evaluation index may include one or more of accuracy, error rate, sensitivity, precision, specificity, and false positive rate of the target model. You can.

The step of limiting the search range of the hyperparameters includes setting the initial search range of the hyperparameters and setting the minimum and maximum values of each first candidate hyperparameter derived from the basis hopping algorithm in the process of learning the target model. can be limited to the search range of the hyperparameter.

The step of recommending one hyperparameter includes determining the performance of the target model for each first candidate hyperparameter and each second candidate hyperparameter derived from the Bayesian optimization algorithm in the process of learning the target model. Evaluation may be performed, and one hyperparameter that outputs the most optimal performance evaluation index among each of the first candidate hyperparameters and each of the second candidate hyperparameters may be recommended.

According to embodiments of the present invention, by limiting the minimum and maximum values of candidate hyperparameters derived by repeatedly performing the Basin hopping algorithm to the search range of the hyperparameter and performing the Bayesian optimization algorithm within the search range, The global optimum can be found efficiently while reducing parameter search time.

Additionally, according to embodiments of the present invention, by determining whether to perform additional learning before deciding whether to end learning early in the hyperparameter search process, the learning time can be shortened, and thus the overall search time can be reduced.

1 is a block diagram showing the detailed configuration of an optimization system according to an embodiment of the present invention.
Figure 2 is a flowchart illustrating a method of selecting learning data in the data selection unit according to the first embodiment of the present invention.
Figure 3 is a flowchart illustrating a method of selecting learning data in the data selection unit according to the second embodiment of the present invention.
Figure 4 is an example showing the learning data selection process according to embodiments of the present invention
Figure 5 is a flowchart illustrating a method of performing a first search process in a parameter search unit according to an embodiment of the present invention.
Figure 6 is a flowchart illustrating a method of performing a second search process in a parameter search unit according to an embodiment of the present invention.
Figure 7 is a flowchart illustrating a method of creating a new model in the model creation unit according to an embodiment of the present invention.
8 is a block diagram illustrating and illustrating a computing environment including a computing device suitable for use in example embodiments.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The detailed description below is provided to provide a comprehensive understanding of the methods, devices and/or systems described herein. However, this is only an example and the present invention is not limited thereto.

In describing the embodiments of the present invention, if it is determined that a detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification. The terminology used in the detailed description is merely for describing embodiments of the present invention and should in no way be limiting. Unless explicitly stated otherwise, singular forms include plural meanings. In this description, expressions such as “comprising” or “comprising” are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, and one or more than those described. It should not be construed to exclude the existence or possibility of any other characteristic, number, step, operation, element, or part or combination thereof.

Figure 1 is a block diagram showing the detailed configuration of the optimization system 100 according to an embodiment of the present invention. The optimization system 100 according to an embodiment of the present invention is a system for optimizing hyperparameters that have a significant impact on the learning speed and performance of a target model.

In these embodiments, the target model is an objective function that is the subject of training and performance evaluation, for example, a deep learning model, SVM (Support Vector Machine) model, etc. This may apply. In addition, hyperparameters are not peripheral numbers that need to be tuned or optimized through learning, but rather variables that people set with a priori knowledge (priori) or are automatically set through an external model mechanism, such as learning progress rate or generalization variables. The hyperparameter may be, for example, a learning rate, a regularization parameter, the number of repetitions of training, the number of hidden units, etc. In addition, learning in the target model refers to the process of minimizing the loss function calculated from the learning data, and hyperparameter optimization refers to the parameter (or the function value of the objective function) that lowers the value of the loss function as much as possible. This means searching for parameters that maximize or minimize), that is, searching for optimal weights that minimize the error.

As shown in FIG. 1, the optimization system 100 according to an embodiment of the present invention includes a data selection unit 102, a parameter search unit 104, and a model creation unit 106.

The data selection unit 102 extracts feature vectors for each of the plurality of training data, and selects the optimal hyperparameter among the plurality of training data using the feature vector and K-NN (K-Nearest Neighbor) algorithm. Select one or more learning data to be used to explore. Conventionally, in order to find the optimal hyperparameter, all training data or training data selected through random sampling were used, but in this case, there was a problem that the training time took a long time or the learning performance was somewhat poor. Accordingly, in embodiments of the present invention, training data with high influence (or uncertainty) on learning performance is selected instead of all training data, and then hyperparameters are searched based on the selected training data.

To this end, the data selection unit 102 may first extract feature vectors for each of the plurality of pre-installed training data. Thereafter, the data selection unit 102 applies the K-NN algorithm to the feature vector of each learning data to select the class of the target data, which is one of the plurality of learning data, and the representative class of the area to which the target data belongs. can be decided. Specifically, the data selection unit 102 calculates the Euclidean distance between each learning data using the feature vector to determine the class of the target data and the class of learning data surrounding the target data, respectively. From this, it is possible to determine the representative class of the area to which the target data belongs, that is, the class that exists most frequently in the area. At this time, the boundary of the area, that is, K of the K-NN algorithm, can be calculated, for example, as in Equation 1 below.

[Equation 1]

here, is the total number of training data, represents the total number of classes to which the learning data belong, and s represents the noise coefficient.

If the class of the target data is different from the representative class, the data selection unit 102 may select the target data as learning data to be used to search for optimal hyperparameters.

As an example, if the class of the target data is a and the representative class of the area to which the target class belongs is b, the target data can be viewed as learning data that has a high impact on learning performance (or classification performance). Accordingly, the data selection unit 102 can select the target data as learning data to be used to search for optimal hyperparameters.

Additionally, the data selection unit 102 may select the one or more learning data in consideration of the number of learning data in the region for each class and the distance between the target data and the learning data in the region. Specifically, the data selection unit 102 obtains the number of training data in the region for each class using the feature vector and the K-NN algorithm, and uses the feature vector and the K-NN algorithm to obtain the number of training data in the region according to the distance between the target data and the training data in the region. Voting results for each class can be derived by applying the borda count algorithm to each class. At this time, the data selection unit 102 may assign a higher weight to learning data that exists close to the target data.

In the example above, there are three learning data having the same class as the class of the target data (e.g., a), and learning data having the same class as the representative class (e.g., b) of the area to which the target class belongs. When there are four, and the learning data of class a are located closer to the target data than the learning data of class b, the data selection unit 102 uses the target data to search for optimal hyperparameters. It may not be selected as training data. In this case, although the number of classes corresponding to the target data is relatively small, the learning data having the classes are relatively concentrated and it is not very difficult to classify them, so the data selection unit 102 selects the target data as an optimal hyperparameter. It may not be selected as training data to be used to explore.

In this way, the data selection unit 102 can select learning data that has a high impact on learning performance instead of all learning data using the K-NN algorithm, Boda Count, etc. However, the method by which the data selection unit 102 selects the learning data is not limited to this, and the data selection unit 102 may select the learning data in various ways. For example, the data selection unit 102 may select learning data using a product quantization technique. Specifically, the data selection unit 102 divides the feature vectors of each learning data into M equal parts to generate subvectors, and applies a k-mean clustering technique to each subvector to create a plurality of subvectors. A centroid can be created. The data selection unit 102 calculates the Euclidean distance for the center closest to each subvector, and thus the feature vector of each learning data has distance values that can be calculated from the distance to the center. The data selection unit 102 may select learning data having the smallest distance value, the largest distance value, and the middle range distance value for each class in order to extract the same number of learning data for each class, and selects in this way. The learned data can be used to explore the hyperparameters later.

The parameter search unit 104 searches for optimal hyperparameters based on the selected learning data. Conventionally, in order to find optimal hyperparameters, an initial search range was set and a Bayesian optimization algorithm was performed using the search range. In general, the Bayesian optimization algorithm uses the trade-off of Exploration-Exploitation to find the global optimum. However, in the case of the Bayesian optimization algorithm, exploration is relatively lacking, so if the initial search range or first candidate hyperparameter is selected incorrectly, there is a possibility of searching for a local optimum. Accordingly, in embodiments of the present invention, a Basin hopping algorithm is performed before performing the Bayesian optimization algorithm to reduce the search range of hyperparameters and the Bayesian optimization algorithm is performed within the reduced search range.

To be more specific, the parameter search unit 104 can determine the optimal hyperparameter through the first and second search processes as follows. Here, the first search process refers to a process of searching hyperparameters through a Basin hopping algorithm, and the second search process refers to a process of searching hyperparameters through a Bayesian optimization algorithm.

* First search process

The parameter search unit 104 may repeatedly perform a basis hopping algorithm based on the selected learning data to derive a plurality of first candidate hyperparameters. The Basin hopping algorithm is an algorithm that stochastically searches for the global optimal solution of the objective function, and has the advantage of being robust in exploration. When a plurality of first candidate hyperparameters are derived, the parameter search unit 104 may limit (or set) the minimum and maximum values of each derived first candidate hyperparameter to the search range of the hyperparameter. Meanwhile, the method of deriving the first candidate hyperparameter through the basis hopping algorithm is generally well known in the technical field to which the present invention pertains, so a detailed description thereof will be omitted.

* Second search process

The parameter search unit 104 repeatedly performs a Bayesian optimization algorithm within the limited search range in the first search process to derive a plurality of second candidate hyperparameters. The Bayesian optimization algorithm, like the Basin hopping algorithm, is an algorithm that searches for the global optimal solution of the objective function. However, the Bayesian optimization algorithm has the advantage of being stronger against exploitation than the Basin hopping algorithm. The parameter search unit 104 may perform the Bayesian optimization algorithm within the search range, and accordingly, each of the second candidate hyperparameters exists within the search range. Meanwhile, the method of deriving the second candidate hyperparameter through the Bayesian optimization algorithm is generally known in the technical field to which the present invention pertains, and detailed description thereof will be omitted.

Thereafter, the parameter search unit 104 may recommend one hyperparameter that outputs the most optimal performance evaluation index among the plurality of first candidate Piper parameters and the plurality of second candidate hyperparameters.

In this way, the parameter search unit 104 can repeatedly perform the Basin hopping algorithm a set number of searches to limit the search range, and repeatedly perform the Bayesian optimization algorithm within the search range a set number of searches. For example, the number of searches may be preset by the user. Here, for convenience of explanation, it is assumed that the number of searches in the first search process and the number of searches in the second search process are each 50.

As an example, the parameter search unit 104 performs the basis hopping algorithm 50 times to derive h ₁ to h ₅₀ , and searches for the minimum value (h _min ) and maximum value (h _max ) of h ₁ to h ₅₀ . It can be limited by range (i.e. h _min < search range < h _max ). Thereafter, the parameter search unit 104 may perform the Bayesian optimization algorithm 50 times within the search range to derive h ₅₁ to h ₁₀₀ . Additionally, the parameter search unit 104 may determine one hyperparameter outputting the most optimal performance evaluation index among h ₁ to h ₁₀₀ as the optimal hyperparameter. Hereinafter, for convenience of explanation, hyperparameters derived through the Basin hopping algorithm will be referred to as first candidate hyperparameters, and hyperparameters derived through the Bayesian optimization algorithm will be referred to as second candidate hyperparameters.

In addition, the parameter search unit 104 may learn a target model for each first candidate hyperparameter and each second candidate hyperparameter, and in this process, the target model for each first candidate hyperparameter and each second candidate hyperparameter Loss indicators and performance evaluation indicators of the target model can be obtained. The parameter search unit 104 can reduce the learning time in the first search process and the second search process by using the loss index and the performance evaluation index.

To this end, the parameter search unit 104 may store a loss index and a performance evaluation index of the target model for a plurality of candidate hyperparameters derived from the Basin hopping algorithm or the Bayesian optimization algorithm, respectively. Here, the loss index is an index indicating the degree of difference (error) between the value predicted through the target model and the actual value when using the candidate hyperparameter. For example, the loss index (nat) that appears during the learning process of the target model It may be a natural unit of information) value. The loss indicator (nats of loss) can be expressed, for example, as Equation 2 below.

[Equation 2]

At this time, batch size refers to the number of learning data included in one batch. Additionally, since the loss value does not have much meaning in the early stages of learning, the loss value used in Equation 2 above may be the loss value when learning has progressed to some extent. The loss value used in Equation 2 may be, for example, the loss value in the p to qth epoch (in this case, 1 < p < q).

In addition, the performance evaluation index is, for example, one of accuracy, error rate, sensitivity, precision, specificity, and false positive rate of the target model. It may include more.

As an example, the parameter search unit 104 may store loss indicators and performance evaluation indicators for h ₁ to h ₂₀ , respectively.

Thereafter, when evaluating the performance of the target model for the k-th candidate hyperparameter, the parameter search unit 104 calculates a moving average of n loss indicators corresponding to a performance evaluation indicator greater than a set value among the stored loss indicators. You can.

In the above example, when evaluating the performance of the target model for the 21st candidate hyperparameter, that is, h ₂₁ , the parameter search unit 104 calculates a moving average of n loss indicators corresponding to a performance evaluation indicator greater than a set value among the stored loss indicators. can be calculated. For example, the parameter search unit 104 may calculate a moving average of 15 loss indicators corresponding to candidate hyperparameters that output a performance evaluation indicator greater than a set value among the 20 loss indicators stored to date.

Thereafter, the parameter search unit 104 may compare the moving average with the loss index of the target model for the k-th candidate hyperparameter.

If the loss index of the target model for the kth candidate hyperparameter is less than the moving average, the parameter search unit 104 determines whether to stop learning early. Additional learning of the target model may be terminated. In this case, since the loss is so large that additional learning is meaningless, the parameter search unit 104 may evaluate the kth candidate hyperparameter without performing additional learning.

If the loss index for the kth candidate hyperparameter is greater than or equal to the moving average, the parameter search unit 104 may continuously perform learning of the target model for the kth candidate hyperparameter. Additionally, the parameter search unit 104 may determine whether to early stop the learning according to the results of later learning. As an example, if the degree of change in loss or change in accuracy during the learning process continuously falls within a set value (eg, 1%), the parameter search unit 104 may terminate the learning early.

In addition, the parameter search unit 104 operates when the loss index for the kth candidate hyperparameter is greater than or equal to the moving average and the performance evaluation index for the kth candidate hyperparameter is greater than one of the stored performance evaluation indexes. The moving average can be recalculated by replacing the loss indicator corresponding to the lowest performance evaluation indicator among the loss indicators with the loss indicator for the kth candidate hyperparameter.

In the example above, if the loss index for the 21st candidate hyperparameter is greater than the moving average and the performance evaluation index for the 21st candidate hyperparameter is greater than one of the stored performance evaluation indexes, the parameter search unit 104 determines the stored performance evaluation index. The moving average can be newly calculated by replacing the loss indicator corresponding to the lowest performance evaluation indicator among the 15 loss indicators with the loss indicator for the 21st candidate hyperparameter. In this case, the moving average may be higher than before.

As such, according to embodiments of the present invention, by determining whether to perform additional learning before deciding whether to end learning early in the hyperparameter search process, the learning time can be shortened, and thus the overall search time can be reduced.

The model creation unit 106 additionally evaluates the learning and performance of the target model based on the hyperparameters recommended by the parameter search unit 104 and creates a new model therefrom.

Figure 2 is a flowchart illustrating a method of selecting learning data in the data selection unit 102 according to the first embodiment of the present invention. In the illustrated flow chart, the method is divided into a plurality of steps, but at least some of the steps are performed in a different order, combined with other steps, omitted, divided into detailed steps, or not shown. One or more steps may be added and performed.

In step S102, the data selection unit 102 extracts feature vectors for each of the plurality of prepared training data.

In step S104, the data selection unit 102 selects random target data and calculates the distance (eg, Euclidean distance) between the target data and surrounding learning data.

In step S106, the data selection unit 102 determines the class of the target data and the representative class of surrounding learning data through the K-NN algorithm.

In step S108, the data selection unit 102 compares the class of target data and the representative class.

In step S110, if the class of the target data and the representative class do not match as a result of the comparison in step S108, the data selection unit 102 selects the target data as learning data to be used for hyperparameter search. If, as a result of the comparison in step S108, the class of the target data matches the representative class, the data selection unit 102 returns to step S104 and selects other target data. Thereafter, the data selection unit 102 repeats the above-described process while performing steps S106 and S108 for the newly selected target data.

Figure 3 is a flowchart illustrating a method of selecting learning data in the data selection unit 102 according to the second embodiment of the present invention. In the illustrated flow chart, the method is divided into a plurality of steps, but at least some of the steps are performed in a different order, combined with other steps, omitted, divided into detailed steps, or not shown. One or more steps may be added and performed.

In step S202, the data selection unit 102 extracts feature vectors for each of the plurality of prepared training data.

In step S204, the data selection unit 102 selects random target data and calculates the distance (eg, Euclidean distance) between the target data and surrounding learning data.

In step S206, the data selection unit 102 determines the class of the target data and the representative class of surrounding learning data through the K-NN algorithm.

In step S208, the data selection unit 102 applies a borda count algorithm to each class according to the distance between the target data and learning data in the area to obtain a voting result for each class, that is, Derive a score.

In step S210, the data selection unit 102 determines whether the score corresponding to the same class as the class of the target data is less than or equal to the score corresponding to the representative class.

In step S212, if the score corresponding to the same class as the class of the target data is less than or equal to the score corresponding to the representative class, the data selection unit 102 selects the target data as learning data to be used for hyperparameter search. If, as a result of the comparison in step S210, the score corresponding to the same class as the class of the target data is greater than the score corresponding to the representative class, the data selection unit 102 returns to step S204 and selects other target data. Thereafter, the data selection unit 102 repeats the above-described process while performing steps S206 to S210 for the newly selected target data.

Figure 4 is an example showing a learning data selection process according to embodiments of the present invention.

As described above, the data selection unit 102 extracts feature vectors for each of the plurality of training data, and uses the feature vectors and the K-NN algorithm to search for hyperparameters among the plurality of training data. The learning data to be used can be selected. Here, for convenience of explanation, it is assumed that the learning data is an image.

Referring to FIG. 4, feature vectors for each of a plurality of images can be extracted based on a previously learned model, and distances between each image can be calculated based on the feature vectors. Figures 4 (a) and (b) respectively show a target image and other images in the area to which the target image belongs. Here, Figure 4 (a) is an example showing the learning data selection process according to the first embodiment of the present invention, and Figure 4 (b) is an example showing the learning data selection process according to the second embodiment of the present invention. am. Additionally, in Figures 4 (a) and (b), images belonging to the same class are displayed with the same hatched shape.

First, referring to (a) of FIG. 4, the data selection unit 102 may compare the class of the target image with the class of other images surrounding the target image. As a result of the comparison, the number of classes (e.g., a) that are the same as the class of the target image is 3, and the number of classes (e.g., b, c, d) that are different from the class of the target image are 4 and 4, respectively. You can see that there are four. In this case, since the class of the target image is different from the representative class of the area to which the target image belongs (that is, the number of classes identical to the class of the target image is smaller than the number of classes different from the class of the target image), the data selection unit (102) may select the target image as learning data to be used for hyperparameter search.

Next, referring to (b) of FIG. 4, the data selection unit 102 applies a counting algorithm to each class according to the distance between the target image and other images in the area to obtain a voting result for each class. result), that is, a score can be derived. At this time, the data selection unit 102 may assign a higher weight to other images that exist close to the target image. That is, the data selection unit 102 selects k, k-1, k-2,... for each of the other images in the order of proximity to the target image. Weights of 2 and 1 can be assigned respectively. Accordingly, the score for the same class as the target image's class, i.e., class a, is 15 + 14 + 13 = 42, and the scores for the classes different from the target image's class, i.e., classes b, c, and d, are respectively 12 + 11 + 5 + 2 = 30, 10 + 9 + 8 + 4 = 31, 7 + 6 + 3 + 1 = 17. Thereafter, the data selection unit 102 may determine whether the score corresponding to the same class as the class of the target image is less than or equal to the score corresponding to the representative class. In the above example, since the score corresponding to the same class as the target image class (i.e., 42 points) is greater than the score corresponding to the representative class (i.e., 30 points, 31 points, and 17 points), the data selection unit 102 ) does not select the target image as learning data for hyperparameter search.

That is, even the same target data may or may not be selected as learning data for hyperparameter search according to the above embodiments.

FIG. 5 is a flowchart illustrating a method of performing a first search process in the parameter search unit 104 according to an embodiment of the present invention. In the illustrated flow chart, the method is divided into a plurality of steps, but at least some of the steps are performed in a different order, combined with other steps, omitted, divided into detailed steps, or not shown. One or more steps may be added and performed.

In step S302, the parameter search unit 104 sets the initial search range of the hyperparameter. For example, the parameter search unit 104 may set the initial search range of hyperparameters using various known statistical techniques.

In step S304, the parameter search unit 104 performs a basis hopping algorithm based on the learning data selected within the initial search range. Accordingly, a first candidate hyperparameter (eg, h ₁ ) may be derived.

In step S306, the parameter search unit 104 trains a target model for the first candidate hyperparameter (eg, h ₁ ) derived from the basis hopping algorithm.

In step S308, the parameter search unit 104 stores a loss index and a performance evaluation index of the target model for the first candidate hyperparameter (e.g., h ₁ ) derived from the basis hopping algorithm, respectively. do.

Thereafter, steps S304 and S308 are repeatedly performed for the next first candidate hyperparameter. Accordingly, the loss index and performance evaluation index of the target model for a plurality of first candidate hyperparameters (eg, h ₁ to h ₂₀ ) are respectively stored.

Thereafter, in step S310, when evaluating the performance of the target model for the kth candidate hyperparameter (e.g., h ₂₁ ), the parameter search unit 104 selects a performance evaluation index corresponding to a set value or more among the stored loss indices. Calculate the moving average of n loss indicators.

In step S312, the parameter search unit 104 compares the loss index of the target model for the kth candidate hyperparameter (eg, h ₂₁ ) with the moving average.

In step S314, the parameter search unit 104 selects the kth candidate hyperparameter if the loss index of the target model for the kth candidate hyperparameter (for example, h ₂₁ ) is greater than or equal to the moving average as a result of the comparison in step S312. The learning for parameters (e.g., h ₂₁ ) is continuously performed, and it is later determined whether or not the learning should be terminated early.

For example, the parameter search unit 104 may determine to terminate the learning early if the degree of change in loss or change in accuracy during the learning process continuously falls within a set value (e.g., 1%), In this case, the previous process may be repeated for the next first candidate hyperparameter (e.g., h ₂₂ ) derived from the basis hopping algorithm by returning to step S304. In addition, if the degree of change in loss or accuracy during the learning process does not fall within a continuously set value (e.g., 1%), the parameter search unit 104 may continuously perform learning of the target model. (S306).

In addition, as a result of the comparison in step S312, if the loss index of the target model for the k-th candidate hyperparameter (for example, h ₂₁ ) is less than the moving average, the parameter search unit 104 determines the k-th candidate hyperparameter Training of the target model for (e.g., h ₂₁ ) is terminated. Thereafter, the parameter search unit 104 may return to step S304 and repeat the previous process for the next first candidate hyperparameter (eg, h ₂₂ ) derived from the basis hopping algorithm.

In this way, the parameter search unit 104 may repeatedly perform steps S304 to S314 for each of the plurality of first candidate hyperparameters (e.g., h ₁ to h ₅₀ ) derived through the basis hopping algorithm. there is.

In step S316, the parameter search unit 104 limits the minimum and maximum values of each first candidate hyperparameter (e.g., h ₁ to h ₅₀ ) derived through a basis hopping algorithm to the search range of the hyperparameter. can do.

Figure 6 is a flowchart illustrating a method of recommending one hyperparameter in the parameter search unit 104 according to an embodiment of the present invention. In the illustrated flow chart, the method is divided into a plurality of steps, but at least some of the steps are performed in a different order, combined with other steps, omitted, divided into detailed steps, or not shown. One or more steps may be added and performed.

In step S402, the parameter search unit 104 performs a Bayesian optimization algorithm within the limited search range, from which a second candidate hyperparameter (eg, h ₅₁ ) is derived.

Thereafter, the parameter search unit 104 performs steps S404 to S412 in the same manner as in FIG. 5. At this time, unlike in FIG. 5 , the parameter search unit 104 performs steps S404 to S412 using second candidate hyperparameters (for example, h ₅₁ to h ₁₀₀ ) instead of the first candidate hyperparameter.

In step S414, the parameter search unit 104 selects the most optimal among each first candidate hyperparameter (e.g., h ₁ to h ₅₀ ) and each second candidate hyperparameter (e.g., h ₅₁ to h ₁₀₀ ). We recommend one hyperparameter that outputs performance evaluation indicators.

FIG. 7 is a flowchart illustrating a method of generating a new model in the model creation unit 106 according to an embodiment of the present invention.

In step S502, the model creation unit 106 receives optimal hyperparameters (eg, h ₇₂ ) from the parameter search unit 104.

In step S504, the model generator 106 generates a new model by additionally performing learning on the target model based on the optimal hyperparameter (eg, h ₇₂ ).

In step S506, the model creation unit 106 evaluates the performance of the newly created model.

8 is a block diagram illustrating and illustrating a computing environment including a computing device suitable for use in example embodiments. In the illustrated embodiment, each component may have different functions and capabilities in addition to those described below, and may include additional components other than those described below.

The illustrated computing environment 10 includes a computing device 12 . In one embodiment, computing device 12 may be optimization system 100, or one or more components included in optimization system 100.

Computing device 12 includes at least one processor 14, a computer-readable storage medium 16, and a communication bus 18. Processor 14 may cause computing device 12 to operate in accordance with the example embodiments noted above. For example, processor 14 may execute one or more programs stored on computer-readable storage medium 16. The one or more programs may include one or more computer-executable instructions, which, when executed by the processor 14, cause computing device 12 to perform operations according to example embodiments. It can be.

Computer-readable storage medium 16 is configured to store computer-executable instructions or program code, program data, and/or other suitable form of information. The program 20 stored in the computer-readable storage medium 16 includes a set of instructions executable by the processor 14. In one embodiment, computer-readable storage medium 16 includes memory (volatile memory, such as random access memory, non-volatile memory, or an appropriate combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash It may be memory devices, another form of storage medium that can be accessed by computing device 12 and store desired information, or a suitable combination thereof.

Communication bus 18 interconnects various other components of computing device 12, including processor 14 and computer-readable storage medium 16.

Computing device 12 may also include one or more input/output interfaces 22 and one or more network communication interfaces 26 that provide an interface for one or more input/output devices 24. The input/output interface 22 and the network communication interface 26 are connected to the communication bus 18. Input/output device 24 may be coupled to other components of computing device 12 through input/output interface 22. Exemplary input/output devices 24 include, but are not limited to, a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touch screen), a voice or sound input device, various types of sensor devices, and/or imaging devices. It may include input devices and/or output devices such as display devices, printers, speakers, and/or network cards. The exemplary input/output device 24 may be included within the computing device 12 as a component constituting the computing device 12, or may be connected to the computing device 12 as a separate device distinct from the computing device 12. It may be possible.

Although the present invention has been described in detail above through representative embodiments, those skilled in the art will recognize that various modifications to the above-described embodiments are possible without departing from the scope of the present invention. You will understand. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined not only by the claims described later but also by equivalents to the claims.

10: Computing environment
12: Computing device
14: processor
16: computer-readable storage medium
18: communication bus
20: Program
22: input/output interface
24: input/output device
26: Network communication interface
100: Optimization system
102: Data selection unit
104: Parameter search unit
106: Model creation unit

Claims (20)

To extract feature vectors for each of the plurality of training data, and to search for optimal hyperparameters among the plurality of training data using the feature vector and K-NN (K-Nearest Neighbor) algorithm. a data selection unit that selects one or more learning data to be used;
Based on the selected learning data, a basis hopping algorithm is repeatedly performed to limit the search range of the hyperparameters, and a Bayesian optimization algorithm is performed within the limited search range to select one hyperparameter. a parameter search unit that recommends parameters; and
A model generator that generates a new model by performing evaluation of learning and performance in the target model based on the recommended hyperparameters,
The parameter search unit repeatedly performs the basis hopping algorithm based on the selected learning data to derive a plurality of first candidate hyperparameters, and sets the minimum and maximum values of each first candidate hyperparameter to the hyperparameters. A hyperparameter optimization system limited to a search range of .
In claim 1,
The data selection unit applies the K-NN algorithm to the feature vector of each training data to determine a class of target data, which is one of the plurality of training data, and a representative class of the area to which the target data belongs, A hyperparameter optimization system that selects the target data as learning data to be used to search for the optimal hyperparameter when the class of the target data is different from the representative class.
In claim 2,
The data selection unit selects the one or more learning data in consideration of the number of learning data in the region for each class and the distance between the target data and the learning data in the region.
In claim 1,
The parameter search unit stores a loss index and a performance evaluation index of the target model for a plurality of candidate hyperparameters derived from the Basin hopping algorithm or the Bayesian optimization algorithm, respectively, and stores the loss index and performance evaluation index for the kth candidate hyperparameter. When evaluating the performance of the target model, a moving average of n loss indicators corresponding to a performance evaluation indicator greater than a set value among the stored loss indicators is calculated, and the loss indicator of the target model for the kth candidate hyperparameter is calculated. A hyperparameter optimization system that compares and the moving average.
In claim 4,
The parameter search unit, when the loss index of the target model for the k-th candidate hyperparameter is less than the moving average, determines whether to early stop learning. A hyperparameter optimization system that terminates learning and continues learning the target model for the kth candidate hyperparameter when the loss index for the kth candidate hyperparameter is greater than or equal to the moving average.
In claim 5,
If the loss index for the kth candidate hyperparameter is greater than or equal to the moving average and the performance evaluation index for the kth candidate hyperparameter is greater than one of the stored performance evaluation indexes, the parameter search unit selects one of the n loss metrics. A hyperparameter optimization system that recalculates the moving average by replacing the loss index corresponding to the lowest performance evaluation index with the loss index for the kth candidate hyperparameter.
In claim 4,
The loss indicator is a hyperparameter optimization system that is a Nat (natural unit of information) value of the loss output during the learning process of the target model.
In claim 4,
The performance evaluation index includes one or more of accuracy, error rate, sensitivity, precision, specificity, and false positive rate of the target model. , Hyperparameter optimization system.
delete In claim 1,
The parameter search unit repeatedly performs the Bayesian optimization algorithm within the limited search range to derive a plurality of second candidate hyperparameters, and the best candidate among the plurality of first candidate Piper parameters and the plurality of second candidate hyperparameters. A hyperparameter optimization system that recommends one hyperparameter that outputs the optimal performance evaluation index.
In the data selection unit, extracting feature vectors for each of the plurality of training data already provided;
In the data selection unit, selecting one or more learning data to be used to search for an optimal hyperparameter among the plurality of learning data using the feature vector and a K-Nearest Neighbor (K-NN) algorithm;
In a parameter search unit, limiting the search range of the hyperparameter by repeatedly performing a basis hopping algorithm based on the selected learning data;
Recommending one hyperparameter by performing a Bayesian optimization algorithm within the limited search range, in the parameter search unit; and
In the model creation unit, generating a new model by learning from a target model and evaluating the performance of the target model based on the recommended hyperparameters,
The step of limiting the search range of the hyperparameters includes setting the initial search range of the hyperparameters and setting the minimum and maximum values of each first candidate hyperparameter derived from the basis hopping algorithm in the process of learning the target model. A hyperparameter optimization method that limits to the search range of the hyperparameter.
In claim 11,
The step of selecting the one or more learning data includes applying the K-NN algorithm to the feature vector of each learning data to determine the class of the target data, which is one of the plurality of learning data, and the area to which the target data belongs. A hyperparameter optimization method that determines a representative class and, if the class of the target data is different from the representative class, selects the target data as learning data to be used to search for the optimal hyperparameter.
In claim 12,
The step of selecting the one or more learning data includes selecting the one or more learning data in consideration of the number of learning data in the region for each class and the distance between the target data and the learning data in the region, optimization of hyperparameters. method.
In claim 11,
In the parameter search unit, storing a loss index and a performance evaluation index of the target model for a plurality of candidate hyperparameters derived from the Basin hopping algorithm or the Bayesian optimization algorithm, respectively;
Calculating, in the parameter search unit, a moving average of n loss indicators corresponding to performance evaluation indicators greater than a set value among the stored loss indicators when evaluating the performance of the target model for the kth candidate hyperparameter; and
A hyperparameter optimization method further comprising comparing, in the parameter search unit, a loss index of the target model for the kth candidate hyperparameter with the moving average.
In claim 14,
After the above comparing step,
In the parameter search unit, if the loss index of the target model for the k-th candidate hyperparameter is less than the moving average, the target model for the k-th candidate hyperparameter before determining whether to early stop learning. terminating learning; or
Optimization of hyperparameters, further comprising, in the parameter search unit, continuously performing learning of the target model for the kth candidate hyperparameter when the loss index for the kth candidate hyperparameter is greater than or equal to the moving average. method.
In claim 15,
After the above comparing step,
In the parameter search unit, when the loss index for the kth candidate hyperparameter is greater than the moving average and the performance evaluation index for the kth candidate hyperparameter is greater than one of the stored performance evaluation indexes, the n loss indexes The hyperparameter optimization method further includes the step of replacing the loss index corresponding to the lowest performance evaluation index with the loss index for the kth candidate hyperparameter and recalculating the moving average.
In claim 14,
The loss indicator is a hyperparameter optimization method wherein the loss index is a Nat (natural unit of information) value of the loss output during the learning process of the target model.
In claim 14,
The performance evaluation index includes one or more of accuracy, error rate, sensitivity, precision, specificity, and false positive rate of the target model. , Hyperparameter optimization method.
delete In claim 11,
The step of recommending one hyperparameter includes determining the performance of the target model for each first candidate hyperparameter and each second candidate hyperparameter derived from the Bayesian optimization algorithm in the process of learning the target model. A hyperparameter optimization method that performs evaluation and recommends one hyperparameter that outputs the most optimal performance evaluation index among each of the first candidate hyperparameters and each of the second candidate hyperparameters.