KR20210151644A - Apparatus and method for extracting deep learning models - Google Patents

Abstract

Disclosed are a device and method for extracting a deep learning model. According to an embodiment of the present invention, the device for extracting the deep learning model includes; one or more first artificial neural networks which are pre-trained to perform a specific function; N (N>=2) second artificial neural networks learning to perform the specific function based on the first artificial neural network; and a learning unit which trains the N second artificial neural networks. The learning unit calculates a total loss function based on the ground truth for a specific input value, the probability score of the first artificial neural network, and the probability score of each of the N second artificial neural networks, and the learning unit can train each of the N second artificial neural networks based on the total loss function. Therefore, it is possible to extract a small network model with better performance than a large network through mutual learning between small networks.

Description

Apparatus and method for extracting deep learning models

The disclosed embodiments relate to deep learning model extraction techniques.

Recently, research on knowledge distillation, which transfers previously learned knowledge of a large network to a small network that can be used in mobile terminals, etc., is being actively researched.

However, the presently disclosed technology has a problem in that it is difficult to exceed the accuracy of a large deep learning model when extracting a small deep learning model with the same type of network or heterogeneous network.

US Patent Publication No. US2015/0356461 (published on Dec. 10, 2015)

Disclosed embodiments are to provide a method and apparatus for extracting a deep learning model.

An apparatus for extracting a deep learning model according to an embodiment includes: one or more first artificial neural networks trained in advance to perform a specific function; N (N≥2) second artificial neural networks learning to perform a specific function based on the first artificial neural network; and a learning unit for learning N second artificial neural networks, wherein the learning unit includes a ground truth for a specific input value, a probability score of the first artificial neural network, and a probability of each of the N second artificial neural networks. A total loss function is calculated based on the score, and N second artificial neural networks may be trained based on the total loss function, respectively.

The number of layers included in the second artificial neural network is equal to or less than the number of layers included in the first artificial neural network, or the number of parameters included in the second artificial neural network is equal to the number of parameters included in the first artificial neural network or you can write

The learning unit, the first loss function generated based on the correct answer value for the n (1≤n≤N)-th second artificial neural network and the probability score of the n-th second artificial neural network, the probability score of the first artificial neural network, and the n-th The second loss function generated based on the probability score of the second artificial neural network, the probability score of the n-th second artificial neural network, and the probability score generated using the probability scores of all second artificial neural networks except for the n-th second artificial neural network It is possible to calculate a third loss function generated based on .

The learner may calculate a total loss function by summing the first loss function, the second loss function, and the third loss function calculated for each of the N second artificial neural networks.

The second loss function is a probability score generated based on a score having a maximum value among scores output from the last layer of each of one or more first artificial neural networks;

At least one of a probability score generated based on a probability score having a maximum value among the probability scores of each of the one or more first artificial neural networks, and a probability score generated by averaging the probability scores of each of the one or more first artificial neural networks and the nth It may be generated based on the probability score of the second artificial neural network.

The third loss function is the first generated based on the score of the second artificial neural network having the largest score among the scores output from the last layer of all the second artificial neural networks except for the n-th second artificial neural network. It may be a conditional probability of the second probability score of the nth second artificial neural network with respect to the probability score.

The third loss function is a second probability score generated based on the first probability score of the n-th second artificial neural network and the scores output from the last layer of all second artificial neural networks except for the n-th second artificial neural network. It may be generated based on a result of subtracting the entropy of the first probability score of the nth second artificial neural network from the cross entropy of .

The second probability score may be generated based on the score of the second artificial neural network having the largest score among the scores output from the last layer of all second artificial neural networks except for the n-th second artificial neural network. .

The learner may apply reverse kullback-leibler divergence to optimize the first probability score of the n-th second artificial neural network.

The correct answer value is a one-hot vector, and the probability score may be a probability vector generated by applying a softmax function to a score output from the last layer of the artificial neural network.

The N second artificial neural networks may be initialized with different initial weight values.

A deep learning model extraction method according to an aspect includes: receiving a probability score for a specific input from one or more first artificial neural networks previously trained to perform a specific function; Receiving a probability score for a specific input from N (N≥2) second artificial neural networks that learn to perform a specific function based on the first artificial neural network; and training N second artificial neural networks, wherein the training includes a ground truth for a specific input value, a probability score of the first artificial neural network, and each of the N second artificial neural networks. A total loss function is calculated based on the probability score of , and N second artificial neural networks can be trained based on the total loss function, respectively.

The number of layers included in the second artificial neural network is equal to or less than the number of layers included in the first artificial neural network, or the number of parameters included in the second artificial neural network is equal to the number of parameters included in the first artificial neural network or you can write

In the learning step, the first loss function generated based on the correct answer value for the n (1≤n≤N)-th second artificial neural network and the probability score of the n-th second artificial neural network, the probability score of the first artificial neural network, and The second loss function generated based on the probability score of the n-th second artificial neural network, the probability score of the n-th second artificial neural network, and the probability score of all second artificial neural networks except for the n-th second artificial neural network. A third loss function may be calculated based on the probability score.

The learning may include calculating a total loss function by summing the first loss function, the second loss function, and the third loss function calculated for each of the N second artificial neural networks.

The second loss function is a probability score generated based on a score having a maximum value among scores output from the last layer of each of the one or more first artificial neural networks, and a maximum value among the probability scores of each of the one or more first artificial neural networks. At least one of a probability score generated based on a probability score having .

The third loss function is the first generated based on the score of the second artificial neural network having the largest score among the scores output from the last layer of all the second artificial neural networks except for the n-th second artificial neural network. It may be a conditional probability of the second probability score of the nth second artificial neural network with respect to the probability score.

The third loss function is a second probability score generated based on the first probability score of the n-th second artificial neural network and the scores output from the last layer of all second artificial neural networks except for the n-th second artificial neural network. It may be generated based on a result of subtracting the entropy of the first probability score of the nth second artificial neural network from the cross entropy of .

The second probability score may be generated based on the score of the second artificial neural network having the largest score among the scores output from the last layer of all second artificial neural networks except for the n-th second artificial neural network. .

In the training step, reverse kullback-leibler divergence may be applied to optimize the first probability score of the n-th second artificial neural network.

The correct answer value is a one-hot vector, and the probability score may be a probability vector generated by applying a softmax function to a score output from the last layer of the artificial neural network.

The N second artificial neural networks may be initialized with different initial weight values.

According to the disclosed embodiments, a small network may be trained through a large network, and a small network model having better performance than a large network may be extracted through mutual learning between small networks.

1 is a block diagram of an apparatus for extracting a deep learning model according to an embodiment;
2 is a block diagram of an apparatus for extracting a deep learning model according to an embodiment;
3 is a flowchart illustrating a method for extracting a deep learning model according to an embodiment;
4 is a flowchart illustrating a method for extracting a deep learning model according to an embodiment;
5 is a block diagram illustrating and explaining a computing environment including a computing device according to an embodiment;

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to provide a comprehensive understanding of the methods, devices, and/or systems described herein. However, this is merely an example, and the present invention is not limited thereto.

In describing the embodiments of the present invention, if it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. And, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification. The terminology used in the detailed description is for the purpose of describing embodiments of the present invention only, and should in no way be limiting. Unless explicitly used otherwise, expressions in the singular include the meaning of the plural. In this description, expressions such as “comprising” or “comprising” are intended to indicate certain features, numbers, steps, acts, elements, some or a combination thereof, one or more other than those described. It should not be construed to exclude the presence or possibility of other features, numbers, steps, acts, elements, or any part or combination thereof.

1 is a block diagram of an apparatus for extracting a deep learning model according to an embodiment.

Referring to FIG. 1 , the deep learning model extraction apparatus 100 includes a first artificial neural network 110 previously trained to perform a specific function, N ( It may include N≥2) second artificial neural networks 121 , 123 , and 125 and the learning unit 130 for learning N second artificial neural networks 121 , 123 , 125 .

For convenience of explanation, each of the N (N≥2) second artificial neural networks 121 , 123 , and 125 may be represented as the second artificial neural network 120 . Accordingly, the embodiments for the second artificial neural network 120 may be interpreted as embodiments for each of the second artificial neural networks 121 , 123 , and 125 .

According to an example, the first artificial neural network 110 may be represented by a large network (or a teacher model), and the second artificial neural network 120 may be represented by a small network (or a student model).

For example, the number of layers included in the second artificial neural network 120 may be less than the number of layers included in the first artificial neural network 110 . For example, each of the second artificial neural networks 121 , 123 , and 125 may include three layers, and the first artificial neural network 110 may include four layers.

As another example, the number of parameters included in the second artificial neural network 120 may be less than the number of parameters included in the first artificial neural network 110 . For example, each of the second artificial neural networks 121 , 123 , and 125 may include 100 parameters, and the first artificial neural network 110 may include 150 parameters.

According to an embodiment, the second artificial neural networks 121 , 123 , and 125 may be initialized with different initial weight values. Accordingly, the second artificial neural networks 121 , 123 , and 125 may output different results for the same input data.

According to one embodiment, the learning unit 130 is a ground truth for a specific input value, a probability score of the first artificial neural network 110, and each of the N second artificial neural networks 120 . A total loss function is calculated based on the probability score, and N second artificial neural networks may be trained based on the total loss function, respectively.

According to an example, the learning unit 130 may calculate an error between the correct answer value and the output result of the second artificial neural network 120 . As an example, the error may be a loss function.

According to an example, the learning unit 130 may calculate a probability score that is a probability vector generated by applying a softmax function to a score output from the last layer of the second artificial neural network 121 . As an example, the probability score may be defined as in Equation (1).

[Equation 1]

Here, x is the input data, and y _n is the score of the n-th second artificial neural network with respect to the input data x. In addition, p _n is a probability score generated by applying the softmax function of Equation 2 to the score of the nth second artificial neural network.

[Equation 2]

Here, I is the number of classes to be classified using the second artificial neural network.

According to an embodiment, the learning unit 130 may generate a first loss function based on a correct answer value for any one of the second artificial neural networks and a probability score of the second artificial neural network.

According to an example, the first loss function may be a cross entropy loss between the probability score of the n-th second artificial neural network and the correct value. As an example, the cross entropy loss may be defined as follows.

[Equation 3]

Here, p denotes a correct answer value, q denotes a probability score of the nth second artificial neural network, and i denotes the number of a class to be classified using the second artificial neural network. As an example, the correct answer value P may be a one-hot vector. For example, P = (p ₁ , p ₂ , …, p _i , …, P _I ) = (0, 1, 0, …, 0).

According to an embodiment, the learner 130 may generate the second loss function based on the probability score of the first artificial neural network and the probability score of the n-th second artificial neural network.

According to an example, the second loss function may be a knowledge distillation loss between the probability score of the first artificial neural network for a specific input and the probability score of the n-th second artificial neural network. As an example, the knowledge distillation loss may be defined as follows.

[Equation 4]

One-hot can be approached when softmaxing the score of the first artificial neural network in knowledge distillation. T _KD is a temperature hyperparameter that performs a role of smoothing to have high entropy by transforming the probability score of the first artificial neural network approaching one-hot.

According to an embodiment, the learning unit 130 is configured to perform a third based on the probability score generated by using the probability score of the n-th second artificial neural network and the probability score of all second artificial neural networks except for the n-th second artificial neural network. You can create a loss function.

According to an example, the third loss function is a collection loss between the probability scores of the n-th second artificial neural network and the probability scores generated using the probability scores of all second artificial neural networks except for the n-th second artificial neural network. can be

According to an embodiment, the third loss function is based on the score of the second artificial neural network having the largest score among the scores output from the last layer of all second artificial neural networks except for the n-th second artificial neural network. may be a conditional probability of the second probability score of the n-th second artificial neural network with respect to the first probability score generated by

As an example, the collection loss, which is the third loss function, may be defined as follows.

[Equation 5]

는

에 대한

의 조건부 확률을 의미한다. 또한,

와

는 다음과 같이 정의된다. 이때, i는 N개의 제 2 인공 신경망 중 i 번째 제 2 인공 신경망을 의미한다.where L _KLD is a kullback-leibler divergence function,

Is

for

is the conditional probability of Also,

Wow

is defined as In this case, i means the i-th second artificial neural network among the N second artificial neural networks.

[Equation 6]

where T _KLD is the temperature hyperparameter.

According to an embodiment, the learner 130 may calculate the total loss function by summing the first loss function, the second loss function, and the third loss function calculated for each of the N second artificial neural networks. As an example, the total loss function may be defined as the following equation.

[Equation 7]

Here, L _n is the loss function of the n-th second artificial neural network, and may be defined as follows.

[Equation 8]

는 각 손실(loss)의 균형을 유지하기 위한 가중치이다.here,

is a weight for maintaining the balance of each loss.

According to an embodiment, the learning unit 130 calculates the gradient of the scalar field according to the loss when learning the second artificial neural network and back propagates it, in this case the n-th second with respect to _{L n .} Instead of training artificial neural networks individually, it is possible to train all N second artificial neural networks with _{L total.}

의

와

두 분포를 모두 최적화하는 반면, L_n을 이용하는 학습은

만을 최적화하게 된다. According to one embodiment as compared to the study of all the second artificial neural network by using the L _n L used as the _total of the n-th second learning neural network, learning using the _total L is

of

Wow

While optimizing both distributions, learning with _{L n}

only to be optimized.

2 is a block diagram of an apparatus for extracting a deep learning model according to an embodiment.

Referring to FIG. 2 , the deep learning model extraction apparatus 200 performs a specific function based on M (M≥2) first artificial neural networks 210 and the first artificial neural networks 210 that have been pre-trained to perform a specific function. may include N (N≥2) second artificial neural networks 220 that learn to perform , and a learning unit 230 that trains N second artificial neural networks 220 .

For convenience of explanation, each of the M (M≥2) first artificial neural networks 211 and 213 is referred to as the first artificial neural network 210, and N (N≥2) second artificial neural networks 221, 223, 225 ) may be represented by the second artificial neural network 220 . Accordingly, the embodiments of the first artificial neural network 210 may be interpreted as embodiments for each of the first artificial neural networks 211 and 213 , and the embodiments of the second artificial neural network 220 are the second artificial neural networks. (221, 223, 225) can be interpreted as an embodiment for each.

According to an example, the number of layers included in the second artificial neural network 220 may be equal to or less than the number of layers included in the first artificial neural network 210 . For example, the number of layers of the first artificial neural network 211 may be three, and the number of layers of the second artificial neural network 221 may also be three. As another example, the number of layers of the second artificial neural network 221 may be two.

According to an example, the number of parameters included in the second artificial neural network 220 may be equal to or less than the number of parameters included in the first artificial neural network 110 . For example, the number of parameters of the first artificial neural network 211 may be 100, and the number of parameters of the second artificial neural network 221 may also be 100. As another example, the number of parameters of the second artificial neural network 221 may be 80.

According to an embodiment, the second artificial neural networks 221 , 223 , and 225 may be initialized to different initial weight values. Accordingly, the second artificial neural networks 221 , 223 , and 225 may output different results with respect to the same input data.

According to an embodiment, the learning unit 230 is a ground truth for a specific input value, a probability score of each of the M first artificial neural networks, and a probability score of each of the N second artificial neural networks. A total loss function is calculated based on the total loss function, and N second artificial neural networks may be trained based on the total loss function, respectively.

According to an example, the learning unit 230 may calculate an error between the correct answer value and the output result of the second artificial neural network 220 . As an example, the error may be a loss function.

According to an example, the learning unit 230 may calculate a probability score that is a probability vector generated by applying a softmax function to a score output from the last layer of the second artificial neural network 221 . As an example, the probability score may be defined as in Equation 1 described with reference to FIG. 1 .

According to an embodiment, the learning unit 230 may generate a first loss function based on a correct answer value for the n (1≤n≤M)-th second artificial neural network and a probability score of the n-th second artificial neural network. have.

According to an example, the first loss function may be a cross entropy loss between the probability score of the n-th second artificial neural network and the correct value. As an example, the cross entropy loss may be defined as in Equation (3).

According to an embodiment, the learning unit 230 is configured to generate a second probability score based on a score output from the last layer of the M first artificial neural networks and a probability score of the n-th second artificial neural network. You can create a loss function.

According to an embodiment, the probability score generated based on the scores output from the last layer of the M first artificial neural networks is the maximum value among the scores output from the last layers of each of the M first artificial neural networks. It may be generated based on a score having . As an example, it may be defined as in the following equation.

[Equation 9]

According to an embodiment, the probability score generated based on the score output from the last layer of the M first artificial neural networks is based on the probability score having the maximum value among the probability scores of each of the M first artificial neural networks. may have been created. As an example, it may be defined as in the following equation.

[Equation 10]

According to an embodiment, the probability score generated based on the scores output from the last layer of the M first artificial neural networks may be generated by averaging the probability scores of each of the M first artificial neural networks. As an example, it may be defined as in the following equation.

[Equation 11]

,

및

중 적어도 하나와 n번째 제 2 인공 신경망의 확률 점수에 기초하여 생성될 수 있다.According to one embodiment, the second loss function is

,

and

It may be generated based on at least one of the following and the probability score of the n-th second artificial neural network.

According to an embodiment, the learning unit 230 is configured to perform the third based on the probability score generated by using the probability score of the n-th second artificial neural network and the probability score of all second artificial neural networks except for the n-th second artificial neural network. You can create a loss function.

According to an example, the third loss function is a collection loss between the probability scores of the n-th second artificial neural network and the probability scores generated using the probability scores of all second artificial neural networks except for the n-th second artificial neural network. can be

According to an embodiment, the third loss function is based on the score of the second artificial neural network having the largest score among the scores output from the last layer of all second artificial neural networks except for the n-th second artificial neural network. may be a conditional probability of the second probability score of the n-th second artificial neural network with respect to the first probability score generated by

As an example, the collection loss, which is the third loss function, may be defined as in Equation (5).

According to an embodiment, the third loss function is generated based on the first probability score of the n-th second artificial neural network and the scores output from the last layer of all second artificial neural networks except for the n-th second artificial neural network. It may be generated based on a result of subtracting the entropy of the first probability score of the n-th second artificial neural network from the cross entropy of the second probability score. As an example, the third loss function may be defined as follows.

[Equation 12]

where H(p) is the entropy of the probability score p. According to the above equation, it can be seen that when p is fixed, optimizing _{L CE with q is the same as optimizing L KLD .} At this time, since L _KLD is asymmetric, L _KLD (p,q) and L _KLD (q,p) have different values.

As an example, when loss L (p, q) is given in the optimization task, p is fixed and q is a distribution to be optimized. If q follows a one-hot distribution such as (0, 1, 0, ..., 0), it is common to optimize q because _{L KLD (p, q) becomes infinite unless p = q.} This may be a more reasonable way.

According to an embodiment, the learner 230 may apply reverse kullback-leibler divergence to optimize the first probability score of the n-th second artificial neural network.

According to Equation 12, it can be seen that entropy (H(p)) can be increased instead of decreasing _{L CE} in order to minimize kullback-leibler divergence, and entropy (H(p) ), a reverse kullback-leibler divergence can be applied. In this case, if p is the optimized distribution, L _KLD (q,p) is called forward kullback-leibler divergence, and L _KLD (p,q) is called backward kullback-leibler divergence (kullback-leibler divergence). leibler divergence).

According to an example, _{instead of updating q in L KLD} (p, q), the reverse L _KLD may be applied as a collection loss. For example, when entropy is small, p may be similar to a one-hot distribution, and p has information about one class. On the other hand, when entropy is large, p may have information about one or more classes, and through this, it is possible to learn using more information between the second artificial neural networks.

According to an embodiment, the second probability score is the score of the second artificial neural network having the largest score among the scores output from the last layer of all second artificial neural networks except for the nth second artificial neural network. It can be created based on

According to an embodiment, the learning unit 230 may calculate the total loss function by summing the first loss function, the second loss function, and the third loss function calculated for each of the N second artificial neural networks. As an example, the total loss function may be defined as in Equation (7).

According to an embodiment, the correct answer value is a one-hot vector, and the probability score may be a probability vector generated by applying a softmax function to a score output from the last layer of the artificial neural network.

According to an embodiment, the N second artificial neural networks may be initialized with different initial weight values.

3 is a flowchart illustrating a method for extracting a deep learning model according to an embodiment.

Referring to FIG. 3 , the apparatus for extracting a deep learning model may receive a probability score for a specific input from the first artificial neural network ( 310 ). As an example, the first artificial neural network may be an artificial neural network that has been previously trained to perform a specific function.

According to an embodiment, the apparatus for extracting a deep learning model may receive a probability score for a specific input from N (N≥2) second artificial neural networks ( 320 ). As an example, the second artificial neural network may be an artificial neural network that learns to perform a specific function based on the first artificial neural network.

According to an example, the first artificial neural network 110 may be represented by a large network (or a teacher model), and the second artificial neural network 120 may be represented by a small network (or a student model).

According to an embodiment, the apparatus for extracting a deep learning model may train N second artificial neural networks ( 330 ).

According to an embodiment, the deep learning model extraction device is a total loss based on a ground truth for a specific input value, a probability score of the first artificial neural network, and a probability score of each of the N second artificial neural networks. A total loss function is calculated, and N second artificial neural networks may be trained on the basis of the total loss function, respectively.

According to an example, the apparatus for extracting a deep learning model may calculate an error between a correct answer value and an output result of the second artificial neural network. As an example, the error may be a loss function.

According to an example, the apparatus for extracting a deep learning model may calculate a probability score that is a probability vector generated by applying a softmax function to a score output from the last layer of the second artificial neural network. As an example, the probability score may be defined as in Equation 1 described above.

According to an embodiment, the apparatus for extracting a deep learning model may generate a first loss function based on a correct answer value for any one of the second artificial neural networks and a probability score of the second artificial neural network.

According to an example, the first loss function may be a cross entropy loss between the probability score of the n-th second artificial neural network and the correct value. As an example, the cross entropy loss may be defined as in Equation (3).

According to an embodiment, the apparatus for extracting a deep learning model may generate a second loss function based on the probability score of the first artificial neural network and the probability score of the n-th second artificial neural network.

According to an example, the second loss function may be a knowledge distillation loss between the probability score of the first artificial neural network for a specific input and the probability score of the n-th second artificial neural network. As an example, the knowledge distillation loss may be defined as Equation (4).

According to one embodiment, the deep learning model extraction apparatus is based on a probability score generated using the probability score of the n-th second artificial neural network and the probability scores of all second artificial neural networks except for the n-th second artificial neural network. You can create a loss function.

According to an example, the third loss function is a collection loss between the probability scores of the n-th second artificial neural network and the probability scores generated using the probability scores of all second artificial neural networks except for the n-th second artificial neural network. can be

According to an embodiment, the third loss function is based on the score of the second artificial neural network having the largest score among the scores output from the last layer of all second artificial neural networks except for the n-th second artificial neural network. may be a conditional probability of the second probability score of the n-th second artificial neural network with respect to the first probability score generated by

As an example, the collection loss, which is the third loss function, may be defined as in Equation (5).

According to an embodiment, the apparatus for extracting a deep learning model may calculate a total loss function by summing the first loss function, the second loss function, and the third loss function calculated for each of the N second artificial neural networks. As an example, the total loss function may be defined as in Equation (7).

4 is a flowchart illustrating a method for extracting a deep learning model according to an embodiment.

Referring to FIG. 4 , the apparatus for extracting a deep learning model may receive a probability score for a specific input from M (M≥2) first artificial neural networks ( 410 ). As an example, the first artificial neural network may be an artificial neural network that has been previously trained to perform a specific function.

According to an embodiment, the apparatus for extracting a deep learning model may receive a probability score for a specific input from N (N≥2) second artificial neural networks ( 420 ). As an example, the second artificial neural network may be an artificial neural network that learns to perform a specific function based on the first artificial neural network.

According to an example, the first artificial neural network 110 may be represented by a large network (or a teacher model), and the second artificial neural network 120 may be represented by a small network (or a student model).

According to an embodiment, N second artificial neural networks may be trained ( 430 ).

According to an embodiment, the deep learning model extraction apparatus is a ground truth for a specific input value, a probability score of each of the M first artificial neural networks, and a probability score of each of the N second artificial neural networks. A total loss function is calculated based on the total loss function, and N second artificial neural networks may be trained based on the total loss function, respectively.

According to an example, the apparatus for extracting a deep learning model may calculate an error between a correct answer value and an output result of the second artificial neural network. As an example, the error may be a loss function.

According to an example, the apparatus for extracting a deep learning model may calculate a probability score that is a probability vector generated by applying a softmax function to a score output from the last layer of the second artificial neural network. As an example, the probability score may be defined as in Equation 1 described with reference to FIG. 1 .

According to an embodiment, the deep learning model extraction apparatus may generate a first loss function based on the correct answer value for the n (1≤n≤M)-th second artificial neural network and the probability score of the n-th second artificial neural network. have.

According to an example, the first loss function may be a cross entropy loss between the probability score of the n-th second artificial neural network and the correct value. As an example, the cross entropy loss may be defined as in Equation (3).

According to an embodiment, the deep learning model extraction apparatus is a second artificial neural network based on a probability score generated based on a score output from the last layer of the M first artificial neural networks and a probability score of the n-th second artificial neural network. You can create a loss function.

According to an embodiment, the probability score generated based on the scores output from the last layer of the M first artificial neural networks is the maximum value among the scores output from the last layers of each of the M first artificial neural networks. It may be generated based on a score having . As an example, a probability score generated based on a score output from the last layer of the first artificial neural network may be defined as in Equation (9).

According to an embodiment, the probability score generated based on the score output from the last layer of the M first artificial neural networks is based on the probability score having the maximum value among the probability scores of each of the M first artificial neural networks. may have been created. As an example, a probability score generated based on a score output from the last layer of the M first artificial neural networks may be defined as in Equation 10.

According to an embodiment, the probability score generated based on the scores output from the last layer of the M first artificial neural networks may be generated by averaging the probability scores of each of the M first artificial neural networks. As an example, it may be defined as in Equation 11.

According to one embodiment, the deep learning model extraction apparatus is based on a probability score generated using the probability score of the n-th second artificial neural network and the probability scores of all second artificial neural networks except for the n-th second artificial neural network. You can create a loss function.

According to an example, the third loss function is a collection loss between the probability scores of the n-th second artificial neural network and the probability scores generated using the probability scores of all second artificial neural networks except for the n-th second artificial neural network. can be

According to an embodiment, the third loss function is based on the score of the second artificial neural network having the largest score among the scores output from the last layer of all second artificial neural networks except for the n-th second artificial neural network. may be a conditional probability of the second probability score of the n-th second artificial neural network with respect to the first probability score generated by

As an example, the collection loss, which is the third loss function, may be defined as in Equation (5).

According to an embodiment, the third loss function is generated based on the first probability score of the n-th second artificial neural network and the scores output from the last layer of all second artificial neural networks except for the n-th second artificial neural network. It may be generated based on a result of subtracting the entropy of the first probability score of the n-th second artificial neural network from the cross entropy of the second probability score. As an example, the third loss function may be defined as in Equation (12).

According to an embodiment, the apparatus for extracting a deep learning model may apply reverse kullback-leibler divergence to optimize the first probability score of the n-th second artificial neural network.

According to Equation 12, it can be seen that entropy (H(p)) can be increased instead of decreasing _{L CE} in order to minimize kullback-leibler divergence, and entropy (H(p) ), a reverse kullback-leibler divergence can be applied. In this case, if p is the optimized distribution, L _KLD (q,p) is called forward kullback-leibler divergence, and L _KLD (p,q) is called backward kullback-leibler divergence (kullback-leibler divergence). leibler divergence).

According to an example, _{instead of updating q in L KLD} (p, q), the reverse L _KLD may be applied as a collection loss. For example, when entropy is small, p may be similar to a one-hot distribution, and p has information about one class. On the other hand, when entropy is large, p may have information about one or more classes, and through this, it is possible to learn using more information between the second artificial neural networks.

According to an embodiment, the second probability score is the score of the second artificial neural network having the largest score among the scores output from the last layer of all second artificial neural networks except for the nth second artificial neural network. It can be created based on

According to an embodiment, the apparatus for extracting a deep learning model may calculate a total loss function by summing the first loss function, the second loss function, and the third loss function calculated for each of the N second artificial neural networks. As an example, the total loss function may be defined as in Equation (7).

5 is a block diagram illustrating and explaining a computing environment including a computing device according to an embodiment.

In the illustrated embodiment, each component may have different functions and capabilities other than those described below, and may include additional components in addition to those described below.

The illustrated computing environment 10 includes a computing device 12 . In one embodiment, the computing device 12 may be one or more components included in the deep learning model extraction device 120 . Computing device 12 includes at least one processor 14 , computer readable storage medium 16 , and communication bus 18 . The processor 14 may cause the computing device 12 to operate in accordance with the exemplary embodiments discussed above. For example, the processor 14 may execute one or more programs stored in the computer-readable storage medium 16 . The one or more programs may include one or more computer-executable instructions that, when executed by the processor 14, configure the computing device 12 to perform operations in accordance with the exemplary embodiment. can be

Computer-readable storage medium 16 is configured to store computer-executable instructions or program code, program data, and/or other suitable form of information. The program 20 stored in the computer readable storage medium 16 includes a set of instructions executable by the processor 14 . In one embodiment, computer-readable storage medium 16 includes memory (volatile memory, such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash It may be memory devices, other forms of storage medium accessed by computing device 12 and capable of storing desired information, or a suitable combination thereof.

Communication bus 18 interconnects various other components of computing device 12 , including processor 14 and computer readable storage medium 16 .

Computing device 12 may also include one or more input/output interfaces 22 and one or more network communication interfaces 26 that provide interfaces for one or more input/output devices 24 . The input/output interface 22 and the network communication interface 26 are coupled to the communication bus 18 . Input/output device 24 may be coupled to other components of computing device 12 via input/output interface 22 . Exemplary input/output device 24 may include a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices, and/or imaging devices. input devices, and/or output devices such as display devices, printers, speakers and/or network cards. The exemplary input/output device 24 may be included in the computing device 12 as a component constituting the computing device 12 , and may be connected to the computing device 12 as a separate device distinct from the computing device 12 . may be

Although the present invention has been described in detail through representative embodiments above, those of ordinary skill in the art to which the present invention pertains can make various modifications to the above-described embodiments without departing from the scope of the present invention. will understand Therefore, the scope of the present invention should not be limited to the described embodiments, and should be defined by the claims described below as well as the claims and equivalents.

10: Computing Environment
12: computing device
14: Processor
16: computer readable storage medium
18: communication bus
20: Program
22: input/output interface
24: input/output device
26: network communication interface
100: deep learning model extraction device
110: first artificial neural network
120, 121, 123, 125: second artificial neural network
130: study unit
200: deep learning model extraction device
210, 211, 213: first artificial neural network
220, 221, 223, 225: second artificial neural network
230: study unit

Claims (22)

one or more first artificial neural networks pre-trained to perform a specific function;
N (N≥2) second artificial neural networks that learn to perform the specific function based on the first artificial neural network; and
It includes a learning unit for learning the N second artificial neural networks,
the learning unit
calculating a total loss function based on a ground truth for a specific input value, a probability score of a first artificial neural network, and a probability score of each of the N second artificial neural networks, wherein A deep learning model extraction apparatus for training each of the N second artificial neural networks based on a total loss function.
The method according to claim 1,
The number of layers included in the second artificial neural network is equal to or less than the number of layers included in the first artificial neural network, or
The number of parameters included in the second artificial neural network is equal to or less than the number of parameters included in the first artificial neural network, a deep learning model extraction apparatus.
The method according to claim 1,
The learning unit,
For the n (1≤n≤N)-th second artificial neural network
A first loss function generated based on the correct value and the probability score of the n-th second artificial neural network;
a second loss function generated based on the probability score of the first artificial neural network and the probability score of the n-th second artificial neural network; and
Deep learning for calculating a third loss function generated based on the probability score generated using the probability score of the n-th second artificial neural network and the probability score of all second artificial neural networks except for the n-th second artificial neural network model extraction device.
4. The method according to claim 3,
The learning unit,
A deep learning model extraction apparatus for calculating a total loss function by summing the first loss function, the second loss function, and the third loss function calculated for each of the N second artificial neural networks.
4. The method according to claim 3,
The second loss function is
a probability score generated based on a score having a maximum value among scores output from the last layer of each of the one or more first artificial neural networks;
a probability score generated based on a probability score having a maximum value among the probability scores of each of the one or more first artificial neural networks; and
The apparatus for extracting a deep learning model, which is generated based on at least one of the probability scores generated by averaging the probability scores of each of the one or more first artificial neural networks and the probability scores of the n-th second artificial neural network.
4. The method according to claim 3,
The third loss function is
The first probability score generated based on the score of the second artificial neural network having the largest score among the scores output from the last layer of all second artificial neural networks except for the n-th second artificial neural network. A conditional probability of the second probability score of the n-th second artificial neural network, a deep learning model extraction apparatus.
4. The method according to claim 3,
The third loss function is
Cross entropy ( Cross entropy) is generated based on the result of subtracting the entropy of the first probability score of the n-th second artificial neural network, a deep learning model extraction apparatus.
8. The method of claim 7,
The second probability score is,
A deep learning model extraction apparatus generated based on the score of the second artificial neural network having the largest score among the scores output from the last layer of all second artificial neural networks except for the n-th second artificial neural network.
8. The method of claim 7,
The learning unit,
In order to optimize the first probability score of the n-th second artificial neural network, a reverse kullback-leibler divergence is applied, a deep learning model extraction apparatus.
The method according to claim 1,
The correct value is a one-hot vector, and the probability score is a probability vector generated by applying a softmax function to a score output from the last layer of an artificial neural network.
The method according to claim 1,
The N second artificial neural networks are initialized to different initial weight values, a deep learning model extraction apparatus.
Receiving a probability score for a specific input from one or more first artificial neural networks that have been previously trained to perform a specific function;
receiving a probability score for a specific input from N (N≥2) second artificial neural networks that learn to perform the specific function based on the first artificial neural network; and
Including the step of learning the N second artificial neural networks,
The learning step
calculating a total loss function based on a ground truth for a specific input value, a probability score of a first artificial neural network, and a probability score of each of the N second artificial neural networks, wherein A deep learning model extraction method for training each of the N second artificial neural networks based on a total loss function.
13. The method of claim 12,
The number of layers included in the second artificial neural network is equal to or less than the number of layers included in the first artificial neural network, or
The number of parameters included in the second artificial neural network is equal to or less than the number of parameters included in the first artificial neural network, a deep learning model extraction method.
13. The method of claim 12,
The learning step is
For the n (1≤n≤N)-th second artificial neural network
A first loss function generated based on the correct value and the probability score of the n-th second artificial neural network;
a second loss function generated based on the probability score of the first artificial neural network and the probability score of the n-th second artificial neural network; and
Deep learning for calculating a third loss function generated based on the probability score generated using the probability score of the n-th second artificial neural network and the probability score of all second artificial neural networks except for the n-th second artificial neural network How to extract the model.
15. The method of claim 14,
The learning step is
A deep learning model extraction method for calculating a total loss function by summing the first loss function, the second loss function, and the third loss function calculated for each of the N second artificial neural networks.
15. The method of claim 14,
The second loss function is
a probability score generated based on a score having a maximum value among scores output from the last layer of each of the one or more first artificial neural networks;
a probability score generated based on a probability score having a maximum value among the probability scores of each of the one or more first artificial neural networks; and
The method for extracting a deep learning model, which is generated based on at least one of the probability scores generated by averaging the probability scores of each of the one or more first artificial neural networks and the probability scores of the n-th second artificial neural network.
15. The method of claim 14,
The third loss function is
The first probability score generated based on the score of the second artificial neural network having the largest score among the scores output from the last layer of all second artificial neural networks except for the n-th second artificial neural network. A method for extracting a deep learning model, which is the conditional probability of the second probability score of the nth second artificial neural network.
15. The method of claim 14,
The third loss function is
Cross entropy ( Cross entropy) is generated based on the result of subtracting the entropy of the first probability score of the n-th second artificial neural network, a deep learning model extraction method.
19. The method of claim 18,
The second probability score is,
A method for extracting a deep learning model, generated based on the score of the second artificial neural network having the largest score among the scores output from the last layer of all second artificial neural networks except for the n-th second artificial neural network.
19. The method of claim 18,
The learning step is
In order to optimize the first probability score of the n-th second artificial neural network, reverse kullback-leibler divergence is applied, a deep learning model extraction method.
15. The method of claim 14,
The correct value is a one-hot vector, and the probability score is a probability vector generated by applying a softmax function to a score output from the last layer of an artificial neural network.
15. The method of claim 14,
The N second artificial neural networks are each initialized to different initial weight values, a deep learning model extraction method.

Images