KR102458103B1 - Reinforcement learning apparatus and method for multiple classification - Google Patents

Abstract

Disclosed are a reinforcement learning device and method for multiple classification. The present invention has an advantage of increasing a classification accuracy by maximizing an expected reward in classification learning of an artificial intelligence model through reinforcement learning by using a policy gradient, and providing a high classification accuracy by improving a defense for an adversarial attack. The reinforcement learning device comprises: a policy part; and a policy loss function part.

Description

REINFORCEMENT LEARNING APPARATUS AND METHOD FOR MULTIPLE CLASSIFICATION

The present invention relates to a reinforcement learning apparatus and method for multiple classification, and more particularly, through reinforcement learning using a policy gradient, by maximizing the expected reward in classification learning of an artificial intelligence model, the classification accuracy is increased, and It relates to a reinforcement learning apparatus and method for multi-class classification with improved defense.

Reinforcement learning is a learning method that deals with an agent that interacts with the environment and achieves a goal. , and the agent selects actions sequentially as the time step passes, and receives a reward based on the action's impact on the environment.

1 is a block diagram showing the configuration of a reinforcement learning apparatus according to the prior art. As shown in FIG. 1, the agent 10 learns a method for determining an action through learning of a reinforcement learning model, and each Actions affect the next state, and the degree of success can be measured as a reward.

The environment 20 is all rules, such as actions that the agent 10 can take and rewards accordingly, states, actions, rewards, etc. are all components of the environment, and all predetermined things other than the agent 10 are the environment.

On the other hand, as social interest in deep learning technology increases in recent years, the technology is widely applied to various technology fields such as autonomous driving and biometrics, and two major disadvantages are emerging.

The first is the inherent uncertainty of a deep learning model, and the second is that compared to machine learning based on explicitly defined features (handcrafted feature), the trained deep learning model has a black-box and As such, the inference result is not explainable.

Adversarial attacks are being attempted by using the shortcomings of these deep learning models, which can deceive the image classification system by interfering with tasks such as object detection and image classification.

In particular, in multi-class image classification, hostile attack is to generate minimal hostile noise in a suitable image. This could be more effective.

However, image classification or multi-class image classification learning using reinforcement learning according to the prior art learns a policy based on a policy gradient.

To this end, the compensation function may be defined as follows.

where k = 1, ... , n,

Also, the policy loss function may be defined as follows.

Here, 'a' means a one-hot vector of a (action).

Also, the policy gradient for the above function can be expressed as follows.

here,

이다.

to be.

In this policy gradient, the right side is a cross entropy loss function used in the existing multiple classification, and the left side is a policy loss function.

는 크로스 엔트로피 손실 기울기의 역방향을 따라 정책의 파라미터를 업데이트하게 되어 학습이 잘 이루어지지 못하는 문제점이 있다.However, the log of the probability is multiplied by the reward (usually positive) if it is well classified, and by the penalty (usually negative) if it is not. In this case, the policy gradient is

, updates the policy parameters along the reverse direction of the cross-entropy loss gradient, so there is a problem in that learning is not performed well.

That is, the policy gradient updates the parameters of the policy according to the cross-entropy loss or the reverse of the cross-entropy loss, and the policy gradient updates the parameters in the direction of increasing accuracy or preventing erroneous operations. There is a problem in focusing on actions that are not.

In addition, supervised learning in reinforcement learning can change the softmax value corresponding to the correct answer by continuously presenting the correct answer to the policy action during the learning process, but the policy gradient is not the correct answer, but the direction that denies the behavior. In particular, there is a problem in that defense learning is not well performed against a hostile attack because the weight is changed.

Korean Patent Publication No. 10-2304661 (Title of Invention: Adversary training method independent of attack technology for strong hostile defense)

In order to solve this problem, the present invention is reinforcement learning for multi-classification that increases classification accuracy by maximizing expected rewards in classification learning of artificial intelligence models through reinforcement learning using policy gradients, and improves defense against hostile attacks. An object of the present invention is to provide an apparatus and method.

In order to achieve the above object, an embodiment of the present invention is a reinforcement learning apparatus for multiple classification, learning a method for determining an action through learning of a reinforcement learning model, and each action is performed in the next state ) and includes an agent 100 that measures the degree of success as a reward, wherein the agent predicts the correct answer of the classification data so that the expected reward is maximized through the policy in multi-class image classification policy department; and a policy loss function unit that learns the policy so that the policy unit predicts the correct answer with the highest probability of the expected reward using a policy gradient.

In addition, the policy loss function unit 120 according to the embodiment has the following formula

.

.

이고, C는 클래스(Class) 수이며, i= 1,‥, c, 임- 을 이용한 정책 그래디언트에 기반하여 정책을 학습하는 것을 특징으로 한다.- here,

, C is the number of classes, and it is characterized in that the policy is learned based on the policy gradient using i = 1, ..., c, im-.

가 낮아지도록 정책을 학습하는 것을 특징으로 한다.In addition, the policy loss function unit 120 according to the embodiment

It is characterized by learning the policy so that

In addition, the policy loss function unit according to the embodiment includes: a first loss function unit for performing learning based on the first policy loss function; and a second loss function unit that performs learning based on a second policy loss function when the learning of the first loss function unit is completed.

In addition, the first policy loss function according to the embodiment is

is available.

In addition, the second policy loss function according to the embodiment is

- Here, k = 1, ..., n, i = 1, ..., c, im- is characterized by using.

In addition, an embodiment of the present invention is a reinforcement learning method for multiple classification, a) the agent predicts the correct answer of the classification data so that the expected reward is maximized through the policy in the multi-class image classification, but the policy gradient (Policy) It may include the step of learning the policy so that the expected reward using the gradient) is predicted with the highest probability.

In addition, the agent according to the embodiment is

.

.

이고, C는 클래스(Class) 수이며, i= 1,‥, c, 임- 을 이용한 정책 그래디언트에 기반하여 정책을 학습하는 것을 특징으로 한다.- here,

, C is the number of classes, and it is characterized in that the policy is learned based on the policy gradient using i = 1, ..., c, im-.

가 낮아지도록 정책을 학습하는 것을 특징으로 한다.In addition, the agent according to the embodiment

It is characterized by learning the policy so that

In addition, step a) according to the above embodiment includes: a-1) the agent performing learning based on the first policy loss function; and a-2) when the learning based on the first policy loss function is completed, performing, by the agent, learning based on the second policy loss function;

In addition, the first policy loss function according to the embodiment is

It is characterized by using

In addition, the second policy loss function according to the embodiment is

- Here, k = 1, ..., n, i = 1, ..., c, im- is characterized by using.

The present invention has the advantage of increasing classification accuracy by maximizing the expected reward in classification learning of an artificial intelligence model through reinforcement learning using a policy gradient.

In addition, the present invention has the advantage of being able to provide high classification accuracy by improving the defense against an adversarial attack.

1 is a block diagram showing the configuration of a general reinforcement learning apparatus.
2 is a block diagram showing the configuration of a reinforcement learning apparatus for multiple classification according to an embodiment of the present invention.
3 is a block diagram illustrating a policy loss function of the reinforcement learning apparatus for multiple classification according to the embodiment of FIG. 2 .
4 is a flowchart illustrating a reinforcement learning method for multiple classification according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to preferred embodiments of the present invention and the accompanying drawings.

Prior to describing the specific contents for carrying out the present invention, it should be noted that components not directly related to the technical gist of the present invention are omitted within the scope of not disturbing the technical gist of the present invention.

In addition, the terms or words used in the present specification and claims have meanings and concepts consistent with the technical idea of the invention based on the principle that the inventor can define the concept of an appropriate term to best describe his invention. should be interpreted as

In this specification, the expression that a part "includes" a certain element does not exclude other elements, but means that other elements may be further included.

Also, terms such as “… unit”, “… group”, and “… module” mean a unit that processes at least one function or operation, which may be divided into hardware, software, or a combination of the two.

In addition, the term "at least one" is defined as a term including the singular and the plural, and even if the term at least one does not exist, each element may exist in the singular or plural, and may mean the singular or plural. will be self-evident.

In addition, that each component is provided in singular or plural may be changed according to an embodiment.

Hereinafter, a preferred embodiment of an apparatus and method for reinforcement learning for multiple classification according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

2 is a block diagram showing the configuration of a reinforcement learning apparatus for multiple classification according to an embodiment of the present invention, and FIG. 3 is a block diagram showing a policy loss function of the reinforcement learning apparatus for multiple classification according to the embodiment of FIG. it is do

2 and 3 , the reinforcement learning apparatus for multiple classification according to an embodiment of the present invention learns a method of determining an action through learning of a reinforcement learning model, and each action becomes the next state An agent 100 that affects the state and measures the degree of success as a reward, an action that the agent 100 can take, and an environment 200 that performs a reward accordingly. can

The agent 100 has a policy unit 110 that predicts the correct answer of the classification data so that the expected reward is maximized through a policy in multi-class image classification, and the policy unit 110 uses a policy gradient The reward may be configured to include a policy loss function unit 120 that learns the policy to predict the correct answer with the highest probability.

In addition, the agent 100 may be based on reinforcement learning to make a decision by learning an instance that is a real object having properties and properties defined in the same class with a label sampled from a data set.

The state(s) is randomly selected data from the same state space as the data set of supervised learning.

The behavior of the agent expressed as a _k (k=1, …N) is the same as the classification of supervised learning, and it is a value between 1 and N by choosing one of the N correct answers, and the correct answer (Lable) corresponds to the state. means a label, which is also a value between 1 and N, and is expressed as y _k (k=1, …N).

A reward is a function that rewards a positive value w _k1 when the action matches the correct answer, and a negative value -w _k2 (w _k2 is positive) when it does not.

The policy refers to the artificial intelligence neural network, and the final value is expressed as π(s) as a value that can predict the correct answer of the data with the highest probability through the softmax function.

The policy (π) predicts a behavior on the given data and a neural network, and the behavior can be taken from the label space according to the predicted probability for a sample x of a given datum.

The policy for predicting the correct answer in the policy unit 110 is an artificial neural network, and may be configured as a CNN-based model.

가 낮아지도록 정책을 학습하고, 손실은 잘못된 동작에 해당하는 클래스를 제외한 나머지 클래스에 대하여 균일한 분포가 제공되도록 할 수 있다.Policy loss function unit 120 of the policy loss function

The policy is learned so that , and the loss can be provided so that a uniform distribution is provided for the remaining classes except for the class corresponding to the erroneous operation.

In addition, the policy loss function unit 120 defines the compensation function as follows,

where k = 1, ... , n, is

The policy loss function can be defined as follows.

Here, 'a' means a one-hot vector of a (action).

As a result, the policy gradient can be expressed as

here,

이다.

to be.

는 크로스 엔트로피 손실(cross-entropy loss)의 기울기와 동일하고, 행동이 레이블 값과 다르면 정책 그래디언트

는 크로스 엔트로피 손실 기울기의 역방향을 따라 정책의 파라미터가 업데이트되도록 한다.On the other hand, if the action is equal to the label value, the policy gradient

is equal to the slope of the cross-entropy loss, and if the behavior is different from the label value, then the policy gradient

Lets the parameters of the policy be updated along the reverse direction of the cross-entropy loss gradient.

However, the policy loss function unit 120 updates the parameters of the policy when the policy gradient follows the cross entropy loss or in the reverse direction of the cross entropy loss, and updates the parameters in the direction that the policy gradient increases accuracy or prevents incorrect operation. A first loss function unit 121 may be included to warm up learning by loss to prevent it from happening.

The first loss function unit 121 is a first policy loss function, the following formula

.

.

이고, C는 클래스(Class) 수이며, i= 1,‥, c, 임- 을 이용한 정책 그래디언트에 기반하여 손실에 의한 학습을 워밍업하는 정책을 학습할 수 있다.- here,

, C is the number of classes, and based on the policy gradient using i = 1, ..., c, im-, a policy of warming up learning by loss can be learned.

를 하기식,In other words, the policy gradient

to the following formula,

We use the loss function replaced by

를 최소화하는 대신, intuitively

Instead of minimizing

를 최대화하는 것과 동일하다.

is equivalent to maximizing

In addition, the policy loss function unit 120 is a second loss function that performs learning based on the second policy loss function when the first loss function unit 121 completes the learning based on the first policy loss function according to a predetermined criterion. Through the unit 122; it is possible to proceed with learning again.

Here, the second policy loss function is

-here, k = 1, ..., n, i = 1, ..., c, im- may be used, and may be an existing loss function.

The following describes a reinforcement learning method for multiple classification according to an embodiment of the present invention.

4 is a flowchart illustrating a reinforcement learning method for multiple classification according to an embodiment of the present invention.

2 to 4, in the reinforcement learning method for multiple classification according to an embodiment of the present invention, the agent 100 corrects the classification data so that the expected reward is maximized through the policy in the multi-class image classification. , but learning based on the first policy loss function is performed so that the policy can be learned so that the expected reward using the policy gradient is predicted with the highest probability (S100).

In step S100, the agent 100 has the following formula,

.

.

이고, C는 클래스(Class) 수이며, i= 1,‥, c, 임- 을 이용한 정책 그래디언트에 기반하여 정책을 학습한다.- here,

, and C is the number of classes, and the policy is learned based on the policy gradient using i = 1, ..., c, im-.

가 낮아지도록 정책을 학습하고, 손실은 잘못된 동작에 해당하는 클래스를 제외한 나머지 클래스에 대하여 균일한 분포가 제공되도록 한다.In addition, in step S100, the agent 100

The policy is learned so that α is lowered, and the loss ensures that a uniform distribution is provided for the rest of the classes except for the class corresponding to the erroneous operation.

를 하기식,That is, in step S100, the agent 100 is a policy gradient

to the following formula,

We use the loss function replaced by

Subsequently, when the learning based on the first policy loss function in step S100 is completed, the agent 100 performs learning based on the second policy loss function ( S200 ).

In step S200, when the agent 100 completes the learning based on the first policy loss function according to a predetermined criterion, the following formula

-Here, learning can be performed using the second policy loss function using k = 1, ..., n, i = 1, ..., c, im-.

Table 1 shows the test accuracy for a series of hostile attacks when ε is 0.04.

Dataset CIFAR-10 Attacks clean FGSM PGD CrossEntropy 92.26 0.1074 0.20 Ours(λ=0.1) 92.40 0.5937 0.3927 Ours(λ=0.25) 92.43 0.6422 0.4444 Ours(λ=0.5) 92.48 0.6641 0.5062 Ours(λ=0.75) 92.17 0.6521 0.4478 Ours(λ=0.1.0) 92.27 0.6178 0.4078 Ours(λ=0.1.25) 91.85 0.5576 0.3202 Ours(λ=0.1.5) 91.95 0.5294 0.3186

의 절대값이며, ε은 공격 강도이고, FGSM(Fast Gradient Sign Methods)은 가장 간단하지만 효율적인 화이트박스 공격이다.Here, the value of λ is

is the absolute value of , ε is the attack strength, and FGSM (Fast Gradient Sign Methods) is the simplest but most efficient white box attack.

Table 2 shows the comparison between cross entropy and the existing policy function, and when the replaced loss function and the existing policy function are used.

Dataset CIRAR-10 Attacks Clean FGSM CrossEntropy with(3)(λ=1.0) 92.21 0.1877 (6)with (3)(λ=1.0) 92.27 0.6178 Use Only (3) (λ=1.0) 10.01 .0

의 절대값이다.Here, the value of λ is

is the absolute value of

In Table 2, it can be seen that the removal result for CIFAR-10 requires the use of a replacement loss function to maintain the effect of the original policy loss.

Therefore, through reinforcement learning using policy gradients, classification accuracy can be increased by maximizing the expected reward in classification learning of an AI model, and high classification accuracy can be provided by improving defense against adversarial attacks. there will be

As described above, although described with reference to preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

In addition, the reference numbers described in the claims of the present invention are only described for clarity and convenience of description, and are not limited thereto, and in the process of describing the embodiment, the thickness of the lines shown in the drawings or the size of components, etc. may be exaggerated for clarity and convenience of explanation.

In addition, the above-mentioned terms are terms defined in consideration of functions in the present invention, which may vary depending on the intention or custom of the user or operator, so the interpretation of these terms should be made based on the content throughout this specification. .

In addition, even if not explicitly shown or described, a person of ordinary skill in the art to which the present invention pertains can make various modifications including the technical idea according to the present invention from the description of the present invention. Obviously, this still falls within the scope of the present invention.

In addition, the above embodiments described with reference to the accompanying drawings have been described for the purpose of explaining the present invention, and the scope of the present invention is not limited to these embodiments.

100: Agent
110: Ministry of Policy
120: policy loss function part
121: first loss function unit
122: second loss function unit
200: Environment

Claims (6)

An agent 100 that learns how to determine an action through learning of a reinforcement learning model, each action affects the next state, and measures the degree of success as a reward. ,
The agent 100 includes: a policy unit 110 for predicting the correct answer of classification data so that an expected reward is maximized through a policy in multi-class image classification; and
and a policy loss function unit 120 for learning the policy so that the policy unit 110 predicts the correct answer with the highest probability of the expected reward using a policy gradient;
The policy loss function unit 120 has the following formula

.
- here,

, C is the number of classes, and i = 1, ..., c, Im- Reinforcement learning apparatus for multiple classification, characterized in that it learns the policy based on the policy gradient using. The method of claim 1,
The policy loss function unit 120 is

Reinforcement learning device for multi-classification, characterized in that learning the policy so that is lowered. The method of claim 1,
The policy loss function unit 120 includes a first loss function unit 121 for performing learning based on the first policy loss function; and
When the learning of the first loss function unit 121 is completed, a second loss function unit 122 that performs learning based on a second policy loss function; includes,
The first policy loss function is

use ,
The second policy loss function is

-here, k = 1, ..., n, i = 1, ..., c, im- Reinforcement learning apparatus for multiple classification, characterized in that using. In the reinforcement learning method for multiple classification performed by a reinforcement learning apparatus for multiple classification,
a) The agent 100 predicts the correct answer of the classification data so that the expected reward is maximized through the policy in multi-class image classification,
Including the step of learning the policy so that the expected reward using the policy gradient is predicted with the highest probability,
The agent 100 has the following formula

.
- here,

, where C is the number of classes, and a reinforcement learning method for multiple classification, characterized in that the policy is learned based on the policy gradient using i = 1, ..., c, im-. 5. The method of claim 4,
The agent 100

Reinforcement learning method for multiple classification, characterized in that learning the policy so that is lowered. 5. The method of claim 4,
The step a) includes: a-1) performing, by the agent 100, learning based on the first policy loss function; and
a-2) when the learning based on the first policy loss function is completed, performing, by the agent 100, learning based on the second policy loss function;
The first policy loss function is

use ,
The second policy loss function is

- Here, k = 1, ..., n, i = 1, ..., c, im- Reinforcement learning method for multiple classification, characterized in that using.

Images