KR101932835B1 - An apparatus for selecting action and method thereof, computer-readable storage medium - Google Patents

Abstract

There is provided a behavior determination method based on reinforcement learning based on Neural Fitted Q-Iteration (NFQ) performed by a processor. The method comprises the steps of training a neural network with a Q function having an input of a current state and an action as inputs and a Q value as an output based on a plurality of state transition sample data, Calculating a Q value according to a plurality of behaviors in a predetermined state based on the function, and determining a behavior having the largest Q value among the plurality of behaviors as a next behavior, Can determine the transition cost to the next state based on the length of the sequence of sample data to be learned. Therefore, the artificial neural network can converge more quickly, and a more accurate result value can be calculated.

Description

[0001] APPARATUS FOR SELECTING ACTION AND METHOD THEREOF, COMPUTER-READABLE STORAGE MEDIUM [0002]

The present invention relates to an apparatus and method for determining behavior, and more particularly, to an apparatus and method for determining behavior based on an artificial neural network through reinforcement learning.

NFQ (Neural Fitted Q-Iteration) is an algorithm based on Q-Learning. Existing Q - learning can only be used in an online learning environment, which is difficult to use. To solve this problem, several techniques for learning the optimal policy have been proposed, and NFQ is one of the most widely used techniques. This is a methodology very similar to the Function Approximation method of Online Reinforcement Learning. It does not store the Q - values of each state (state, action) in table form, It is a method to learn the approximate value of Q - value by using Artificial Neural Network. However, when learning an artificial neural network, if there is no transition cost in the learning target data, the transition cost should be manually determined using expert knowledge.

Specifically, in learning the artificial neural network in the conventional NFQ technique, if the learning target is the final state, reward of the final state is used for learning. Otherwise, the transition cost and the Q value of the next state If the sum was used for learning, but the transfer cost was not given, the transfer cost had to be declared manually. At this time, there is a problem that the convergence speed of the artificial neural network is slowed if the declaration is wrong.

Korean Patent Registration No. 1171054 ("Adaptive Learning Device and Method of Handheld Terminal Phasing Pattern ", Samsung Electronics Co., Ltd.)

It is an object of the present invention to solve the aforementioned problems by providing a method for determining a behavior based on NFQ-based reinforcement learning by setting a transition cost based on a compensation of a final state and a length of a sequence of learning data.

Another object of the present invention to solve the above problems is to provide a behavior decision apparatus based on NFQ-based reinforcement learning by setting a transition cost based on a compensation of a final state and a length of a sequence of learning data .

It should be understood, however, that the present invention is not limited to the above-described embodiments, but may be variously modified without departing from the spirit and scope of the invention.

According to another aspect of the present invention, there is provided a behavior determination method based on enhanced learning based on NFQ (Neural Fitted Q-Iteration) performed by a processor, Training the artificial neural network to a Q-value function having inputs of a current state and an action and outputting a Q value according to the behavior based on the data; Calculating a Q value according to a plurality of behaviors in a predetermined state based on the Q value function; And determining a behavior having a largest Q value among the plurality of behaviors as a next action, wherein the step of training includes a step of calculating a transition cost to a next state based on a length of a sequence of sample data to be learned, Can be determined.

According to an aspect of the present invention, the step of training may set a compensation value of the next state to the Q value when the next state is the final state.

According to an aspect, the training step may set the Q value based on the transition cost to the next state and the maximum Q value of the next state, if the next state is not the final state.

According to an aspect, the step of training may set the sum of the transition cost to the next state and the maximum Q value of the next discounted state to the Q value, when the next state is not the final state.

According to an aspect, the transition cost may be determined based on a length of a sequence of sample data to be learned and a final state compensation of the sequence.

According to an aspect of the present invention, the transition cost may be determined by subtracting 1 from a value obtained by dividing a final state compensation of a sequence of sample data as a learning target by a length of the sequence.

According to an aspect, the final state compensation may be a normalized value.

According to one aspect, the status can indicate activities performed to date.

According to one aspect, the behavior may indicate an activity that is currently being performed.

According to another aspect of the present invention, there is provided an apparatus for determining behavior based on NFQ (Neural Fitted Q-Iteration) based reinforcement learning, the apparatus including a processor, Training a Q value function in an artificial neural network based on a plurality of state transition sample data and having a current state and an action as inputs and a Q value according to the behavior as an output; Calculating a Q value according to a plurality of behaviors in a predetermined state based on the Q value function; And determining that the behavior with the largest Q value among the plurality of behaviors is the next behavior, the training being based on a transition cost to a next state based on a length of a sequence of sample data to be learned, Can be determined.

According to an aspect of the present invention, the training may set a next state Reward to the Q value when the next state is the final state.

According to one aspect, the training may set the Q value based on the transition cost to the next state and the maximum Q value of the next state, if the next state is not the final state.

According to one aspect, the training may set the sum of the transition cost to the next state and the maximum Q value of the next discounted state to the Q value, if the next state is not the final state.

According to an aspect, the transition cost may be determined based on a length of a sequence of sample data to be learned and a final state compensation of the sequence.

According to an aspect of the present invention, the transition cost may be determined by subtracting 1 from a value obtained by dividing a final state compensation of a sequence of sample data as a learning target by a length of the sequence.

According to an aspect, the final state compensation may be a normalized value.

According to one aspect, the status can indicate activities performed to date.

According to one aspect, the behavior may indicate an activity that is currently being performed.

According to another aspect of the present invention, there is provided a computer-readable storage medium for storing a program for causing a processor included in the computer to perform a behavior decision based on Neural Fitted Q-Iteration (NFQ) To train the artificial neural network with a Q-value function having a current state and an action as inputs and a Q value as an output based on a plurality of state transition sample data, ; And to calculate a Q value according to a plurality of behaviors in a predetermined state based on the Q value function; And a command to determine a behavior having a greatest Q value among the plurality of behaviors as a next action, wherein the training is a transition cost to a next state based on a length of a sequence of sample data to be learned Can be determined.

The disclosed technique may have the following effects. It is to be understood, however, that the scope of the disclosed technology is not to be construed as limited thereby, as it is not meant to imply that a particular embodiment should include all of the following effects or only the following effects.

According to the method and apparatus for determining behavior based on reinforcement learning based on NFQ (Neural Fitted Q-Iteration) according to an embodiment of the present invention, it is possible to reduce the transition cost based on the compensation of the final state and the length of the sequence of learning data It is possible to perform the behavior decision using the artificial neural network more efficiently.

Specifically, it is possible to achieve faster convergence in training the Q-value function in the artificial neural network based on the sample data to be learned, more accurate result can be calculated, A determination can be made.

1 is a flow chart of Q-learning.
2 is an illustration of an artificial neural network.
3 is a structure of an artificial neural network for NFQ (Neural Fitted Q-Iteration).
4 is a flowchart of a behavior determination method according to an embodiment of the present invention.
5 is a block diagram illustrating a configuration of a behavior determination apparatus according to another embodiment of the present invention.
Figure 6 shows a performance comparison of behavior decisions according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail.

It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and are to be interpreted in an ideal or overly formal sense unless explicitly defined in the present application Do not.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate the understanding of the present invention, the same reference numerals are used for the same constituent elements in the drawings and redundant explanations for the same constituent elements are omitted.

Reinforcement learning is a way to learn the best behavior in each state little by little by trial and error experience rather than by calculating the best behavior in the current state. The system learns its learning objectives in real time through interaction with the given environment and through interaction with the environment. The concept of learning is the basic basis of artificial intelligence theory, and the approach that allows the processor to perform such learning is machine learning.

The conventional method used in reinforcement learning is Q-learning, and it is simple to implement because it learns Q value that represents suitability of behavior without model information by constructing it as a table. Since then, Q-learning has been extended to the continuous state and behavior space domain to treat state space and behavior in a continuous state.

1 is a flow chart of Q-learning. Reinforcement learning does not determine optimal behavior in the current state through calculation, but learns optimal behavior in each state by experience based on multiple trial and error. Q-Learning, which is the most widely used of reinforcement learning, is a method of learning control policies that maximize compensation through interaction with the environment. As shown in FIG. 1, Q-learning is performed through successive interactions of an agent and an environment. In this paper, we propose a new approach to the learning process. In this paper, we propose a new learning environment for the learning environment. ), And the agent reselects the action to maximize future compensation.

In the existing Q - learning, learning was performed through discontinuous state and behavior using a lookup table to store the Q value. However, when the lookup table is used, a problem is solved when a pair of state and action, that is, a sample data to be learned, is small, and when the sample data is too large, There is a problem in that the Q value is directly approximated through the artificial neural network.

Artificial neural networks are artificial intelligence algorithms that model human brain or brain structure and function. Artificial neural networks have the ability to approximate a mathematical function that has a relationship between input and output, where input and output can be real numbers, and the approximating function can be a nonlinear function. 2 is an illustration of an artificial neural network. As shown in FIG. 2, a widely used multi-layer artificial neural network has one or more hidden layers between an input layer and an output layer. Input neurons and output neurons, the number of hidden layers, etc., can be different depending on the task to be solved through the artificial neural network or the type of function to be approximated.

Among the methods of approximating the above Q values, in particular, the NFQ (Neural Fitted Q-Iteration) algorithm is a batch reinforcement learning algorithm for approximating an optimal behavior selection policy. 3 is a structure of an artificial neural network for NFQ (Neural Fitted Q-Iteration). The optimal behavior selection policy may be to determine a sequence of at least some of the actions among a plurality of actions that may be performed to achieve a final goal. Here, the state can be represented by the actions that have been performed so far, and the action can express the action that is currently performed. Based on the sample data to be learned, it is possible to learn or train the Q value function in the artificial neural network, determine the behavior with the highest Q value in the current state based on the trained Q value function, By repeating the determination of behavior, you can determine the sequence of optimal behaviors for the final goal.

As shown in FIG. 3, the NFQ algorithm requires an appropriate representation of states, actions, and compensation of a given task to reach an optimal goal. (E.g., Input 1) and behavior (e.g., Input 2) as input to the artificial neural network, and may have a Q value as a target value. Typically, these attributes can be expressed from the task itself to be solved, otherwise it is required to be defined manually by an expert. However, manual definition is very difficult and expert knowledge may not be sufficient. If incorrect information is provided to the algorithm, the algorithm can operate incorrectly and the convergence speed of the artificial neural network can be reduced. In particular, since the transition cost between states is an immediate compensation for each step from one state to the next, there is a large effect on the convergence speed of the algorithm over other attributes.

In the present invention, a method for determining the transition cost for the NFQ algorithm is disclosed. NFQ can represent artificial neural networks that have state and behavior as inputs and approximate Q values in batch mode with Q values as target values. NFQ requires several updates to converge. However, if the compensation in the final state is known, the cost of each state transition on the path to the final target can be approximated, which allows the artificial neural network to converge faster and have better performance.

First of all, the artificial neural network has a merit that it can approximate functions with high accuracy and perform good generalization from few training samples. This capability can be used to express the Q-value function used in Q-learning. By expressing the Q value estimation function as the objective function of the artificial neural network, it is possible to allow the artificial neural network to train the Q value of each state. Then, by repeating the updating and training of the NFQ, the Q value of all states can be approximated. A major advantage is that, even if the new state is not included in the data set for training, it is readily possible to obtain an approximated Q value for that state. Thus, new policies can be created based on existing state transitions and feedback can be performed. The NFQ algorithm is as follows.

Here, NN The artificial neural network (Neural Network) to, I is the number of instances (instance) of the data set, s, a is the status (state), action (action) pair from the set of data, R (s, a, s ' ) Represents the transition cost (or transition compensation) from the current state s to the next state s' according to the action a in all possible sets of actions A.

As shown above, in training the artificial neural network in the normal NFQ, the target value is expressed by Equation 1 below.

를 적절히 설정하여 반영할 수 있다. That is, in setting the Q value as the target value according to the behavior a in the current state s, when the next state s 'is the final state, the final state compensation R (s') is set to the Q value, can learn the neural network. However, the next state s' this case is not the final state, the target value of the next state s in accordance with the action a in the current state s' by a transition costs R (s, a, s') and, Can be set to a sum of values obtained by appropriately discounting the maximum Q value according to the behavior a 'in the next state s'. If not the final state, the maximum Q value in the next state is a delayed compensation rather than an immediate compensation, so the discount factor

Can be appropriately set and reflected.

Here, as described above, when the transition cost does not appear in itself as a data set to be learned, it should be appropriately set by using expert knowledge. If the setting is difficult and erroneously set, the convergence speed of the artificial neural network is remarkably reduced And the accuracy of the Q value approximation can also be reduced.

According to an embodiment of the present invention, the transition cost is automatically determined based on the length of the sequence of the data samples to be learned, thereby improving the convergence speed of the artificial neural network and improving the accuracy of the approximation.

According to an embodiment of the present invention, the transition cost can be determined according to Equation (2) below.

는 샘플 데이터에 포함된 모든 시퀀스들 중 하나인, 현재 학습 대상인 시퀀스

의 최종 상태의 보상을,

은 시퀀스

의 길이를 나타낼 수 있다. here,

Which is one of all the sequences included in the sample data,

Lt; RTI ID = 0.0 >

Sequence

Can be expressed by the following formula

That is, when the next state s 'is the final state, it is possible to train the artificial neural network by setting the final state compensation r to the target value like the conventional NFQ algorithm, but the next state s' The transition cost to the next state can be calculated based on the length of the sequence of the sample data to be learned. More specifically, as shown in Equation (2), based on the compensation of the final state of the current learning target sequence and the length of the sequence, the average compensation of the respective states included in the current sequence can be determined. In addition, the state transition cost is always negative, and the cost is always larger. In this case, the state transition cost is always normalized. And the cost is small, the transition cost can be determined so that the absolute value is small.

Table 1 below shows exemplary sample data.

The behavior determination method according to an embodiment of the present invention can be used for educational process mining. As illustrated in Table 1, Final Grade can be used as a final reward for the training process, and multiple actions for it, such as Diagram, Study Exercise, and at least some of the Working in Texk Editors The sequences may be used as sample data. The sequences of behaviors performed by each student and the final scores of each student may be included in the sample data as a plurality of sequences. By determining the optimal behavior in states using a Q-value function that is learned based on a plurality of sample data, the behaviors and the order in which they are performed can be different for each student, the final best result is finally calculated You can determine the sequence of actions you can do.

4 is a flowchart of a behavior determination method according to an embodiment of the present invention. Hereinafter, a behavior determination method according to an embodiment of the present invention will be described with reference to FIG.

The behavior determination method according to an exemplary embodiment of the present invention is a behavior determination method based on NFQ (Neural Fitted Q-Iteration) based reinforcement learning, which is performed by a processor. As shown in FIG. 4, first, a Q-value function having a current state and an action as inputs and a Q value as an output based on a plurality of state transition sample data is output as an artificial neural network (Step 410).

As described above, in the training of the artificial neural network, in the case where the next state is the final state in the sample data to be learned, the reward of the next state can be set to the Q value. However, if the next state is not the final state, as shown in Equation (1), the Q value as the target value for training the artificial neural network can be determined based on the transition cost to the next state and the maximum Q value of the next state . More specifically, in order to reflect that the maximum Q value of the next state is not the immediate compensation but the delayed compensation, the sum of the transition cost to the next state and the maximum Q value of the next discounted state may be set to the Q value.

On the other hand, as described above, the determination of the transition cost has a significant effect on the convergence speed of the artificial neural network and the accuracy of the approximated Q value. According to one aspect, the transition cost to the next state can be determined based on the length of the sequence of sample data to be learned. The sample data to be learned may have a current state and behavior in that state, and may have a sequence containing the same. The Q value can be set based on the length of this current sequence and the final state compensation of the sequence. More specifically, as shown in Equation (2), the final state compensation of the sequence of the sample data to be learned can be determined by subtracting 1 from the value obtained by dividing by the length of the sequence. Here, the final state compensation may be a normalized value, and thus the determined Q value may always have a negative value.

Through the learning process as described above, the behavior decision method according to an embodiment of the present invention can converge the artificial neural network more quickly and calculate accurate result values.

Referring again to FIG. 4, after learning a Q-value function in the artificial neural network based on a plurality of sample data, a Q value according to a plurality of behaviors in a predetermined state can be calculated based on the Q-value function (Step 420). The state can represent the activities performed so far, and the behavior can represent the activity that is currently being performed. Therefore, in the state of the actions performed so far, each Q value when performing an action can be approximated based on the Q value function learned in the artificial neural network.

Thereafter, the action with the highest Q value among the plurality of actions may be determined as the next action (step 430). Thus, the optimal behavior policy can be determined by repeating the process of determining the action with the greatest Q value as the next action to the final state.

5 is a block diagram illustrating a configuration of a behavior determination apparatus according to another embodiment of the present invention. 5, the behavior determining apparatus 500 according to an exemplary embodiment of the present invention includes a processor 510 and an artificial neural network (hereinafter, referred to as " neural network " Neural Network, 520). Here, the artificial neural network 520 may be independently configured as a separate system from the behavior determining apparatus 500, or integrated into the behavior determining apparatus 500 and implemented in software.

Processor 510 may be configured to train a Q value function on an artificial neural network based on a plurality of state transition sample data, with a current state and an action as inputs and a Q value according to the behavior as an output, Calculating a Q value according to a plurality of behaviors in a predetermined state on the basis of the Q value function, and determining a behavior having the largest Q value among the plurality of behaviors as a next behavior have. The processor 510 may include a training unit 511 for training the artificial neural network, a calculation unit 513 for calculating the Q value, and a determination unit 515 for determining the next behavior, and the training unit 511, The calculating unit 513 and the determining unit 515 may be implemented on separate processors, and may be implemented in a processor that integrates at least some of them.

The concrete operation of the behavior determining apparatus 500 according to an embodiment of the present invention may be based on the behavior determining method according to the embodiment of the present invention described above.

Figure 6 shows a performance comparison of behavior decisions according to an embodiment of the present invention. As described above, a behavior determination method according to an embodiment of the present invention can be applied to process mining for learning. In an experimental example, records of activities for learning of each of 115 students, Was included in the target sample data. The state represents the number of behaviors for the learning performed at the current time, and the behavior represents the next activity for learning. The reward in the final state represents the student's performance according to the overall education process. The trained Q value function is used to learn NFQ by applying the behavior decision method according to an embodiment of the present invention and to generate a new training process sequence, and the artificial neural network trained in the given grade is used. As shown in FIG. 6, the sequence of the training process according to the modified NFQ method according to an embodiment of the present invention has a higher result compared with the original data and the conventional NFQ algorithm.

The above-described method of determining a behavior according to the present invention can be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording media storing data that can be decoded by a computer system. For example, there may be a ROM (Read Only Memory), a RAM (Random Access Memory), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device and the like. The computer-readable recording medium may also be distributed and executed in a computer system connected to a computer network and stored and executed as a code that can be read in a distributed manner.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention as defined by the appended claims. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.

500: Behavior determination device
510: Processor
511: Training Department
513:
515:
520: Artificial Neural Network

Claims (19)

1. A method for determining behavior based on reinforcement learning based on NFQ (Neural Fitted Q-Iteration), which is performed by a processor,
Training a Q value function in an artificial neural network based on a plurality of state transition sample data and having a current state and an action as inputs and a Q value according to the behavior as an output;
Calculating a Q value according to a plurality of behaviors in a predetermined state based on the Q value function; And
Determining a behavior having a largest Q value among the plurality of behaviors as a next behavior,
Wherein the training comprises determining a transition cost to a next state based on a length of a sequence of sample data to be learned,
The transition cost is determined based on a length of a sequence of sample data to be learned and a final state compensation of the sequence,
Wherein the transition cost is determined by subtracting 1 from a value obtained by dividing the final state compensation of the sequence of sample data to be learned by the length of the sequence.
The method according to claim 1,
Wherein the step of training includes setting a next state of compensation (Reward) to the Q value if the next state is a final state.
The method according to claim 1,
Wherein the training comprises setting the Q value based on a transition cost to the next state and a maximum Q value of the next state if the next state is not the final state.
The method according to claim 1,
Wherein the step of training includes setting a sum of the transition cost to the next state and the maximum Q value of the next discounted state to the Q value if the next state is not the final state .
delete delete The method according to claim 1,
Wherein the final state compensation is a normalized value.
The method according to claim 1,
Wherein the current state represents activities performed so far.
The method according to claim 1,
Wherein said action represents an activity to be performed at present,
An apparatus for behavior determination according to reinforcement learning based on NFQ (Neural Fitted Q-Iteration), the apparatus comprising a processor,
Training a Q value function in an artificial neural network based on a plurality of state transition sample data and having a current state and an action as inputs and a Q value according to the behavior as an output;
Calculating a Q value according to a plurality of behaviors in a predetermined state based on the Q value function; And
And to determine, as a next action, a behavior having the largest Q value among the plurality of behaviors,
Wherein the training comprises determining a transition cost to a next state based on a length of a sequence of sample data to be learned,
The transition cost is determined based on a length of a sequence of sample data to be learned and a final state compensation of the sequence,
Wherein the transition cost is determined by subtracting 1 from a value obtained by dividing a final state compensation of a sequence of sample data as a learning object by the length of the sequence.
11. The method of claim 10,
Wherein the training comprises setting a next state Reward to the Q value if the next state is a final state.
11. The method of claim 10,
Wherein the training comprises setting the Q value based on a transition cost to the next state and a maximum Q value of the next state if the next state is not the final state.
11. The method of claim 10,
Wherein the training comprises setting a sum of the transition cost to the next state and the maximum Q value of the next discounted state to the Q value if the next state is not the final state.
delete delete 11. The method of claim 10,
Wherein the final state compensation is a normalized value.
11. The method of claim 10,
Wherein the current state indicates activities performed so far.
11. The method of claim 10,
Wherein the action represents an activity to be performed at present.
A computer-readable storage medium having stored thereon a processor included in the computer for causing a processor to perform behavioral decisions based on Neural Fitted Q-Iteration (NFQ)
Instructions for training a Q value function in an artificial neural network based on a plurality of state transition sample data and having a current state and an action as inputs and a Q value according to the behavior as an output;
And to calculate a Q value according to a plurality of behaviors in a predetermined state based on the Q value function; And
Storing a command for causing a behavior having the largest Q value among the plurality of behaviors to be determined as a next action,
The instruction for training includes a command for determining a transition cost to a next state based on a length of a sequence of sample data to be learned,
The transition cost is determined based on a length of a sequence of sample data to be learned and a final state compensation of the sequence,
Wherein the transition cost is determined by subtracting 1 from a value obtained by dividing the final state compensation of the sequence of sample data to be learned by the length of the sequence.

Images