JP2019079227A - State transition rule acquisition device, action selection learning device, action selection device, state transition rule acquisition method, action selection method, and program - Google Patents

Abstract

To enable the acquisition of a state and a state transition rule for selecting action even in environment in which the state and the state transition rule are not clear.SOLUTION: A state acquisition part 210 acquires a state of an environment after action when the selected action is performed, a reward calculation part 220 calculates a reward when the action is performed in the state on the basis of the acquired state and the selected action, a parameter update part 240 updates a parameter of a model for selecting the action with the state as an input on the basis of the selected action and the reward, an action selection part 270 selects the action with a state after the action as an input by using the model, and repeats acquisition, calculation, update and selection until a repetition end condition is satisfied, and the state acquisition part 210 compares the acquired state with a set of states which have already been acquired, adds the acquired state to the set of states if it is a new state, and acquires a state transition rule on the basis of the set of states.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a state transition rule acquisition device, an action selection learning device, an action selection device, a state transition rule acquisition method, an action selection method, and a program, and in particular, a state transition rule acquisition device for selecting an action in a state The present invention relates to a selective learning device, an action selection device, a state transition rule acquisition method, an action selection method, and a program.

  Decision support systems that support human decision making include a wide range of practical systems in various fields, such as medical diagnosis, bank loan decisions, and business decisions by companies. Such strengthening of decision support systems is considered to be one of the important issues in artificial intelligence research that has been rapidly developing in recent years.

  Although there is no fixed form in the configuration of the decision support system, here we describe an event as a single state, and consider a state transition model that transitions from the current state to the next state by taking a certain action as one methodology . We then assume a system that uses the strategy of presenting the action to be taken in the current state so that the final state with the highest expected gain can be reached. In other words, the decision support system described here is a system that automatically selects and presents the action necessary to reach the best final state from the current state.

  In general, in a decision support system, the degree of difficulty of a problem changes depending on the number of selectable actions and the total number of possible states. If it is possible to obtain all the information necessary to select an action as information on the current state, and if the transition of the state to the action is crucial (no uncertainty factor), the calculation time is infinite. It is likely that the system can automatically calculate the best behavior in the current state if it is allowed to

  However, in reality, there is almost no situation where all information on the current state can be obtained. Also, the transition of the state to the action is not necessarily unique, and in most cases the next state is determined probabilistically. In other words, it is usually forced to select the optimal action while considering various uncertain factors.

  In the recent development of artificial intelligence technology, in a closed environment such as poker "If you decide your own action (hands), the information of the other party's arrangement is unknown" and "The other party to your own action In an environment where the information is “indeterminate” is incomplete and indeterminate, a methodology has been devised which can construct an agent with strength exceeding human professional (for example, Non-Patent Document 1). Here, "closed environment" means that the types of possible states in the environment do not increase or decrease with time, and are immutable and can be enumerated. A concrete method is to search for the transition from the current state to the future state in the uncertain incomplete information environment, and to reach the state with the least failure (no regret) state. It uses a strategy to take. In such a closed environment, and in an environment with a relatively small number of states, even if the information needed to determine the behavior is incomplete and uncertain, the best or near-best results It is becoming possible to automatically select the action to get

Regret minimization in games within complete information. In JC Platt, D. Koller, Y. Singer, and ST Roweis, editors, Advances in Neural Information Processing Systems 20, pages 1729 {1736 {2008. 2008} Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. .

  In the above-mentioned Non-Patent Document 1, the system is built on the assumption that the environment (game rules), the state, and the state transition rule are given and known in advance. If you know the states and state transition rules, it is possible to use it to simulate the winning percentage and gain, and a huge amount of simulation calculation gives guidance on what action should be taken in all states. be able to.

  However, in real-world problems, it is difficult to obtain transition rules as to how the state transitions when the state definition and certain actions are taken, and it is almost impossible to write out all. In real problems, transition rules of states are generally unknown, and the number of types of states is also enormous. That is, since the state transition rules and the definitions of the states can not be given to the system in advance, there is a problem that Non-Patent Document 1 or the like, which is a promising methodology currently in closed environment, can not be applied to a practical application system.

  The present invention has been made in view of this problem, and a state transition rule acquisition device capable of acquiring a state or state transition rule for selecting an action even in an environment where the state or state transition rule is unknown. An object of the present invention is to provide an action selection learning device, a state transition rule acquiring method, and a program.

  Another object of the present invention is to provide an action selection device, an action selection method, and a program capable of selecting an appropriate action even in an environment where the state or state transition rule is unknown.

  A state transition rule acquisition device according to the present invention is a state acquisition unit for acquiring a state after an action when a selected action is performed, and the state based on the acquired state and the selected action. A model for selecting an action based on the selected action and the reward calculated by the reward calculation unit, which calculates a reward when the action is performed, and selecting the action A parameter updating unit for updating the parameters of the parameter, an action selecting unit for selecting an action using the model after the state after the action is input, and the condition acquiring unit until a predetermined iteration end condition is satisfied. Acquisition, calculation by the reward calculation unit, update by the parameter update unit, and an end determination unit for repeating selection by the action selection unit, and the state acquisition unit is in the acquired state, Compared to the set of the acquired state, if the new state, a state in which the acquired and added to the set of the state, based on a set of the state, and obtains the state transition rule.

  In the state transition rule acquiring method according to the present invention, the state acquiring unit acquires the state of the environment after the action when the selected action is performed, and the reward calculating unit determines the acquired state. Calculating a reward when the action is performed in the state based on the selected action, and the parameter updating unit is based on the selected action and the reward calculated by the reward calculation unit Step of updating parameters of a model for selecting an action based on the state, and selecting an action using the model after the action selecting unit inputs a state after the action; Acquisition by the state acquisition unit, calculation by the reward calculation unit, update by the parameter update unit, and update by the action selection unit until an end determination unit satisfies a predetermined iteration end condition Repeating the selection step, and the step of acquiring by the state acquisition unit may compare the acquired state with the acquired state if the acquired state is a new state as compared to the set of already acquired states. Add to the set of, and acquire state transition rules based on the set of states.

  According to the state transition rule acquisition device and the state transition rule acquisition method of the present invention, the state acquisition unit acquires the state of the environment after the action when the input action is performed, and the reward calculation unit acquires the state. Based on the state and the selected action, the reward for performing the action in the state is calculated, and the parameter updating unit inputs the state based on the selected action and the reward calculated by the reward calculation unit. And update the parameters of the model to select the action.

  Then, the action selecting unit takes the state after the action as an input, selects the action using the model, and the end determining unit acquires the reward calculation unit by the state acquiring unit until it satisfies the predetermined iteration end condition. The state acquisition unit repeats the calculation by the parameter update unit, the update by the parameter update unit, and the selection by the action selection unit, and if the acquired state is a new state as compared to the set of acquired states, the acquired state is , Add to the set of states, and acquire state transition rules based on the set of states.

  In this way, the state of the environment after the action when the input action is performed is acquired until the predetermined repetitive end condition is satisfied, and the action in the state is performed based on the acquired state and the selected action. Calculate the reward at the time of performing, and based on the selected action and the reward calculated by the reward calculation unit, input the state, update the parameters of the model for selecting the action, and the state after the action Using the model as input, repeat selecting the action, add the acquired state to the set of states if the acquired state is new compared to the set of acquired states By acquiring state transition rules based on a set of states, it is possible to acquire states or state transition rules for selecting an action, even in an environment where states or state transition rules are unknown.

  The action selection learning device according to the present invention comprises each pair of the state and the action based on the state transition rule acquiring device described above, the set of states obtained by the state transition rule acquiring device, and the state transition rule. Against the action selection policy acquisition unit that learns an expected gain when an action is performed in the state, and the processing by the state transition rule acquisition device until the learning in the activity selection policy acquisition unit converges; And a convergence determination unit that repeats learning by the selection policy acquisition unit.

  In this way, the state transition rule acquisition device and the action selection policy acquisition unit perform the state for each pair of state and action based on the set of states obtained by the state transition rule acquisition device and the state transition rule. An environment where the state or state transition rule is unknown by repeating the processing by the state transition rule acquisition device and the learning by the action selection policy acquisition unit until the expected gain at the time of performing the action is learned and the learning converges. Even in this case, it is possible to acquire a state or state transition rule for selecting an action.

  In the state transition rule acquisition device according to the present invention, the model may be a multilayer neural network for selecting an action, using the state after the action as an input.

  In the action selection learning device according to the present invention, the reward calculating unit may calculate the reward when the action is performed in the state, the number of visits to the state, the number of times the action is selected in the state, and the state And the expected gain when the action is taken.

  The action selection apparatus according to the present invention selects an action that maximizes the expected gain with respect to the input state based on the expected gain obtained when performing the action in the state of the environment, which has been learned in advance. The behavior selection apparatus according to claim 1, further comprising: a selection unit, wherein the expected gain acquires a state of an environment after the action when the selected action is performed, and the acquired state and the selected state. A reward for performing the action in the state is calculated based on the action, and the state is used as an input based on the selected action and the calculated reward, and a model for selecting the action The parameter is updated, the state after the action is input, and selecting the action using the model is repeated until a predetermined iteration end condition is satisfied, and the set of states and the state transition rule are used. The For each pair of said action and the state, and learning the expected gain when performing an action in the state, is learned in advance by alternately repeating.

  Further, in the action selection method according to the present invention, the expected gain is maximum with respect to the input state based on the expected gain when the action selection unit performs the action in the state of the environment, which is learned in advance. The expected gain is obtained by acquiring the state of the environment after the action when the selected action is performed, and the acquired state Calculating a reward when performing the action in the state based on the selected action and the selected action, and selecting the action based on the selected action and the calculated reward, using the state as an input Parameters of the model to be updated, the state after the action is taken as an input, and selecting the action using the model is repeated until a predetermined iteration end condition is satisfied, the set of states and State It is characterized in that learning is performed in advance by alternately repeating learning of an expected gain when an action is performed in the state with respect to each pair of the state and the action based on a transfer rule. Do.

  According to the action selection device and the action selection method according to the present invention, the expected gain is maximized with respect to the input state based on the expected gain when performing the action in the environmental state, which is learned in advance. Choose an action.

  The expected gain is obtained by acquiring the state of the environment after the action when the selected action is performed, and calculating the reward when the action is performed in the state based on the acquired state and the selected action. Based on the selected action and the calculated reward, the state is input, the parameters of the model for selecting the action are updated, the state after action is input, and the action is selected using the model Learn the expected gain when performing an action in a state with respect to each pair of a state and an action on the basis of a set of states and a state transition rule, by repeating the above until a predetermined iteration end condition is satisfied, and Learning is done in advance by alternately repeating.

  Thus, the state of the environment after the action when the selected action is performed is acquired, and the reward when performing the action in the state is calculated and selected based on the acquired state and the selected action. Based on the calculated action and the calculated reward, the state is input, the parameters of the model for selecting the action are updated, the state after action is input, and the action is selected using the model. Repeating until a predetermined iteration end condition is satisfied, and learning expected gain when performing an action in the state with respect to each pair of the state and the action based on the set of states and the state transition rule And by alternately repeating, and based on the expected gain at the time of performing the action in the state of the environment, which has been learned in advance, by selecting the action for which the expected gain is maximum for the input state, State or condition Transition rule even unknown environment, it is possible to select the appropriate action.

  A program according to the present invention is a program for functioning as each unit of the above-mentioned state transition rule acquisition device, action selection learning device, or action selection device.

  According to the action selection learning device, the action selection learning method, and the program of the present invention, it is possible to obtain a state or state transition rule for selecting an action even in an environment where the state or state transition rule is unknown. .

  Further, according to the action selection device, the action selection method, and the program of the present invention, an appropriate action can be selected even in an environment where the state or the state transition rule is unknown.

It is a block diagram which shows the structure of the action selection learning apparatus based on embodiment of this invention. It is a block diagram which shows the structure of the state transition rule acquisition part which concerns on embodiment of this invention. It is a flowchart which shows the action selection learning process routine of the action selection learning apparatus which concerns on embodiment of this invention. It is a flowchart which shows the acquisition process routine of the state which concerns on embodiment of this invention, and a state transition rule. It is a block diagram which shows the structure of the action selection apparatus which concerns on embodiment of this invention. It is a flowchart which shows the action selection processing routine of the action selection apparatus which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described using the drawings.

<Principle of the embodiment of the present invention>
First, the principle of the embodiment of the present invention will be described.

  In the embodiment of the present invention, a simulator that virtually realizes a real environment is assumed. This simulator presents what happens in the environment when a decision support system selects a certain action. This simulator presents information based on past cases and the like. At the same time, the gain of the current state is also presented.

  First, the decision support system performs a process of acquiring states and state transition rules (process 1). Here, we aim to acquire possible states and state transition rules under the environment. To this end, under simulation, a search is made to discover new states and state transitions. For this search, for example, deep Q-network (DQN) or the like in which deep learning is applied to Q learning, which is a type of reinforcement learning, is used.

  The agent acquires the state and the gain for the state from the environment, and learns an action that can preferentially transition to the state considered to be able to obtain more gain.

  Next, the decision support system performs a process of acquiring an action selection policy (process 2). Here, the gain calculation of the action in each state is performed using the obtained state and the one corresponding to the state transition rule.

Specifically, the decision support system proceeds learning so that, for a pair of a state and an action for the state, the one with the largest loss becomes smaller when it transitions to the final state. That is, assuming that the loss at the time of transition to the final state is x _i , the largest value (max (x _i )) of the losses x _i is as small as possible (min (max (x _i )) )learn. By repeating this learning, the goodness of the state and the action pair for the state will be gradually obtained. Finally, when simulation is performed a sufficient number of times, actions to be taken in each state can be obtained.

  Then, the decision support system performs processing of convergence judgment of the action selection policy (Process 3). If the action selection strategy converges, the process ends; otherwise, the process returns to process 1. That is, the process 1 and the process 2 are repeated to obtain a final action selection strategy.

<< Process 1. Acquisition of states and state transition rules >>
Let S be a set of states. Further, a set of state transitions is T = S × S. Basically, S assumes an empty set in the initial state. However, when this process is performed by repeated processing, the final state of the immediately preceding process is the initial state of this process. Therefore, in this case, S is not an empty set, but the initial state of S does not affect the present processing, so any set may be given.

Next, let A be a set of actions. Also, a and s are used as symbols representing action (independent of time) and state, respectively. In other words

And

It is. In the present invention, a state and state transition rule are searched using a framework of reinforcement learning by a multi-layered neural network. Here, the entire multilayer neural network is represented by M _θ , and its parameter is θ.

Let t be time, and state of time t

I will write it as. For each set of possible actions A (s _t ) in the state of time t, each action

Are modeled by probability values using a multi-layered neural network. Also, the end state f _t of time t is a function that returns 0 or 1, and returns 1 if it is an end state, 0 otherwise.

Then, according to the following algorithm, states and state transitions are acquired.
Step 1. (Initialization) t = 1, read the configuration of the multilayer neural network M _θ .
Step 2. Put the action a _t-1 one time before in the environment, in the case of t = 0 assume that "no action".
Step 3. State s _t for action, reward r (s _{t -1} , at ₁ ) for action, and end state f _t from environment.
Step 4. If you find a new state add the state s _t to S.
Step 5. Update the parameter θ of the network using the action at _-1 and the reward r for the action ( _st-1 , at _-1 ).
Step 6. End determination: If f _t is 1 (in the end state), the process ends, and if f _t is 0 (not in the end state), the following processing is continued.
Step 7. Step 3. Using the state acquired in step 1 as an input. Calculate the value of each element of the network according to the definition of the network built in and act as the action of time t

Choose the action with the highest value of.
Step 8. s _t reward for taking the action _{a t} in the _{r (s t, a} t) update. For example, it updates using Formula (1) mentioned later.
Step 9. Let t = t + 1, step 2. Return to

Reward r (s _t, a _t) at the time of taking the action a _t in a certain state s _t is, given a high evaluation value when the state s _{t + 1} obtained as a result of a _t was an unknown state, known In the case of a state, a function is used in which the reward decreases as the number of visits increases. This function is used in updating the reward r (s _t , a _t ) in step 8 above.

  Although the specific definition of the reward can be considered infinitely, for example, it is considered to calculate based on a formula called upper confidence bound (UCB).

The number of times _{here, g (s t, a t} ) a visit to the state _{s t} of the selected in the state _{s t} of the time t action _{a t} the expected gain to be example when I went, n _{(s t)} the time t represents _{n (s} t, _{a t)} number of selecting an action _{a t} in state _{s t} of the time t, respectively. Further, α is a weighting factor of the first term and the second term.

When the action selection based on UCB is performed infinite times, the second term of the equation (1) asymptotically approaches 0, so that the evaluation value according to the value of r (s _t , a _t ) is obtained. Conversely, when the state s _t is visited for the first time or hardly visited, an evaluation value close to random is obtained.

Further, if corresponding to the just middle, an evaluation value such as the past to preferentially selected as when a small number of taking the same action with respect to action a _t taken in some states s _t.

<< Process 2. Acquisition of action selection policy >>
Using the set S of state obtained and the state transition rule T, a policy of action selection is obtained using CFR (non-patent document 1) or the like which is a conventional method. More specifically, the expected gain _{g (s} t, _{a t)} on behavior _{a t} selected at state _{s t} at a certain time t and holds calculated.

<< Process 3. Convergence judgment of action selection policy >>
If the expected gains g (s _t , a _t ) obtained for all the s _t and a _t pairs are sufficiently small from the previous processing result, the learning processing ends. If the difference between the expected gains is not sufficiently small, the process returns to the process 1 again.

<< Behavior selection >>
Finally, decision support systems, to select an action to the state s _t based on g (s, a) obtained by the learning. More specifically, the action that gives the best g (s, a) for a certain state s
To present.

  Hereinafter, an action selection learning device that automatically acquires an action selection strategy for a given state will be described, and then an action selection device that is actually presented will be described using the acquired action selection strategy.

<Configuration of Action Selection Learning Device According to Embodiment of the Present Invention>
The configuration of the action selection learning device according to the embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration of an action selection learning device according to an embodiment of the present invention.

  The action selection learning device 10 is configured by a computer including a CPU, a RAM, and a ROM storing a program for executing an action selection learning processing routine described later, and is functionally configured as follows: ing.

  As shown in FIG. 1, the action selection learning device 10 according to the present embodiment includes a state transition rule acquisition unit 100, a simulation unit 110, a state storage unit 120, an expected gain storage unit 130, and an action selection policy acquisition unit. 140, a convergence determination unit 150, and an output unit 160.

  The state transition rule acquiring unit 100 acquires the state of the environment after the action when the input action is performed, and acquires the acquired state if it is a new state in comparison with the set of already acquired states. State is added to the set of states, and a state transition rule is acquired based on the set of states.

  Specifically, as shown in FIG. 2, the state transition rule acquisition unit 100 includes an initialization unit 200, a state acquisition unit 210, a reward calculation unit 220, a calculation data storage unit 230, and a parameter update unit 240. , A model storage unit 250, an end determination unit 260, an action selection unit 270, and an action storage unit 280.

Initializing unit 200 is started the process of the state transition rule acquisition unit 100, or receives the process is carried out instructions of the convergence determination portion 150 the state transition rule acquisition section 100 from, initializes the time t 1, action a _{t -1} (a ₀ : no action) is input to the state acquisition unit 210.

  The state acquisition unit 210 acquires the state of the environment after the action when the selected action is performed.

Specifically, the state acquisition unit 210 passes the behavior at _-1 input by the initialization unit 200 or selected by the behavior selection unit 270 to the simulation unit 110. Then, when the action at _-1 calculated by the simulation unit 110 is performed, the state s _t of the environment after the action and the end state f _t are acquired.

  Here, the state s is, for example, if the environment is a card game such as playing cards, the state of the hand after the action of acquiring or discarding the bill, the state of the bill put out to the place, etc. , An image captured by a camera mounted on a robot or the like after an action such as movement.

In addition, the end state f _t achieved the purpose if the environment was a card game such as playing cards or not, and if the game was finalized (or lost), if it was a robot etc, it should be stopped or not. It is an end state of the environment, such as whether or not it is possible to judge the possibility of achieving the purpose.

The status acquisition unit 210 acquires state s _t is compared with the set S of states already acquired, if the new state, the acquired state s _t, stored in the state storage unit 120 conditions State transition rule T based on the set S of states.

  At this time, whether the acquired state is a new state or not is determined based on whether or not a coincident state is included if the states are discrete, such as a combination of hands. If the state is continuous like an image, it is determined that the state is new if the similarity between the states (for example, the similarity of the images) is equal to or higher than a threshold. Besides this, since various states may exist in an open environment, various criteria can be provided by comparing the states.

Then, the state acquisition unit 210 passes the acquired state _st to the reward calculation unit 220.

Fee calculator 220, reward _{r (s} t, _{a t)} when the acquired state _{s t} was action _{a t} the visited state _{s t} at time t stored in the calculation data storage section 230 number n _(s t), and state _{s t} at action _{a t} times the selected _{n (s} t, _{a t)} and, when performing an action _{a t} in state _{s t} stored in the expected gain storage unit 130 It calculates based on the expected gain g (s _t , a _t ) of

Specifically, compensation calculation unit 220, the state _s when performing an action _{a t} that can be selected at _{t n (s t), n} (s t, a t), and the expected gain _{g (s} t, _{a t} Based on the above equation (1), calculate the reward r (s _t , a _t ). Compensation calculation unit 220, the action _{a t} all be selected in the state _{s t,} reward _{r (s} t, _{a t)} calculations performed.

Then, the reward calculating unit 220 passes the calculated reward r (s _t , a _t ) to the parameter updating unit 240.

Calculation data storage unit 230, the number of visits _{s t} at time t n _(s t), and time t in state _{s t} at action _{a t} times the selected _{n (s} t, _{a t)} stores.

Parameter update unit 240, the input action _{a t-1} and the compensation calculator 220 reward for that acted _{a t-1} in the previous state _{s t-1} calculated by _{r (s t-1,} a _t-1) and on the basis, as input state s _t, to update the model parameters for selecting the action a _t. In the present embodiment, the model is a multi-layered neural network M _θ for selecting an action based on the state after the action.

Specifically, the parameter updating unit 240, an action _{a t-1,} based on the reward _r on behavior _{a t-1 (s t-} 1, a t-1), to select the reward is high action The parameter θ of the multilayer neural network M _θ is updated, and the updated parameter θ is stored in the model storage unit 250.

The model storage unit 250 stores parameters θ of the multilayer neural network M _θ . When the model storage unit 250 receives an update of the parameter θ from the parameter update unit 240, the model storage unit 250 updates the parameter θ.

  The end determination unit 260 repeats the acquisition by the state acquisition unit 210, the calculation by the reward calculation unit 220, the update by the parameter update unit 240, and the selection by the action selection unit 270 until the predetermined repetition end condition is satisfied.

Specifically, the end determination unit 260 determines whether the end state f _t acquired by the state acquisition unit 210 represents an end state. If the end state f _t is 1 (end state), the action selection strategy acquisition unit 140 starts processing.

If the end state f _t is 0 (not the end state), the action selection unit 270 is made to perform processing.

Action selection unit 270 inputs the state _{s t} after the action _{a t-1,} using a multi-layer neural network M _theta, selects an action _{a t.}

Specifically, the action selection unit 270 can take on the state s _t by the action set A stored in the action storage unit 280 and the multi-layered neural network M _θ stored in the model storage unit 250. Action

Calculate the probability value of For example, according to the definition of multi-layer neural network M _theta a s _t as input, calculates the value of each element of M _theta, as the action of the time t, selecting a most probable action a _t.

The action selecting section 270 calculates the data visits n _(s t) of _{s t} at time t in the storage unit 230 are stored, and the number of times n (s selecting an action _{a t} in state _{s t} at time t _t, to add 1 to _{a t),} reward _{r (s} t, to update _{a t),} and passes to the state acquisition unit 210 the selected action _{a t} as the action _{a t-1.}

  In the action storage unit 280, a set A of actions a in the environment is stored in advance.

  The simulation unit 110 calculates the environment after performing the input action a, and returns the environment after the action. The post-action environment includes the state of the environment, the end state of the environment, and the like. Although the environment used here is an environment (open environment) whose state and state transition rule are unknown, it may be a closed environment.

  The state storage unit 120 stores the set S of states s and the state transition rules T obtained by the state transition rule acquisition unit 100. Also, a state determined to be a new state by the state acquisition unit 210 is added to the state set S. Note that the set of states S may be an empty set until the states are added by the state acquisition unit 210.

  The expected gain storage unit 130 stores the expected gain g learned by the action selection and strategy acquisition unit 140.

  The action selection strategy acquisition unit 140 performs the action a in the state s for each pair of the state s and the action a based on the set S of states and the state transition rule T obtained by the state transition rule acquisition unit 100. Learn the expected gain g (s, a) when done.

Specifically, the action selection strategy acquisition unit 140 performs calculation by the simulation unit 110 using the set of states S stored in the state storage unit 120 and the state transition rule T, and the CFR (non-patent document is a conventional method). Obtain a policy of action selection using the literature 1). More specifically, the expected gain g (s _t , a _t ) is calculated for all the pairs of the state s and the action a, that is, the action a _t selected in the state s _{t at} a certain time t. , To get that measure.

  Then, the action selection policy acquisition unit 140 stores the acquired expected gain g in the expected gain storage unit 130.

  The convergence determination unit 150 repeats the process by the state transition rule acquisition unit 100 and the learning by the action selection policy acquisition unit 140 until the learning in the behavior selection policy acquisition unit 140 converges.

Specifically, the convergence determination unit 150, the expected were obtained for pairs of all _{s t} and _{a t} gain _{g (s} t, _{a t)} and the expected gain g _{(s t} a previous processing results If the difference from ( ₋₁ , at ₋₁ ) is sufficiently small, it is determined that the learning has converged, and the convergence determination result is passed to the output unit 160.

  If the difference is not sufficiently small, the state transition rule acquiring unit 100 is caused to perform processing again.

  The output unit 160 outputs the convergence determination result acquired from the convergence determination unit 150.

<Operation of action selection learning device according to the embodiment of the present invention>
FIG. 3 is a flowchart showing an action selection learning processing routine of the action selection learning device according to the embodiment of the present invention.

  When the action selection learning device is activated, an action selection learning processing routine shown in FIG. 3 is executed.

  First, in step S100, the state transition rule acquiring unit 100 stores the state set S and the state transition rule T in the state storage unit 120 by executing an acquisition process routine for states and state transition rules described later. .

  Next, in step S110, the action selection strategy acquisition unit 140 determines each pair of the state s and the action a based on the set S of states and the state transition rule T obtained by the state transition rule acquisition unit 100. Then, learn the expected gain g (s, a) when the action a is performed in the state s.

  In step S120, the convergence determination unit 150 determines whether the learning in the action selection and strategy acquisition unit 140 has converged.

  If it is determined that the learning has not converged (NO in step S120), the convergence determination unit 150 returns to step S100, and the process by the state transition rule acquisition unit 100 (step S100) and the learning by the action selection policy acquisition unit 140 (Step S110) is repeated.

  On the other hand, when it is determined that the learning has converged (YES in step S120), the output unit 160 outputs the convergence determination result in step S130.

  Next, the acquisition process routine of the state and the state transition rule in step S100 will be described using FIG.

  When the process of the state transition rule acquisition unit 100 is started or an instruction to perform the process of the state transition rule acquisition unit 100 is received from the convergence determination unit 150, the acquisition process routine of the state and the state transition rule shown in FIG. .

In step S200, the initialization unit 200 initializes the time t 1, action _{a t-1:} inputs _{(a 0} no action) to the state acquisition section 210.

In step S210, the state acquisition unit 210 passes the action at _-1 to the simulation unit 110.

In step S220, the state acquisition unit 210 acquires the state s _t of the environment and the end state f _t when the action at _-1 calculated by the simulation unit 110 is performed.

In step S230, the status acquisition unit 210 acquires state s _t is compared with the set S of states already acquired, determines whether the new state.

If the acquired state s _t is a new state (YES in step S 230), in step S 240, the state acquiring unit 210 sets the acquired state s _t to the set S of states stored in the state storage unit 120. A state transition rule T is acquired based on the set S of states.

When the acquired state s _t is not a new state (NO in step S230), the process of step S240 is not performed, and the process proceeds to step S250.

In step S250, parameter update unit 240, compensation calculation unit 220 has calculated the last time, reward _r when performing an action _{a t-1} in the previous state _{s t-1 (s t-} 1, a t-1) To get

In step S260, the parameter updating unit 240 updates the parameters of the multilayer neural network M _θ based on the input action at _-1 and the acquired reward r ( _st-1 , at _-1 ).

In step S270, the end determination unit 260 determines whether the end state f _t acquired by the state acquisition unit 210 is 1 representing the end state.

If the end state f _t is not 1 (NO in step S270), it is determined that the end state f _t is not the end state, and the process proceeds to step S280.

In step S280, the action selecting section 270 inputs the state _{s t} after the action _{a t-1,} using a multi-layer neural network M _theta, selects an action _{a t.}

In step S290, the action selecting section 270, the number n of times calculated data number of visits _{s t} at time t in the storage unit 230 are stored n _(s t), and were selected action _{a t} in state _{s t} at time t _{(s t, a} t) to add 1 to, reward _{r (s t, a} t) to update. Reward _{r (s} t, _{a t)} updates the compensation calculation unit 220, reward _{r (s} t, _{a t)} when the acquired state _{s t} was action _{a t} a, the calculation data storage section 230 number visited state _{s t} at the stored time t n _(s t), and the number of times _{n (s} t, _{a t)} to the action _{a t} selected at state _{s t} a, are stored in the expected gain storage unit 130 performed in the state _{s t} in which action _a expectations _{when t} was performed gain _{g (s} t, _{a t)} by calculating, based on the.

In step S300, the action selecting section 270 passes the state acquisition section 210 the selected action _{a t} as an action _{a t-1,} the flow returns to step S210.

On the other hand, when the end state f _t is 1 (YES in step S270), the end determination unit 260

  As described above, according to the action selection learning device according to the present embodiment, the state of the environment after the action when the input action is performed is acquired and acquired until the predetermined iteration end condition is satisfied. Based on the selected state and the selected action, the reward at the time of performing the action in the state is calculated, and based on the selected action and the reward calculated by the reward calculating unit, the state is input and the action is Update the parameters of the model for selection, take the post-action state as input, repeat selecting the action using the model, and compare the acquired states with the already acquired set of states. Then, the acquired state is added to the set of states, and by acquiring the state transition rule based on the set of states, the action is selected even in an environment where the state and the state transition rule are unknown State or condition for You can acquire the transition rules.

  Further, according to the action selection learning device according to the present embodiment, the state transition rule acquiring device and the action selection policy acquiring unit are states based on the set of states acquired by the state transition rule acquiring device and the state transition rule. For each pair of behavior and behavior, learn expected gain when behavior is performed in the state, and repeat processing by the state transition rule acquisition device and learning by the action selection strategy acquisition unit until learning converges. Thus, even in an environment where the state or state transition rule is unknown, it is possible to obtain the state or state transition rule for selecting an action.

<Configuration of Action Selection Device According to Embodiment of the Present Invention>
Next, the action selection device according to the present embodiment will be described. In the present embodiment, the behavior selection device is described as being mounted on a control target (for example, a robot) which is a target to be controlled to perform the selected behavior. The action selection apparatus according to the present embodiment acts in a state obtained by the sensor of the control target (for example, an image captured by a camera mounted on the control target) in an actual environment in which the control target actually moves to the destination Control the controlled object so as to perform an action (eg, turn to the right, go straight, etc.) to the selection device with which the expected gain is maximized (for example, to reach the destination) Shall be configured to

  The action selection apparatus according to the present embodiment will be described with reference to FIG. FIG. 5 is a block diagram showing the configuration of the action selection device according to the embodiment of the present invention.

  The action selection device 20 is configured by a computer including a CPU, a RAM, and a ROM storing a program for executing an action selection learning processing routine described later, and is functionally configured as shown below There is.

  As shown in FIG. 5, the action selection apparatus 20 according to the present embodiment includes an input unit 300, an action selection unit 310, an expected gain storage unit 320, an output unit 330, and a control unit 340. Ru.

  The input unit 300 receives an input of the state s obtained by the sensor to be controlled. Then, the input unit 300 passes the received state s to the action selection unit 310.

  The action selection unit 310 selects, for the input state s, an action a with which the expected gain is maximum, based on the expected gain obtained when performing an action in the state of the environment, which has been learned in advance.

Specifically, based on the learned expected gain g (s, a) stored in the expected gain storage unit 320, the behavior selecting unit 310 obtains the expected gain g (s) for the input state s. , A) select the action a with the largest. That is, the action that is the best g (s, a) for a certain state s

Choose Then, the action selection unit 310 passes the selected action a to the output unit 330.

  The expected gain storage unit 320 stores the expected gain learned by the action selection learning device 10 described above.

  The output unit 330 is implemented by a display, a printer, a magnetic disk, etc., and outputs the action a selected by the action selection unit 310.

  Also, the output unit 330 outputs the action a selected by the action selection unit 310 to the control unit 340.

  The control unit 340 controls the behavior of the control target so as to perform the behavior input by the output unit 330. For example, the robot is instructed to take actions such as turning to the right or going straight.

<Operation of Behavior Selection Device According to Embodiment of the Present Invention>
FIG. 6 is a flowchart showing an action selection processing routine of the action selection device according to the embodiment of the present invention.

  When the state obtained by the sensor to be controlled is input to the input unit 300, an action selection processing routine shown in FIG. 6 is executed.

  First, in step S400, the input unit 300 receives an input of the state s obtained by the sensor to be controlled.

  Next, in step S410, based on the input state s, it is determined whether it is an end state. If it is in the end state (YES in step S410), the action selection processing routine is ended. On the other hand, if it is not in the end state (NO in step S410), the process proceeds to step S420.

  In step S 420, the behavior selection unit 310 reads the expected gain stored in the expected gain storage unit 320.

  In step S430, based on the expected gain read in step S420, the action a with the largest expected gain is selected with respect to the state s acquired in step S400.

  In step S440, the output unit 330 outputs the action a selected by the action selection unit 310. Further, the output unit 330 outputs the action a selected by the action selection unit 310 to the control unit 340, and returns to step S400.

  Thereafter, the control unit 340 controls the behavior of the control target, and the sensor of the control target acquires a state, whereby the behavior selection processing routine is repeated.

  As described above, according to the action selection device, the state of the environment after the action when the selected action is performed is acquired, and the action is performed in the state based on the acquired state and the selected action. Calculate the rewards of the user, input the state based on the selected action and the calculated reward, update the parameters of the model for selecting the action, input the post action state, and enter the model Use the action to repeat the action selection until the predetermined iteration end condition is satisfied, and perform the action in the state for each pair of the state and the action based on the set of states and the state transition rule The expected gain is maximum for the input state based on the expected gain when the user performs an action in the environmental state, which has been learned in advance by learning the expected gain at the same time and repeating alternately Became action By selecting the state and the state transition rule even unknown environment, it is possible to select the appropriate action.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the scope of the present invention.

  In the present embodiment, the action selection device uses the state of the real environment acquired by the sensor to be controlled as an input, but the state of the environment calculated by the simulator may be an input. For example, if the environment is a card game such as playing cards, the state of the hand after the action such as obtaining / discarding the bill calculated by the simulator or the state of the bill put out to the place may be input. It is also good.

  Furthermore, although the present invention has been described as an embodiment in which the program is installed in advance, it is also possible to provide the program by storing the program in a computer readable recording medium.

10 action selection learning device 20 action selection device 100 state transition rule acquisition unit 110 simulation unit 120 state storage unit 130 expectation gain storage unit 140 action selection strategy acquisition unit 150 convergence determination unit 160 output unit 200 initialization unit 210 state acquisition unit 220 reward Calculation unit 230 Calculation data storage unit 240 Parameter update unit 250 Model storage unit 260 End determination unit 270 Action selection unit 280 Action storage unit 300 Input unit 310 Action selection unit 320 Expected gain storage unit 330 Output unit 340 Control unit

Claims (8)

A state acquisition unit for acquiring the state of the environment after the action when the selected action is performed;
A reward calculation unit that calculates a reward when the action is performed in the state based on the acquired state and the selected action;
A parameter updating unit that updates the parameters of a model for selecting an action based on the selected action and the reward calculated by the reward calculation unit, using the state as an input;
An action selecting unit that selects an action using the state after the action as an input and using the model;
An end determination unit that repeats acquisition by the state acquisition unit, calculation by the reward calculation unit, update by the parameter update unit, and selection by the action selection unit until a predetermined iteration end condition is satisfied;
Equipped with
The state acquisition unit adds the acquired state to the set of states if the acquired state is a new state as compared to the set of already acquired states, and based on the set of states, A state transition rule acquisition device characterized by acquiring a state transition rule. The state transition rule acquisition device according to claim 1;
An action of learning an expected gain when performing an action in the state with respect to each pair of the state and the action based on the set of states and the state transition rule obtained by the state transition rule acquisition device A selection policy acquisition unit,
A convergence determination unit that repeats the processing by the state transition rule acquisition device and the learning by the behavior selection policy acquisition unit until the learning in the activity selection policy acquisition unit converges;
An action selection learning device comprising:   The state transition rule acquisition device according to claim 1, wherein the model is a multilayer neural network for selecting an action, using the state after the action as an input.   The reward calculation unit is configured to calculate the reward when the action is performed in the state, the number of visits to the state, the number of times the action is selected in the state, and an expected gain when the action is performed in the state 3. The action selection learning device according to claim 2, wherein the action selection learning device calculates on the basis of. The apparatus is characterized by comprising an action selection unit which selects an action having the maximum expected gain with respect to the input state based on the expected gain obtained when performing an action in the state of the environment, which has been learned in advance. A behavior selection device,
The expected gain is
The state of the environment after the action when the selected action is performed is acquired, and the reward when the action is performed in the state is calculated based on the acquired state and the selected action, Based on the selected action and the calculated reward, the state is used as an input, parameters of a model for selecting an action are updated, the state after action is used as an input, and the action is performed using the model. Repeating the selection until the predetermined iteration end condition is satisfied;
Learning an expected gain when an action is performed in the state with respect to each pair of the state and the action based on the set of states and a state transition rule;
An action selection device which is learned in advance by alternately repeating. A step of acquiring a state of the environment after the action when the state acquisition unit performs the selected action;
Calculating a reward at the time of performing the action in the state based on the acquired state and the selected action;
The parameter updating unit, based on the selected action and the reward calculated by the reward calculation unit, updating the parameters of the model for selecting the action, using the state as an input;
The action selection unit selects an action using the state after the action as an input, and using the model;
Causing an end determination unit to repeat acquisition by the state acquisition unit, calculation by the reward calculation unit, update by the parameter update unit, and selection by the action selection unit until a predetermined iteration end condition is satisfied;
Equipped with
The step of acquiring the state acquiring unit adds the acquired state to the set of states if the acquired state is a new state as compared to the set of already acquired states, and the set of states is acquired. A method of acquiring a state transition rule, comprising acquiring a state transition rule based on. The action selecting unit comprises a step of selecting an action having the maximum expected gain with respect to the input state based on the expected gain when performing an action in the state of the environment, which has been learned in advance. It is an action selection method characterized by
The expected gain is
The state of the environment after the action when the selected action is performed is acquired, and the reward when the action is performed in the state is calculated based on the acquired state and the selected action, Based on the selected action and the calculated reward, the state is used as an input, parameters of a model for selecting an action are updated, the state after action is used as an input, and the action is performed using the model. Repeating the selection until the predetermined iteration end condition is satisfied;
Learning an expected gain when an action is performed in the state with respect to each pair of the state and the action based on the set of states and a state transition rule;
A method of behavior selection characterized in that it is learned in advance by alternately repeating.   The program for functioning a computer as a state transition rule acquisition apparatus of Claim 1 or Claim 3, the action selection learning apparatus of Claim 2 or Claim 4, or the action selection apparatus of Claim 5.