JP2019219741A - Learning control method and computer system - Google Patents

Abstract

To achieve machine learning of short computing time while avoiding overlearning.SOLUTION: In a learning control method for learning a policy, a computer system includes: a learning unit which executes a plurality of times of simulations based on a transition model parameter obtained by modifying a part of a target model parameter for executing a simulation, and executes learning processing for updating the policy; and a history database which manages information related to the policy updated by execution of the transition simulation as a learning history. The learning unit stores the learning history in the history database, and executes a plurality of times of simulations based on the transition model parameter obtained by calculation based on the learning history, when it is necessary to execute the simulation using the learning history.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for improving the computational performance of machine learning, particularly, reinforcement learning.

  In recent years, machine learning has been used in various situations. For example, a system that automatically presents an operation plan or the like for improving the productivity of a product in a manufacturing factory has been receiving attention.

  Reinforcement learning is known as one of the machine learning. A system using reinforcement learning is a policy that serves as a guideline for selecting actions by performing trial and error of actions using an environment that imitates the business environment of a manufacturing factory and agents that perform actions such as product manufacturing work. Alternatively, an action plan or the like is output.

  Various methods have been proposed as calculation methods for reinforcement learning. For example, a technique described in Patent Document 1 is known. Patent Literature 1 describes that in order to optimize a policy, initial parameters of the policy are determined in advance, trial and error of behavior and environmental state transition are performed, and the policy is repeatedly updated. .

  In the case of a complex environment (problem), the search space is large, so that it takes time to optimize the policy by iteratively updating the policy. Therefore, a calculation time reduction method as described in Non-Patent Document 1 is known.

  Non-Patent Document 1 describes that a complicated environment is replaced with a simple environment, machine learning is performed on the simple environment, and machine learning is performed on the original environment using the obtained result. I have. By combining the technology described in Non-Patent Document 1 with the technology described in Patent Document 1, the technology can be applied to various methods of machine learning.

US Patent Application Publication No. 2017/0278018

Sermanet, Pierre, et al, "Pedestrian detection with unsupervised multi-stage feature learning.", Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, 2013.

  When a result obtained from machine learning for a simple environment is used, a policy specialized for a simple environment may be output. That is, it may converge to a local solution of the original environment. The phenomenon described above is called overlearning. It is known that the technique described in Patent Document 1 has a high tendency to cause over-learning. Therefore, a device for suppressing the occurrence of over-learning is required.

  As a method of avoiding over-learning, a method of randomly setting parameters such as a policy is known. However, this method has a problem that the calculation time of machine learning becomes long.

  The object of the present invention is to explain the above problems. That is, a method and system that avoids over-learning and realizes machine learning with a short operation time are realized.

  A typical example of the invention disclosed in the present application is as follows. That is, a learning control method in a computer system that learns a policy for determining a control content of a process for controlling an object, wherein the computer system selects the control content of the process based on an arbitrary policy. A learning control unit that calculates a transition model parameter obtained by partially changing a target model parameter for realizing a simulation; and the transition calculated based on the transition model parameter input from the learning control unit or a result of a previous simulation. A learning device that executes the simulation based on the model parameters a plurality of times, and executes a learning process of updating the policy based on the result of the simulation; and the transition model parameters and the policy updated by executing the transition simulation. Relevant information and learning history A learning database, the learning control method comprising: a first step in which the learning device stores the learning history in the history database at an arbitrary timing; A second step of determining whether it is necessary to execute the simulation using the learning history based on the evaluation value of the policy updated by the simulation executed only, and using the learning history. If it is determined that it is necessary to execute the simulation, the learning device executes the simulation based on the transition model parameters calculated based on the use learning history selected from the history database a plurality of times, Updating the policy based on a result of the simulation.

  According to one embodiment of the present invention, machine learning that avoids over-learning and has a short operation time can be realized. Problems, configurations, and effects other than those described above will be apparent from the following description of the embodiments.

FIG. 1 is a diagram illustrating a configuration example of a system according to a first embodiment. FIG. 3 is a diagram illustrating a configuration example of a sub process controller according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of learning condition parameter information according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of an environmental parameter according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of an agent parameter according to the first embodiment. FIG. 6 is a diagram illustrating an example of a data structure of history relation management information according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of a learning result DB according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of a history DB according to the first embodiment. 6 is a flowchart illustrating an outline of a process executed by a computer according to the first embodiment. 6 is a flowchart illustrating a process executed by the learning controller according to the first embodiment. 6 is a flowchart illustrating a process executed by the learning controller according to the first embodiment. 6 is a flowchart illustrating a process executed by a sub process controller according to the first embodiment. 6 is a flowchart illustrating a process executed by a score determination module according to the first embodiment. FIG. 6 is a diagram illustrating an example of a GUI displayed by the computer according to the first embodiment. FIG. 6 is a diagram illustrating an example of a GUI displayed by the computer according to the first embodiment. FIG. 2 is a diagram illustrating a modification of the configuration of the system according to the first embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the present invention is not construed as being limited to the description of the embodiments below. It is easily understood by those skilled in the art that the specific configuration can be changed without departing from the spirit or spirit of the present invention.

  In the structures of the invention described below, the same or similar structures or functions are denoted by the same reference numerals, and redundant description will be omitted.

  Notations such as “first”, “second”, and “third” in this specification and the like are used to identify components, and do not necessarily limit the number or order.

  The position, size, shape, range, or the like of each component illustrated in the drawings and the like is not accurately represented in some cases in order to facilitate understanding of the invention. Therefore, the present invention is not limited to the position, size, shape, range, and the like disclosed in the drawings and the like.

  In the present specification, the invention will be described by taking reinforcement learning which is one of machine learning as an example. In reinforcement learning, a desired result is obtained by executing a simulation using an environment and an agent.

  FIG. 1 is a diagram illustrating a configuration example of the system according to the first embodiment.

  The system includes a computer 100 and a terminal 101. The computer 100 and the terminal 101 are connected to each other via a network. As the network, for example, a LAN (Local Area Network) and a WAN (Wide Area Network) can be considered. The network connection method may be either wireless or wired. Note that the computer 100 and the terminal 101 may be directly connected.

  The terminal 101 is a terminal operated by a user. The terminal 101 is a general-purpose computer or a portable terminal having a processor, a memory, and a network interface.

  Using the terminal 101, the user sets parameters necessary for executing reinforcement learning, that is, learning condition parameters, and inputs learning condition parameter information 170 for storing the parameters to the computer 100. In addition, the user uses the terminal 101 to check information output from the computer 100. The data structure of the learning condition parameter information 170 will be described with reference to FIG.

  The computer 100 executes reinforcement learning related to processing for controlling an arbitrary target based on the learning condition parameter information 170. For example, reinforcement learning is performed to search for a policy for selecting an optimal processing procedure or processing content for loading work using a crane. Note that the present invention is not limited to learning targets and learning contents.

  The computer 100 has a processor 110, a memory 111, and a network interface 112 as hardware. Note that the computer 100 may include an IO interface connected to an input device and an output device, and a storage medium such as a hard disk drive (HDD) and a solid state drive (SSD).

  The processor 110 executes a program stored in the memory 111. The processor 110 operates as a module that implements a specific function by executing processing according to a program. In the following description, when processing is described using a module as a subject, it indicates that the processor 110 is executing a program that implements the module.

  The memory 111 stores a program executed by the processor 110 and information used by the program. Further, the memory 111 includes a work area temporarily used by the program.

  The network interface 112 is an interface for connecting to another device via a network.

  Here, programs and information stored in the memory 111 will be described. The memory 111 stores a program that implements the learning controller 120, the sub-process controller 130, and the score determination module 140. The memory 111 stores a history DB 150 and a learning result DB 160.

  The sub-process controller 130 performs reinforcement learning. Specifically, the sub-process controller 130 repeatedly executes a simulation using the environment module 131 constructed based on the environment parameters 171 and the agent module 132 constructed based on the agent parameters 172.

  In the first embodiment, the sub-process controller 130 executes the simulation using the environment module 131 corresponding to the environment that realizes the simulation whose difficulty is lower than the difficulty of the target simulation. When a desired learning result is obtained, the sub-process controller 130 executes a simulation using the environment module 131 corresponding to the environment in which the difficulty level of the current simulation has been changed and the learning result. The transition of the reinforcement learning according to the difficulty level as described above speeds up the calculation of the simulation using the environment module 131 having the target difficulty level.

  The learning controller 120 controls the reinforcement learning. In the learning controller 120, a learning model for performing a simulation on the target is set. The learning model includes a model of the environment and a model of the agent. The learning controller 120 generates an environment parameter 171 and an agent parameter 172 based on the learning condition parameter information 170 and the learning model, and outputs each parameter to the sub process controller. In the following description, when the environment parameter 171 and the agent parameter 172 are not distinguished, they are described as model parameters.

  The sub-process controller 130 according to the first embodiment stores the model parameters and the learning result (learned policy) of the currently executed simulation in the history DB 150 at an arbitrary timing as a learning history. Further, the sub-process controller 130 requests the score determination module 140 to evaluate the learning result at an arbitrary timing. The data included in the learning history can be set arbitrarily. For example, only the policy internal data 241 may be stored as the learning history.

  The data structure of the environment parameter 171 will be described with reference to FIG. 4, and the data structure of the agent parameter 172 will be described with reference to FIG.

  The learning controller 120 includes an environment rollback controller 121 and a policy rule back controller 122 for restoring the environment module 131 and the agent module 132 reflecting the learning history. The learning controller 120 manages history relationship management information 123 for selecting a learning history to be used. Further, the learning controller 120 manages the total value of the number of executions of the reinforcement learning of any simulation difficulty.

  When the learning controller 120 receives the learning condition parameter information 170, the learning controller 120 calculates an environment parameter 171 for realizing an environment with the lowest difficulty of the simulation. When a desired learning result is obtained, the learning controller 120 calculates an environment parameter 171 for realizing a simulation whose difficulty is higher than the current difficulty of the simulation.

  The learning controller 120 calculates an environment parameter 171 in which some of the parameters included in the parameter for realizing the target environment are changed, or includes the environment parameter 171 in the parameter for realizing the target environment. By calculating the environmental parameters 171 that do not include some of the parameters, the difficulty of the simulation can be changed.

  The score determination module 140 evaluates the learning result. When determining that the optimal policy has been calculated, the score determination module 140 stores the learning result in the learning result DB 160. The score determination module 140 determines an execution plan of the reinforcement learning based on the evaluation of the learning result, and outputs an instruction based on the execution plan to the learning controller 120.

  FIG. 2 is a diagram illustrating a configuration example of the sub-process controller 130 according to the first embodiment.

  The environment module 131 constructed based on the environment parameters 171 includes an environment control module 210 and a reward calculation module 220.

  The environment control module 210 manages the state of the environment in reinforcement learning, and simulates a state transition. The environment control module 210 has a simulation management module 211, and holds an environment state 212 as an internal parameter.

  The environment state 212 is a parameter indicating the current environment state. The simulation management module 211 simulates a state transition based on the behavior 252 output from the agent module 132.

  The reward calculation module 220 calculates the reward 251 based on the state 250 and outputs the calculated reward 251 to the agent module 132.

  The agent module 132 constructed based on the agent parameters 172 includes an optimizer 230 and a policy controller 240.

  The policy controller 240 holds policy internal data 241 corresponding to the policy. The optimizer 230 holds update data 231 for updating a policy and optimizer internal data 232. The update data 231 stores data including a state parameter, a reward parameter, and an action parameter.

  Here, the internal operation of the sub-process controller 130 will be described.

  The environment module 131 is in a waiting state until the state confirmation flag 253 or the action 252 is received.

  When receiving the state confirmation flag 253, the environment control module 210 of the environment module 131 outputs the state 250 set in the environment state 212 to the agent module 132. At this time, data indicating the reward 251 is not output.

  The environment module 131 may output the reward 251 corresponding to “0” to the agent module 132 together with the state 250. In this case, the reward calculation module 220 calculates “0” based on the state 250 output from the environment control module 210.

  When the action 252 is received, the environment control module 210 of the environment module 131 inputs the action 252 and the environment state 212 to the simulation management module 211, executes a simulation, and calculates the state 250. The environment control module 210 sets the calculated state 250 as the environment state 212 and outputs the calculated state 250 to the reward calculation module 220.

  The reward calculation module 220 of the environment module 131 calculates the reward 251 based on a predetermined calculation method using the state 250 as an input. The calculation method is included in, for example, the learning condition parameter.

  The environment module 131 outputs the state 250 and the reward 251 to the agent module 132.

  The agent module 132 first outputs a state confirmation flag 253 to the environment module 131. The agent module 132 receives the status 250 from the environment module 131 as a response to the status confirmation flag 253. At this time, the optimizer 230 of the agent module 132 sets an initial value in the update data 231. Specifically, the optimizer 230 adds, to the update data 231, data in which the state parameter is the received state 250, the behavior parameter is “none”, and the reward parameter is “no reward”.

  The policy controller 240 of the agent module 132 selects the action 252 based on the state 250 and the policy internal data 241, and outputs the action 252 to the environment module 131. When the optimizer 230 of the agent module 132 receives the status 250 and the reward 251 from the environment module 131 as a response to the action 252, the optimizer 230 updates the update data 231. Specifically, the optimizer 230 adds, to the update data 231, data that is the state 250 received by the state parameter, the behavior 252 output by the behavior parameter, and the reward 251 received by the reward parameter.

  The optimizer 230 determines whether the policy internal data 241 needs to be updated. For example, when the optimizer 230 receives the status 250, it determines that the policy internal data 241 needs to be updated. The optimizer 230 may determine that the policy internal data 241 needs to be updated each time the status 250 is received, or may update the policy internal data 241 when the status 250 is received a certain number of times. It may be determined that it is necessary to do so.

  When it is determined that the policy internal data 241 needs to be updated, the optimizer 230 updates the optimizer internal data 232 based on the update data 231. Further, the optimizer 230 updates the policy internal data 241 based on the updated optimizer internal data 232.

  The sub-process controller 130 according to the first embodiment determines whether or not to store the learning history during the execution of the simulation in the reinforcement learning. When it is determined that the learning history is stored in the history DB 150, the sub-process controller 130 stores the internal parameters and the like held by the environment module 131 and the agent module 132 in the history DB 150 as the learning history.

  As for each module of the computer, a plurality of modules may be combined into one module, or one module may be divided into a plurality of modules for each function. For example, the sub process controller 130 may include the score determination module 140.

  FIG. 3 is a diagram illustrating an example of a data structure of the learning condition parameter information 170 according to the first embodiment.

  The learning condition parameter information 170 includes a learning mode 301, a learning frequency 302, an upper limit frequency 303, a transition condition 304, presentation information 305, a storage condition 306, and a selection method 307.

  The learning mode 301 is a field for storing a value indicating a learning method of reinforcement learning. The number of times of learning 302 is a field for storing the number of times of execution of the reinforcement learning indicating the timing of saving the policy. The upper limit number 303 is a field for storing an upper limit value of the number of executions of the reinforcement learning of an arbitrary simulation difficulty level. The transition condition 304 is a field for storing information for adjusting the difficulty of the simulation. The presentation information 305 is a field that stores a value that specifies information to be output as a processing result of reinforcement learning. The storage condition 306 is a field for storing a value designating data to be stored in the history DB 150. The selection method 307 is a field that stores a selection method of a learning history to be used.

  The learning condition parameter information 170 may include a field for setting the definition of the evaluation value.

  FIG. 4 is a diagram illustrating an example of a data structure of the environment parameter 171 according to the first embodiment.

  The environment parameter 171 includes a time step 401, a relational expression 402, an equation 403, a state 404, and a reward 405.

  The time step 401 is a field that stores a value that specifies a state transition interval. The relational expression 402 and the expression 403 are fields for storing information that defines an environment, such as a mathematical expression. The status 404 is a field for storing information that defines the status of the environment. The reward 405 is a field for storing information, such as a mathematical expression, that defines a reward calculation method.

  FIG. 5 is a diagram illustrating an example of a data structure of the agent parameter 172 according to the first embodiment.

  The agent parameter 172 includes a policy internal variable 501 and an optimizer internal variable 502.

  The policy internal variable 501 is a field for storing a value of a variable set in the policy internal data 241. The optimizer internal variable 502 is a field for storing a value of a variable set in the optimizer internal data 232.

  The weight coefficient of the neural network corresponding to the policy is stored in the policy internal variable 501 shown in FIG. The optimizer internal variable 502 stores parameters α and β of the gradient method and a parameter η that controls the update frequency.

  FIG. 6 is a diagram illustrating an example of a data structure of the history relationship management information 123 according to the first embodiment.

  The history relationship management information 123 is data for managing the relationship between learning histories as a tree structure, and includes an entry including a node ID 601, a parent node ID 602, a child node ID 603, a difficulty coefficient 604, and a search flag 605. . One entry corresponds to one learning history.

  The node ID 601 is a field for storing identification information of a node corresponding to the learning history. The parent node ID 602 is a field for storing identification information of the parent node. The child node ID 603 is a field for storing identification information of the child node. The difficulty coefficient 604 is a field for storing the difficulty of the simulation executed to obtain the learning history. The search flag 605 stores a flag indicating whether the learning history can be used. “ON” indicates that the learning history can be used, and “OFF” indicates that the learning history cannot be used. Note that a blank column indicates that the node has not been determined.

  FIG. 7 is a diagram illustrating an example of a data structure of the learning result DB 160 according to the first embodiment.

  The learning result DB 160 includes an entry including a result ID 701, a policy internal variable 702, and a cumulative reward 703. One entry corresponds to an optimal learning result calculated by reinforcement learning of any difficulty.

  The result ID 701 is a field for storing identification information for identifying an entry in the learning result DB 160. The policy internal variable 702 is a field for storing the policy internal data 241 output as a learning result. The cumulative reward 703 is a field for storing a cumulative reward that is an evaluation value for evaluating a learning result. The cumulative reward is also used as an index for determining whether or not over-learning has occurred. In addition to the cumulative reward, a key performance evaluation index (KPI) can be used as an evaluation value. A plurality of KPIs may exist.

  FIG. 8 is a diagram illustrating an example of a data structure of the history DB 150 according to the first embodiment.

  The history DB 150 includes an entry including a history ID 801, a model parameter 802, and an output parameter 803. One entry corresponds to a learning result of reinforcement learning of any difficulty.

  The history ID 801 is a field for storing identification information for identifying an entry in the history DB 150. The model parameters 802 are a group of fields for storing parameters input for executing reinforcement learning of an arbitrary difficulty level. The model parameters 802 include environment parameters 811 and agent parameters 812. The output parameter 803 is a group of fields that stores a learning result calculated by executing reinforcement learning of an arbitrary difficulty level. The output parameters 803 include a policy internal variable 821 and a cumulative reward 822.

  Note that the entry may include a field for storing a learning condition or the like. Further, the model parameters 802 may include only environmental parameters.

  FIG. 9 is a flowchart illustrating an outline of processing executed by the computer 100 according to the first embodiment.

  When receiving the learning condition parameter information 170 from the terminal 101 (step S101), the computer 100 sets a model parameter (transition model parameter) based on the learning condition parameter information 170 (step S102) and executes reinforcement learning. (Step S103). At this point, the computer 100 sets the environment parameters 171 for realizing the environment with the lowest simulation difficulty. In the reinforcement learning, when the output of the learning history is detected, the learning history is stored in the history DB 150.

  The computer 100 performs a score determination process at an arbitrary timing (step S104).

  The computer 100 determines whether or not the optimal policy at any simulation difficulty level has been calculated based on the processing result of the score determination processing (Step S105). Here, the optimal policy is a policy calculated in a state where over-learning or a decrease in learning efficiency has not occurred, and means a policy that maximizes rewards and satisfies constraint conditions.

  If it is determined that the optimal policy has not been calculated, the computer 100 determines whether to use the learning history (step S106). It is determined whether or not the optimal policy has not been calculated due to the occurrence of over-learning or a decrease in learning efficiency.

  If it is determined that the learning history is not used, that is, if it is determined that learning is to be continued with the current parameters, the computer 100 sets a new model parameter based on the learning condition parameter information 170 and the learning result (step (S102), reinforcement learning is executed (step S103).

  Specifically, the policy internal data 241 at the time of the previous execution of the reinforcement learning is set in the agent parameter 172 included in the model parameter.

  If it is determined that the learning history is to be used, the computer 100 selects a learning history to be used (step S107), sets a new model parameter based on the learning condition parameter information 170 and the learning history (step S102), and enhances. The learning is performed (step S103).

  For example, the computer 100 calculates an agent parameter 172 reflecting a policy internal variable included in the learning history, and calculates an environment parameter 171 reflecting the environment parameter 171 included in the learning history. For example, an agent parameter 172 in which a policy internal variable included in the learning history is set as the policy internal variable 501 is calculated.

  The learning history may be reflected on only one of the environment parameter 171 and the agent parameter 172.

  If it is determined in step S105 that the optimal policy has been calculated, the computer 100 determines whether to change the simulation difficulty level (step S108).

  If it is determined that the simulation difficulty level is not changed, the computer 100 ends the processing. This indicates that the optimal policy at the target simulation difficulty level has been obtained.

  If it is determined that the simulation difficulty level is changed, the computer 100 changes the simulation difficulty level (step S109).

  Specifically, the computer 100 calculates the agent parameter 172 based on the policy calculated by the previous reinforcement learning, and further calculates the environment parameter 171 of the environment for realizing the simulation with high difficulty.

  Thereafter, the computer 100 sets the changed model parameters (Step S102), and executes the reinforcement learning (Step S103).

  The reinforcement learning algorithm according to the first embodiment has the following features.

  (Feature 1) When the computer 100 executes a simulation with a low difficulty and executes a simulation with a changed difficulty, the computer 100 calculates model parameters based on a learning result calculated from reinforcement learning before the change in the difficulty. Set. Thereby, efficient reinforcement learning can be realized, and the time required for learning can be reduced.

  (Feature 2) In the reinforcement learning of any difficulty level, the computer 100 executes the reinforcement learning again using the learning result of the past reinforcement learning without using the learning result of the previous reinforcement learning. As a result, it is possible to suppress the execution of the reinforcement learning in which the increase of the accumulated reward (evaluation value) is not expected, and it is possible to suppress the continuation of the reinforcement learning when the overlearning occurs.

  In order to realize the processing of (Feature 2), the computer 100 stores the learning history in the history DB 150 at an arbitrary timing.

  FIGS. 10A and 10B are flowcharts illustrating processing executed by the learning controller 120 according to the first embodiment. When the learning controller 120 receives an external input, the learning controller 120 executes a process described below. The learning controller 120 receives any of the learning condition parameter information 170, the optimal policy notification, the continuation instruction, the history use instruction, and the history update notification as an external input.

  The learning controller 120 determines whether or not the learning condition parameter information 170 has been received (Step S201).

  When it is determined that the learning condition parameter information 170 has been received, the learning controller 120 initializes the total number of learning times and the history relation management information 123 (Step S202).

  Specifically, the learning controller 120 sets the total number of times of learning to “0”. Further, the learning controller 120 deletes all entries of the history relationship management information 123, adds one entry, and sets “1” to the node ID 601 of the added entry.

  Next, the learning controller 120 calculates an initial model parameter based on the learning condition parameter information 170 (Step S203).

  In step S203, the learning controller 120 calculates a difficulty coefficient indicating the simulation difficulty when calculating the model parameters. The learning controller 120 refers to the history relationship management information 123 and sets the calculated difficulty coefficient as the difficulty coefficient 604 of the added entry. Further, the learning controller 120 holds the identification information of the root node as a pointer.

  Next, the learning controller 120 outputs the initial model parameters to the sub-process controller 130 (Step S204). After that, the learning controller 120 shifts to a waiting state and ends the processing. At this time, the learning controller 120 outputs the identification information set in the node ID 601 of the added entry together with the initial model parameters.

  When it is determined in step S201 that the learning condition parameter information 170 has not been received, the learning controller 120 determines whether an optimal policy notification has been received (step S205).

  When it is determined that the optimal policy notification has been received, the learning controller 120 updates the history relationship management information 123 based on the update determination list included in the optimal policy notification (Step S206). The update determination list is a list of identification information of the nodes. The update determination list will be described with reference to FIG.

  Specifically, the learning controller 120 refers to the update determination list, and sets “ON” to the search flag 605 of the entry corresponding to the node to which the exclusion flag indicating that the node is to be excluded from selection is not added. Further, the learning controller 120 sets “OFF” to the search flag 605 of the entry corresponding to the node to which the exclusion flag has been added. In the following description, a node that is registered in the update determination list and has not been given an exclusion flag is referred to as a candidate node.

  Next, the learning controller 120 determines whether to change the simulation difficulty level (step S207).

  For example, the learning controller 120 determines whether or not some of the values included in the previously output environment parameter 171 match the target value.

  When some values included in the previously output environment parameter 171 do not match the target values, the learning controller 120 determines that the simulation difficulty level is to be changed.

  When it is determined that the simulation difficulty level is changed, the learning controller 120 calculates a new model parameter for realizing the environment in which the simulation difficulty level has been changed (Step S208). Specifically, the following processing is executed.

  The learning controller 120 selects one node from the candidate nodes. Here, it is assumed that the node with the largest accumulated reward is selected. The learning controller 120 holds the identification information of the selected node as a pointer.

  The learning controller 120 calculates a new environment parameter 171 based on the learning condition parameter information 170 and the environment parameter 171 included in the learning history corresponding to the selected node. Further, the learning controller 120 calculates a new agent parameter 172 based on the policy internal data included in the learning history corresponding to the selected node. The learning controller 120 calculates a difficulty coefficient indicating the simulation difficulty based on the environment parameter 171.

  The learning controller 120 adds an entry to the history relationship management information 123, sets the identification information to the node ID 601 of the added entry, sets the identification information of the node set to the pointer to the parent node ID 602, and sets the difficulty coefficient A difficulty coefficient is set to 604.

  The learning controller 120 sets the identification information set in the node ID 601 of the added entry to the child node ID 603 of the entry corresponding to the node identification information set in the pointer. The above is the description of the process in step S208.

  Next, the learning controller 120 outputs a new model parameter to the sub-process controller 130 (Step S209). After that, the learning controller 120 shifts to a waiting state and ends the processing. At this time, the learning controller 120 outputs the identification information set in the node ID 601 of the entry added together with the new model parameter.

  If it is determined that the simulation difficulty level is not changed, the learning controller 120 shifts to the waiting state and ends the processing.

  If it is determined in step S205 that the optimal policy notification has not been received, the learning controller 120 determines whether a continuation instruction has been received (step S211).

  When it is determined that the continuation instruction has been received, the learning controller 120 updates the history relationship management information 123 (Step S212).

  Specifically, the learning controller 120 specifies the entry corresponding to the node registered in the update determination list, and sets “OFF” to the search flag 605 of the specified entry.

  Next, the learning controller 120 determines whether or not the total number of times of learning is equal to or less than the upper limit number (Step S213). That is, it is determined whether to continue the reinforcement learning based on the current model parameters.

  When it is determined that the total number of times of learning is larger than the upper limit number of times, the learning controller 120 shifts to a waiting state and ends the processing.

  When it is determined that the total number of learning times is equal to or less than the upper limit number, the learning controller 120 calculates an updated model parameter reflecting the learning result of the previous reinforcement learning (step S214). Specifically, the following processing is executed.

  The learning controller 120 calculates an agent parameter 172 for setting the policy internal data 241 at the time of executing the previous reinforcement learning as an initial value. The learning controller 120 calculates the same environment parameters 171 as those in the previous reinforcement learning.

  The learning controller 120 adds an entry to the history relationship management information 123, sets the identification information to the node ID 601 of the added entry, sets the identification information of the node set to the pointer to the parent node ID 602, and sets the difficulty coefficient At 604, the difficulty coefficient of the previous reinforcement learning is set.

  The learning controller 120 sets the identification information set in the node ID 601 of the added entry to the child node ID 603 of the entry corresponding to the node identification information set in the pointer.

  Further, the learning controller 120 holds the identification information set in the node ID 601 of the added entry as a pointer. The above is the description of the process in step S214.

  Next, the learning controller 120 outputs the updated model parameters to the sub process controller 130 (Step S215). After that, the learning controller 120 shifts to a waiting state and ends the processing. At this time, the learning controller 120 outputs the identification information set in the node ID 601 of the added entry together with the update model parameter.

  If it is determined in step S211 that the continuation instruction has not been received, the learning controller 120 determines whether a history use instruction has been received (step S216).

  When it is determined that the history use instruction has been received, the learning controller 120 updates the history relationship management information 123 (Step S217).

  Specifically, the learning controller 120 refers to the update determination list, sets “ON” to the search flag 605 of the entry corresponding to the node to which the exclusion flag is not assigned, and sets the search flag 605 to the node to which the exclusion flag is assigned. The search flag 605 of the corresponding entry is set to “OFF”.

  Next, the learning controller 120 executes a node selection process for selecting a learning history to be used (Step S218). Specifically, the following processing is executed.

  The learning controller 120 refers to the history relationship management information 123 and specifies an entry corresponding to the identification information of the node set in the pointer. The learning controller 120 sets the specified entry as a reference, and selects a node according to the search method set in the selection method 307. The learning controller 120 holds the identification information of the selected node as a pointer.

  For example, when the selection method 307 is “depth priority”, the learning controller 120 selects a node whose difficulty coefficient 604 matches the difficulty coefficient of the specified node. Note that nodes for which the search flag 605 is “OFF” and that are blank are excluded from search targets. When there are a plurality of applicable nodes, the learning controller 120 refers to the history DB 150 and searches for an entry that matches the identification information of the node whose history ID 801 is specified. The learning controller 120 selects a node corresponding to the entry with the largest accumulated reward.

  As another selection method, the learning controller 120 selects a node whose parent node matches the parent node of the node corresponding to the pointer, or a node with the largest accumulated reward. In the first embodiment, since the learning history includes the environment parameter 171, it is possible to execute simulations having different degrees of difficulty.

  The learning controller 120 adds an entry to the history relationship management information 123, sets the identification information to the node ID 601 of the added entry, sets the identification information of the node set to the pointer to the parent node ID 602, and sets the difficulty coefficient A difficulty coefficient is set to 604. The same value as the difficulty coefficient of the node before updating the pointer is set in the difficulty coefficient 604.

  The learning controller 120 refers to the history relation management information 123, and sets the identification information set in the node ID 601 of the added entry in the child node ID 603 of the entry corresponding to the identification information of the node set in the pointer. The above is the description of the process in step S218.

  Next, the learning controller 120 refers to the history DB 150, searches for an entry corresponding to the selected node, calculates a model parameter based on the searched entry, and outputs the model parameter to the sub-process controller 130 as a restored model parameter. (Step S219). After that, the learning controller 120 shifts to a waiting state and ends the processing. At this time, the learning controller 120 outputs the identification information set in the node ID 601 of the added entry together with the restoration model parameters. Specifically, the following processing is executed.

  The learning controller 120 calculates an agent parameter 172 reflecting a policy internal variable included in the learning history. For example, an agent parameter 172 in which a policy internal variable included in the learning history is set as the policy internal variable 501 is calculated.

  When the difficulty of the current simulation is the same as the difficulty of the simulation corresponding to the learning history, the learning controller 120 uses the current environment parameter 171 as it is. When the difficulty level of the current simulation is different from the difficulty level of the simulation corresponding to the learning history, the learning controller 120 calculates an environment parameter 171 reflecting the environment parameter 171 included in the learning history.

  That is, when the difficulty of the current simulation is the same as that of the simulation corresponding to the learning history, model parameters having different agent parameters 172 are calculated. When the difficulty of the current simulation differs from the difficulty of the simulation corresponding to the learning history, model parameters having different environment parameters 171 and agent parameters 172 are calculated.

  If the learning history does not include the environment parameter 171, the current environment parameter 171 is calculated. The above is the description of the process in step S219.

  If it is determined in step S216 that the continuation instruction has not been received, that is, if it is determined that the history update notification has been received, the learning controller 120 updates the history relationship management information 123 (step S220). Specifically, the following processing is executed.

  The learning controller 120 adds an entry to the history relationship management information 123, sets the identification information to the node ID 601 of the added entry, sets the identification information of the node set to the pointer to the parent node ID 602, and sets the difficulty coefficient At 604, a difficulty coefficient of the reinforcement learning being executed is set.

  The learning controller 120 refers to the history relationship management information 123, and sets the identification information set in the node ID 601 of the added entry in the child node ID 603 of the entry corresponding to the node identification information set in the point.

  The learning controller 120 outputs the identification information set in the node ID 601 of the added entry to the sub process controller 130. The above is the description of the process in step S220.

  Next, the learning controller 120 adds the number of times of learning included in the history update notification to the total number of times of learning (step S221). After that, the learning controller 120 shifts to a waiting state and ends the processing.

  The learning controller 120 periodically refers to the history relation management information 123, deletes the entry for which the search flag 605 is set to “OFF”, and deletes the corresponding entry from the history DB 150. Good.

  FIG. 11 is a flowchart illustrating a process executed by the sub-process controller 130 according to the first embodiment. When receiving the model parameters from the learning controller 120, the sub-process controller 130 executes processing described below.

  The sub-process controller 130 constructs an environment module 131 and an agent module 132 based on the received model parameters (Step S301).

  The sub-process controller 130 executes a simulation using the environment module 131 and the agent module 132 (Step S302). In the simulation, acquisition of a current state, selection of an action, and update of a state are performed. In the first embodiment, the policy is updated for each simulation. Note that the sub-process controller 130 (optimizer 230) may update the policy when the learning end condition is satisfied.

  The sub process controller 130 determines whether the storage condition is satisfied (step S303).

  For example, the sub-process controller 130 determines that the storage condition is satisfied when the learning end condition is satisfied, or when the number of times of executing the simulation matches the value of the learning number 302. When the policy internal data 241 is updated, it may be determined that the storage condition is satisfied.

  If it is determined that the storage condition is not satisfied, the sub-process controller 130 proceeds to step S306.

  If it is determined that the storage condition is satisfied, the sub-process controller 130 stores the model parameters and the learning result in the history DB 150 (Step S304).

  Specifically, the sub-process controller 130 adds an entry to the history DB 150. The sub-process controller 130 sets the identification information of the node notified from the learning controller 120 to the history ID 801 of the added entry. Thereby, the entry of the history DB 150 and the entry of the history relationship management information 123 are associated with each other. Further, the sub-process controller 130 registers the identification information of the node in the update determination list.

  Next, the sub-process controller 130 outputs a history update notification to the learning controller 120 (Step S305). The sub-process controller 130 shifts to a waiting state until the identification information of the node is input from the learning controller 120. If the node identification information has been input, the sub-process controller 130 proceeds to step S306.

  Next, the sub-process controller 130 determines whether or not a learning end condition is satisfied (step S306).

  For example, the sub-process controller 130 determines that the learning end condition is satisfied when the number of times of execution of the simulation matches the value of the number of times of learning 302 or when the updated state matches the end state.

  If it is determined that the learning end condition is not satisfied, the sub-process controller 130 returns to step S302.

  When it is determined that the learning end condition is satisfied, the sub-process controller 130 outputs a score determination request to the score determination module 140 (Step S307). Thereafter, the sub-process controller 130 ends the processing. Note that the score determination request includes an update determination list.

  FIG. 12 is a flowchart illustrating a process performed by the score determination module 140 according to the first embodiment. When receiving a score determination request from the sub-process controller 130, the score determination module 140 executes processing described below.

  The score determination module 140 starts a loop processing of the update determination list (step S401).

  Specifically, the score determination module 140 selects a target node from the nodes registered in the update determination list.

  Next, the score determination module 140 constructs an environment module 131 and an agent module 132 in the score determination module 140 (Step S402).

  Specifically, the score determination module 140 acquires the parameters of the environment module 131 and the parameters of the agent module 132 with reference to the history DB 150 based on the identification information of the target node. The score determination module 140 constructs an environment module 131 and an agent module 132 using the acquired parameters.

  Next, the score determination module 140 calculates an evaluation value by executing a simulation using the environment module 131 and the agent module 132 (step S403).

  Specifically, the score determination module 140 repeatedly executes the simulation until the termination condition is satisfied, and calculates the cumulative reward. In the simulation, control is performed so that the policy is not updated.

  Next, the score determination module 140 determines whether the accumulated reward is larger than a threshold (Step S404).

  When a plurality of types of evaluation values are set, the score determination module 140 determines whether or not a criterion defined from a combination of evaluation values is satisfied.

  When it is determined that the cumulative reward is larger than the threshold, the score determination module 140 updates the learning result DB 160 (Step S405), and then proceeds to Step S407.

  Specifically, the score determination module 140 adds an entry to the learning result DB 160, and sets identification information in the result ID 701 of the added entry. The score determination module 140 sets the policy internal data 241 acquired as a parameter of the agent module 132 in the policy internal variable 702 of the added entry, and sets the calculated cumulative reward in the cumulative reward 703 of the entry.

  When it is determined that the accumulated reward is equal to or less than the threshold, the score determination module 140 assigns an exclusion flag to the target node (Step S406), and then proceeds to Step S407.

  In step S407, the score determination module 140 determines whether the processing has been completed for all nodes registered in the update determination list (step S407).

  When it is determined that the processing has not been completed for all the nodes registered in the update determination list, the score determination module 140 returns to step S401, and selects a new target node.

  When it is determined that the processing has been completed for all nodes registered in the update determination list, the score determination module 140 determines whether or not an optimal policy exists (step S408).

  Specifically, the score determination module 140 determines whether there is a node to which the exclusion flag has not been added among the nodes registered in the update determination list. When there is a node to which the exclusion flag has not been added among the nodes registered in the update determination list, the score determination module 140 determines that the optimal policy exists.

  If it is determined that the optimal policy exists, the score determination module 140 outputs an optimal policy notification to the learning controller 120 (Step S409). Thereafter, the score determination module 140 ends the processing. The optimal policy notification includes an update determination list.

  When it is determined that the optimal policy does not exist, the score determination module 140 determines whether the use condition is satisfied (step S410).

  For example, when the rate of increase of the cumulative reward is smaller than the threshold, or when the cumulative reward of each learning result (node) is smaller than the threshold, the score determination module 140 determines that the use condition is satisfied.

  In the first embodiment, when the possibility that the optimal policy is calculated is low even if the reinforcement learning is continued, or when over-learning occurs, the execution of the reinforcement learning based on the current model parameters is stopped and a new model is created. Start reinforcement learning based on parameters.

  When it is determined that the use condition is satisfied, the score determination module 140 outputs a history use instruction to the learning controller 120 (Step S411). Thereafter, the score determination module 140 ends the processing. The history use instruction includes an update determination list.

  When it is determined that the use condition is not satisfied, the score determination module 140 outputs a continuation instruction to the learning controller 120 (Step S412). Thereafter, the score determination module 140 ends the processing. The continuation instruction includes an update determination list.

  FIG. 13 and FIG. 14 are diagrams illustrating an example of a GUI displayed by the computer 100 according to the first embodiment. FIG. 13 shows a GUI 1300 presented for the user to make various settings for reinforcement learning. FIG. 14 shows a GUI 1400 presented for the user to confirm the transition of learning.

  The GUI 1300 includes a learning mode column 1301, a learning frequency column 1302, an upper limit frequency column 1303, a transition condition column 1304, a presentation information column 1305, a storage target column 1306, a selection method column 1307, and a setting button 1308.

  The learning mode column 1301 is a column for selecting a learning mode. In the first embodiment, a drop-down list for selecting “On-Policy”, “Off-Policy”, or the like is presented.

  The learning frequency column 1302 is a column for setting the learning frequency. The upper limit number column 1303 is a column for setting the upper limit number.

  The transition condition column 1304 is a column for setting a method of adjusting the difficulty of the simulation.

  The presentation information column 1305 is a column for setting information to be output as a result of reinforcement learning. In the first embodiment, a drop-down list for selecting a policy, an action, and the like is presented.

  The storage target column 1306 is a column for setting a storage target. The box in the storage target column 1306 can be added or deleted as needed.

  The selection method column 1307 is a column for setting a selection method of a learning history to be used.

  The setting button 1308 is a button for setting the value of each column in the computer 100. When the user operates the button, the learning condition parameter information 170 including the value of each column is input to the computer 100.

  The GUI 1400 includes a display button 1401, a setting button 1402, a history-related display field 1403, and a detail display field 1404.

  The display button 1401 is a button for displaying a history-related display column 1403. The setting button 1402 is a button for reflecting the operation result of the history-related display column 1403 on the computer 100.

  The history relation display column 1403 includes a difficulty coefficient 1411 and a history structure 1412. The difficulty coefficient 1411 displays a difficulty coefficient. In the history structure 1412, a graph indicating the connection relationship between the nodes of the history relationship management information 123 is displayed. As shown in FIG. 14, a graph for forming a layer for each difficulty coefficient is displayed. The black circle indicates that the search flag 605 is “OFF”. Dotted circles indicate nodes where the learning result is not stored.

  The detail display column 1404 is a column for displaying a learning result corresponding to a node. When the user positions the cursor on a node in the history structure 1412, a learning result corresponding to the node is displayed in a detail display column 1404.

  The user can use the detail display field 1404 to select a node to be selected as a selection target and a node to be selected. When the user operates the setting button 1402 after performing the above operation, the value of the search flag 605 of the history relationship management information 123 is updated.

(Modification)
FIG. 15 is a diagram illustrating a modification of the configuration of the system according to the first embodiment.

  The system includes a computer 1500, a plurality of computers 1510, a computer 1520, and a terminal 1530. The computer 1500, the computer 1510, and the computer 1520 are connected to each other via a network 1550. The computer 1500 and the terminal 1530 are connected directly or via a network.

  The computer 1500 has a learning controller 120, the computer 1510 has a sub-process controller 130 and a history DB 150, and the computer 1520 has a score determination module 140 and a learning result DB 160. In this system, a plurality of computers 1510 execute reinforcement learning in parallel.

  The learning controller 120 holds the history relation management information 123 for each computer 1510. Further, a field for storing identification information of the computer 1510 is added to the learning result DB 160.

  Note that the present invention is not limited to the above-described embodiment, and includes various modifications. Further, for example, in the above-described embodiment, the configuration has been described in detail in order to explain the present invention in an easily understandable manner, and the present invention is not necessarily limited to those having all the configurations described above. Further, a part of the configuration of each embodiment can be added, deleted, or replaced with another configuration.

  Further, each of the above-described configurations, functions, processing units, processing means, and the like may be partially or entirely realized by hardware, for example, by designing an integrated circuit. The present invention can also be realized by software program codes for realizing the functions of the embodiments. In this case, a storage medium storing the program code is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the function of the above-described embodiment, and the program code itself and the storage medium storing the program code constitute the present invention. As a storage medium for supplying such a program code, for example, a flexible disk, a CD-ROM, a DVD-ROM, a hard disk, an SSD (Solid State Drive), an optical disk, a magneto-optical disk, a CD-R, a magnetic tape, A non-volatile memory card, ROM, or the like is used.

  Further, the program code for realizing the functions described in the present embodiment can be implemented by a wide range of programs or script languages such as assembler, C / C ++, perl, Shell, PHP, and Java (registered trademark).

  Furthermore, by distributing the program code of the software for realizing the functions of the embodiment via a network, the program code is stored in a storage unit such as a hard disk or a memory of a computer or a storage medium such as a CD-RW or a CD-R. Alternatively, a processor included in a computer may read out and execute the program code stored in the storage unit or the storage medium.

  In the above-described embodiment, the control lines and the information lines are considered to be necessary for the explanation, and do not necessarily indicate all the control lines and the information lines on the product. All components may be interconnected.

Reference Signs List 100 computer 101 terminal 110 processor 111 memory 112 network interface 120 learning controller 121 environment rollback controller 122 policy rule back controller 123 history relation management information 130 sub process controller 131 environment module 132 agent module 140 score determination module 150 history DB
160 learning result DB
170 Learning condition parameter information 171 Environment parameter 172 Agent parameter 200 Environment module 201 Agent module 210 Environment control module 211 Simulation management module 212 Environment state 220 Reward calculation module 230 Optimizer 231 Update data 232 Optimizer internal data 234 Optimizer internal data 240 Policy controller 241 Internal policy data 250 State 251 Reward 252 Action 253 State confirmation flag 1300, 1400 GUI

Claims (10)

A learning control method in a computer system for learning a policy for determining a control content of a process for controlling an object,
The computer system includes:
A learning control unit for calculating a transition model parameter obtained by partially changing a target model parameter for realizing a simulation for selecting the control content of the processing based on an arbitrary policy;
The simulation based on the transition model parameter input from the learning control unit or the transition model parameter calculated based on the result of the previous simulation is executed a plurality of times, and the policy is updated based on the result of the simulation. A learning device for performing a learning process;
A history database that manages, as a learning history, information related to the transition model parameters and the policy updated by the execution of the simulation,
The learning control method includes:
A first step in which the learning device stores the learning history in the history database at an arbitrary timing;
A second determining unit that determines whether it is necessary to execute the simulation using the learning history based on the evaluation value of the policy updated by the simulation executed an arbitrary number of times; Steps and
When it is determined that it is necessary to execute the simulation using the learning history, the learning device performs the simulation based on the transition model parameter calculated based on the use learning history selected from the history database. A third step of executing a plurality of times and updating the policy based on a result of the simulation. The learning control method according to claim 1, wherein
A fourth step in which the learning device determines whether to update the transition model parameter based on the evaluation value;
A fifth step of, when it is determined that the learning device updates the transition model parameter, outputting an instruction to update the transition model parameter to the learning control unit;
A step of, when the learning control unit receives the update instruction of the transition model parameter, updating the current transition model parameter, and outputting the updated transition model parameter to the learning device. A learning control method characterized in that: The learning control method according to claim 2, wherein
The learning control unit manages history relationship management information indicating a relationship between the learning histories,
The first step is
When the learning device stores a new learning history in the history database, notifying the learning control unit of a storage notification of the new learning history,
The learning control unit updates the history relationship management information so as to be associated with the learning history used to calculate the transition model parameter used in the simulation from which the new learning history is generated. , A learning control method. The learning control method according to claim 3, wherein
The third step is
The learning control unit, based on the history relationship management information, selecting the use learning history,
A step in which the learning control unit calculates the transition model parameter based on the use learning history and outputs the transition model parameter to the learning device. The learning control method according to claim 4, wherein
The third step is
The learning control unit calculates a new transition model parameter by changing a part of the transition model parameter used in the previous simulation based on the use learning history, and outputs the calculated new transition model parameter to the learning device. ,
A learning control method, wherein the learning device executes the simulation based on the new transition model parameter a plurality of times. A computer system for learning a policy for determining a control content of a process for controlling an object,
A learning control unit for calculating a transition model parameter obtained by partially changing a target model parameter for realizing a simulation for selecting the control content of the processing based on an arbitrary policy;
The simulation based on the transition model parameters input from the learning control unit or the transition model parameters calculated based on the previous simulation result is executed a plurality of times, and the policy is updated based on the simulation result. A learning device for performing a learning process;
A history database that manages, as a learning history, information related to the transition model parameters and the policy updated by the execution of the simulation,
The learning device includes:
At any time, storing the learning history in the history database,
Based on the evaluation value of the policy updated by the simulation executed an arbitrary number of times, determine whether it is necessary to execute the simulation using the learning history,
When it is determined that it is necessary to execute the simulation using the learning history, the learning device performs the simulation based on the transition model parameters calculated based on the use learning history selected from the history database. A computer system which is executed a plurality of times and updates the policy based on a result of the simulation. The computer system according to claim 6, wherein:
The learning device includes:
Based on the evaluation value, determine whether to update the transition model parameters,
If it is determined to update the transition model parameters, output an instruction to update the transition model parameters to the learning control unit,
The computer system, wherein the learning control unit updates a current transition model parameter when receiving an instruction to update the transition model parameter, and outputs the updated transition model parameter to the learning device. The computer system according to claim 7, wherein
The learning control unit manages history relationship management information indicating a relationship between the learning histories,
The learning device, when storing a new learning history in the history database, notifies the learning control unit of a storage notification of the new learning history,
The learning control unit, when receiving the storage notification of the new learning history, associates the learning history with the learning history used to calculate the transition model parameter used in the simulation from which the new learning history is generated. A computer system for updating the history relation management information as described above. The computer system according to claim 8, wherein
The learning control unit includes:
When it is determined that the simulation using the learning history needs to be performed by the learning device, the use learning history is selected based on the history relationship management information,
A computer system, wherein the transition model parameters are calculated based on the use learning history and output to the learning device. The computer system according to claim 9, wherein:
The learning control unit calculates a new transition model parameter by changing a part of the transition model parameter used in the previous simulation based on the use learning history, and outputs the new transition model parameter to the learning device.
The computer system, wherein the learning device executes the simulation based on the new transition model parameter a plurality of times.