JP6764143B2 - Reinforcement learning equipment, reinforcement learning methods, and reinforcement learning programs - Google Patents

Description

The present invention relates to a reinforcement learning device, a reinforcement learning method, and a reinforcement learning program .

Reinforcement learning is a type of machine learning that deals with the problem of agents in an environment observing their current state and deciding what action to take. Agents get rewards from the environment by choosing actions. Reinforcement learning learns policies that maximize rewards through a series of actions.

As one of such reinforcement learnings, Non-Patent Document 1 proposes "hierarchical reinforcement learning" composed of two reinforcement learning agents, Meta-Controller and Controller. Suppose that there are multiple states between the start point and the goal (Goal), and you want to reach the goal by the shortest route from the start point. Here, each state is also called a subgoal. In Non-Patent Document 1, Meta-Controller presents to Controller the sub-goal to be achieved next from a plurality of sub-goals given in advance (however, in Non-Patent Document 1, it is described as "goal"). are doing.

Meta-Controller is also called a high-level planner, and Controller is also called a low-level planner. Therefore, in Non-Patent Document 1, the high-level planner determines a specific subgoal from a plurality of subgoals, and the Robel planner determines the actual action based on the specific subgoal. The high-level planner has a sub-goal determination unit. Let ε be a variable between 0 and 1 (0 ≤ ε ≤ 1). The initial value of the variable ε is 1. While the number of trials is small, the value of the variable ε is close to 1. As the number of trials increases, the experience value accumulates, so the value of the variable ε gradually decreases as it approaches zero. In this situation, the subgoal determination unit randomly selects a specific subgoal from a plurality of subgoals with a probability of ε, and empirically selects a specific subgoal with a probability of (1-ε).

Further, Patent Document 1 discloses a "learning control device" capable of realizing learning of an autonomous agent capable of selecting a target task by itself and expanding its ability one after another. .. The learning control device disclosed in Patent Document 1 includes a prediction unit, an evaluation unit, a control unit, and a planning unit. The prediction unit performs predictive learning using the environment as a teacher. The evaluation unit observes prediction errors by the prediction unit, calculation errors by the planning unit, and behavior control errors by the control unit, and based on these, the achievement status on the sensor state space that the autonomous agent should achieve is determined. Set and give the target achievement state (target state) to the planning department. The planning department plans the action sequence from the current state to the target state given by the evaluation department. The control unit executes learning by the planning unit with the planning and the environment as teachers, and controls the actions of the autonomous agent. When the learning of the prediction unit and the control unit is sufficiently advanced, the target state can be hierarchized as one action.

The prediction unit constantly learns the relationship between the actions taken by itself and changes in the environment (changes in sensor input), and the prediction accuracy of the prediction unit is improved by executing even an erroneous plan. The predictor can perform predictive learning without being cursed by the dimension by utilizing the ability of the function approximator to withstand the learning of large-scale samples and large-dimensional inputs. Further, even if an erroneous plan is executed by the plan generated by the immature prediction unit, the prediction unit can experience a weak state space and improve the prediction performance. By using the heuristics search method by the planning department, even if the input dimension increases and the state space becomes large, the search combination explodes compared to the case of using Q-learning or dynamic programming. It is possible to suppress the storage. In addition, it is possible to generalize the control unit by repeating the learning of the success sequence.

Patent Document 2 provides a method for improving the movement of a robot that is operated based on a predetermined set of actions. Patent Document 2 describes the following. A composite action is generated by combining at least two actions in the set of original actions stored in the action library. After the policy has been learned, including compound actions, many of those compound actions cannot be used. One reason is that it can violate robot constraints such as joints limits and collisions, and the other reason is that compound actions do not provide any benefit in certain scenarios. Therefore, such nonsensical compound actions are removed from the action library in order to keep the action library small for the reasons mentioned above.

Japanese Unexamined Patent Publication No. 2006-268812 Japanese Unexamined Patent Publication No. 2016-196079

Tejas D. Kulkarni, et al. "Hierarchical Deep Reinforcement Learning: Integrating Tmporal Abstraction and Intrinsic Motivation." 30th Conference on Nural Information Processing Systems (NIPS 2016), Barcelona, Spein.

It is assumed that the operation of a complicated system is learned by hierarchical reinforcement learning as disclosed in Non-Patent Document 1. In this case, the number of subgoals increases. In other words, the search space for searching the subgoal becomes enormous. For learning, the subgoal determination department needs to try and error various subgoals. As a result, the hierarchical reinforcement learning method disclosed in Non-Patent Document 1 has a problem that the learning time becomes very long.

Patent Document 1 also only discloses hierarchical reinforcement learning. Further, Patent Document 1 does not disclose or suggest any starting point. Further, in Patent Document 1, the target (goal) is not set in advance, the evaluation unit sets the target state based on the above-mentioned error observation, and the planning unit changes from the current state to the target state. We are planning an action sequence to reach it. Therefore, Patent Document 1 does not disclose or suggest the concept of a plurality of subgoals from the starting point to reaching the goal.

Patent Document 2 merely describes that a meaningless compound action is removed from the action library.

An object of the present invention is to provide a reinforcement learning device, a reinforcement learning method, and a reinforcement learning program capable of solving the above-mentioned problems.

Strengthening one aspect of the present invention, comprising a high level planner to determine the subgoal in either et part subgoal from the start point to reach the goal, and a low level planner to determine the actions in the subgoals of the portion What learning apparatus der has a storage device for previously storing a rule to select the sub-goal, the high level planner, in accordance with the rules, that having a sub-goal decision unit for determining a sub-goal of the part.

In one embodiment of the present invention, the data processing device determines a part of the subgoals according to the rule of selecting the subgoals from the subgoals from the start point to the goal, and determines the action according to the part of the subgoals. , I reinforcement learning methods der, may be stored in advance in the storage device a rule to select the sub-goal, said data processing apparatus, in accordance with the rules, that determine the subgoals of the part.

One form of the present invention, the data processing apparatus, and a high level planner function of determining the subgoal in either et part subgoal from the start point to reach the goal, the low level to determine the actions in the subgoals of the portion What reinforcement learning program der to achieve a planner function, and may be stored in advance in the storage device a rule to select the sub-goal, said high level planner function, in accordance with the rules, that determine the subgoals of the portion ..

According to the present invention, the number of trials can be reduced and the learning time can be shortened.

It is a schematic block diagram of the target system to which the reinforcement learning device which concerns on embodiment of this invention is applied. It is a block diagram which shows the hardware structure of the reinforcement learning apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows one detailed configuration example of a task knowledge and a sub-goal determination part shown in FIG. It is a flow chart which shows the determination flow of the sub-goal in the high-level planner shown in FIG. It is a flow chart which shows the subgoal decision flow in a high level planner when a knowledge task consists only of a priority rule. It is a flow chart which shows the subgoal decision flow in a high level planner when a knowledge task consists only of a suppression rule. It is a block diagram which shows one configuration example which creates a knowledge task from a task rule. It is a figure which shows the field of the 13 × 13th square in which an item is arranged. It is a figure which shows an example of the item arrangement in the field shown in FIG. It is a figure which shows the Craft rule which is a task rule in 1st Example. It is a figure which shows an example of a priority rule. It is a figure which shows an example of the suppression rule. It is a figure which shows the comparison result (experimental result) of the reinforcement learning apparatus in this embodiment, and the hierarchical reinforcement learning (prior art) disclosed in Non-Patent Document 1. It is a figure which shows "background knowledge" and "objective state" necessary for deriving a priority rule by using an inference device. It is a figure which shows an example of the priority rule derived by an inference device. It is a figure which shows an example of the "non-purpose state" defined in an inference device. It is a figure which shows an example of the suppression rule derived by an inference device.

FIG. 1 is a schematic configuration diagram of a target system to which the reinforcement learning device according to the embodiment of the present invention is applied.

The target system has a starting point S and a goal G. In the target system, there are N (N is an integer of 3 or more) subgoals between the start point S and the goal G. In the example shown in FIG. 1, three subgoals represented by A, B, and C are typically illustrated as N subgoals. Here, subgoal A is referred to as a first subgoal, subgoal B is referred to as a second subgoal, and subgoal C is referred to as a third subgoal.

In the target system, task rules to be satisfied from the start point S to the goal G are defined. In the case of the target system shown in FIG. 1, the goal G must be reached from the starting point S via the first subgoal A, the second subgoal B, and the third subcall C in the shortest time according to the task rule. Here is an example of what you can do.

However, in general, the target system has a large number of subgoals, and as a result, the search space for searching the subgoals becomes enormous. Therefore, in the reinforcement learning device according to the present embodiment, as will be described later, the search range is narrowed down by utilizing the task knowledge to improve the learning efficiency.

[Embodiment]
FIG. 2 is a block diagram showing a hardware configuration of the reinforcement learning device 100 according to the embodiment of the present invention. The illustrated reinforcement learning device 100 can be realized by a computer operated by program control.

The illustrated reinforcement learning device 100 is a device that searches for a subgoal in a target system as shown in FIG.

The reinforcement learning device 100 includes an input device 101 for inputting data, an output device 102 for outputting data, a storage device 104 for storing programs and data described later, and a data processing device 105 for processing data. ..

The output device 102 includes a display device such as an LCD (Liquid Crystal Display) or a PDP (Plasma Display Panel), or a printer. The output device 102 has a function of displaying various information such as an operation menu and printing out the final result in response to an instruction from the data processing device 105.

The storage device 104 includes a memory such as a hard disk, a read-only memory (ROM), and a random access memory (RAM). The storage device 104 has a function of storing processing information (described later) and a program 201 required for various processes in the data processing device 105.

The data processing device 105 includes a microprocessor such as an MPU (micro processing unit) and a central processing unit (CPU). The data processing device 105 has a function of reading the program 201 from the storage device 104 and realizing various processing units that process data according to the program 201.

The main processing unit realized by the data processing device 105 includes a high-level planner 301 and a Robel planner 302.

The high level planner 301 determines a specific subgoal from the above N subgoals, as will be described later. The low level planner 302 determines the actual action according to its particular subgoal.

That is, the high level planner 301 sequentially instructs the low level planner 302 to subgoals up to the target G as shown in FIG. The low level planner 302 operates a simulator (not shown) to achieve its indicated subgoal. The low level planner 302 feeds back the result of achieving the goal to the high level planner 301.

More specifically, the storage device 104 stores the task knowledge 202 in advance, as will be described later. The task knowledge 202 is knowledge determined as will be described later based on the above task rules.

The high level planner 301 includes a subgoal determination unit 303. Using the task knowledge 202, the subgoal determination unit 303 narrows down the above N subgoals to M (M is an integer of 1 or more smaller than N), and preferentially among the M subgoal candidates. Determine the specific subgoal above.

FIG. 3 is a block diagram showing a detailed configuration example of the task knowledge 202 and the subgoal determination unit 303.

The illustrated task knowledge 202 includes a priority rule 204 and a suppression rule 206. Priority rule 204 is a rule that gives priority to a subgoal that contributes to the achievement of goal G, which is obtained based on the above task rule. On the other hand, the suppression rule 206 is a rule for suppressing a sub-goal that does not contribute to the achievement of the goal G, which is obtained based on the above task rule.

The sub-goal determination unit 303 includes a priority selection unit 305 and a sub-goal check unit 307. The priority selection unit 305 preferentially extracts and selects M subgoal candidates from the N subgoals according to the priority rule 204.

More specifically, the priority selection unit 305 includes a subgoal candidate extraction unit 311 and a subgoal selection unit 313. The subgoal candidate extraction unit 311 extracts M subgoal candidates from N subgoals according to the priority rule 204. The sub-goal selection unit 313 preferentially selects one sub-goal from the M sub-goal candidates and outputs the selected sub-goal.

The subgoal check unit 307 determines whether the selected subgoal is OK or NG as the specific subgoal based on the suppression rule 206. If OK, the subgoal check unit 307 outputs the selected subgoal as a specific subgoal. It is assumed that the subgoal check unit 307 determines that the result is NG. In this case, with a predetermined probability p, the subgoal selection unit 313 redoes the subgoal selection. Further, with a probability (1-p), the subgoal check unit 307 outputs the NG subgoal as it is as a specific subgoal.

[Description of operation]
Next, the operation of determining the subgoal in the high-level planner 301 (that is, the operation of the subgoal determination unit 303) will be described in detail with reference to the flowchart of FIG.

Here, ε is a variable between 0 and 1 (0 ≦ ε ≦ 1), as described above. While the number of trials is small, the value of the variable ε is close to 1. As the number of trials increases, the experience value is accumulated, so the value of the variable ε gradually decreases so as to approach 0. In this situation, the subgoal determination unit 303 according to the present embodiment selects and determines a specific subgoal using the task knowledge 202 with a probability of ε, as will be described later. On the other hand, as in the case of the prior art, the subgoal determination unit 303 empirically selects a specific subgoal with a probability of (1-ε) (step S101) and determines a specific subgoal (step S102).

Next, the operation when a specific subgoal is selected and determined by using the task knowledge 202 with a probability of ε will be described.

First, the subgoal candidate extraction unit 311 extracts M subgoal candidates from N subgoals according to the priority rule 204 (step S103). Next, the subgoal selection unit 313 selects one subgoal from the extracted M subgoal candidates and outputs the selected subgoal (step S104).

Next, the subgoal check unit 307 determines whether the selected subgoal is OK or NG as a specific subgoal based on the suppression rule 206 (step S105). If it is OK, the subgoal check unit 307 determines the selected subgoal as a specific subgoal (step S102). On the other hand, when the subgoal check unit 307 determines that the subgoal is NG, the subgoal selection unit 313 returns to step S104 with a predetermined probability p, and the subgoal selection unit 313 reselects one subgoal from the extracted M subgoal candidates. Further, with a probability (1-p), the subgoal check unit 307 outputs the NG subgoal as it is as a specific subgoal.

In the above embodiment, the task knowledge 202 includes, but is not limited to, priority rule 204 and suppression rule 206. For example, the task knowledge 202 may consist only of the priority rule 204 or the suppression rule 206 only.

FIG. 5 is a flow chart showing a subgoal determination flow in the high level planner 301 when the knowledge task 202 consists of only the priority rule 204. As is clear from FIG. 5, step S105 is omitted from FIG.

FIG. 6 is a flow chart showing a subgoal determination flow in the high level planner 301 when the knowledge task 202 consists of only the suppression rule 206. As is clear from FIG. 6, step S103 is omitted from FIG. In this case, the subgoal selection unit 313 randomly selects one subgoal from the N subgoals (step S104).

The priority rule 204 and the suppression rule 206 may be manually created. Alternatively, as shown in FIG. 7, the inference device 320 may be used to dynamically create the priority rule 204 and the suppression rule 206 from the task rule 210.

[Explanation of effect]
Next, the effect of this embodiment will be described.

According to the embodiment of the present invention, the number of trials can be reduced and the learning time can be shortened. The reason is that the task knowledge is used to narrow down the search range (subgoal candidates to be selected) and speed up learning.

Each part of the reinforcement learning device 100 may be realized by using a combination of hardware and software. In the form of combining hardware and software, a reinforcement learning program is deployed in RAM (random access memory), and hardware such as a control unit (CPU (central processing unit)) is operated based on the reinforcement learning program. Each part is realized as various means. Further, the reinforcement learning program may be recorded on a recording medium and distributed. The reinforcement learning program recorded on the recording medium is read into the memory via wired, wireless, or the recording medium itself, and operates the control unit and the like. Examples of recording media include optical disks, magnetic disks, semiconductor memory devices, hard disks, and the like.

To explain the above embodiment in another expression, the computer operated as the reinforcement learning device 100 is a priority selection unit 305 (subgoal candidate extraction unit 311 and subgoal selection unit 313) based on the reinforcement learning program developed in the RAM. , And it can be realized by operating as a subgoal check unit 307.

Next, a first embodiment when the reinforcement learning device 100 according to the embodiment of the present invention is applied to a specific target system will be described. The target system according to the first embodiment is a craft game that imitates Minecraft. That is, the task of collecting / crafting materials in the field and crafting the target item.

The mission definition in the first embodiment will be described below. The purpose (goal) is to collect materials and make rabbit_stew. However, if you don't collect the ingredients in the proper order, you will end up with something different (eg stick, mushroom_stew) and you will fail.

You don't get sequential rewards, you only get rewards for success or failure.

As shown in FIG. 8, various items are arranged in the field of 13 × 13 squares. FIG. 9 shows an example of the item arrangement. Therefore, the materials are in eight fixed locations (subgoals). Always start the mission from the same initial state (start).

The action is only movement in four directions. Collection / crafting is automatic. FIG. 10 is a diagram showing a Craft rule, which is a task rule 210 of the toy task in this example. The toy task in this example has a minimum of 39 moves.

In the first embodiment, the task knowledge 202 is manually created. Priority rule 204 in this example is a rule of the position of the material that is the premise of the target item. Further, the suppression rule 206 is a rule of the position of the material that is the premise of the failed item.

FIG. 11 is a diagram showing an example of the priority rule 204. FIG. 12 is a diagram showing an example of the suppression rule 206.

FIG. 13 is a diagram showing a comparison result (experimental result) of the reinforcement learning device 100 in the present embodiment and the hierarchical reinforcement learning (prior art) disclosed in Non-Patent Document 1. In FIG. 13, the horizontal axis shows the number of trials, and the vertical axis shows the task success rate. Further, in FIG. 13, the one-dot dashed line indicates the experimental result of the prior art, the two-dot dashed line indicates the experimental result using only the suppression rule 206 as the task knowledge 202, and the dashed line indicates only the priority rule 204 as the task knowledge 202. The experimental results using the above are shown. The solid line shows the experimental result in which the priority rule 204 and the suppression rule 206 are used together as the task knowledge 202.

As is clear from FIG. 13, the learning speed of the reinforcement learning device 100 according to the present embodiment in which the priority rule 204 and the suppression rule 206 are used together as the task knowledge 202 is about 5 as compared with the learning speed of the prior art. It can be seen that the speed is doubled. Further, even when only the priority rule 204 is used as the task knowledge 202, it can be seen that the learning speed of the reinforcement learning device 100 according to the present embodiment is higher than the learning speed of the prior art.

In the first embodiment described above, the priority rule 204 and the suppression rule 206 are manually created. On the other hand, in the second embodiment described below, the inference device 320 is used to dynamically create the priority rule 204 and the suppression rule 206.

First, an example of deriving the priority rule 204 using the inferior 320 will be described. However, the task rule 210 is different from that shown in FIG. 10 in order to simplify the explanation.

FIG. 14 is a diagram showing "background knowledge" and "objective state" necessary for deriving the priority rule 204 using the inference device 320. Action predicates (goto) and state predicates (have) are defined as predicates. In FIG. 14, the pickup rule of “background knowledge” expresses the item arrangement shown in FIG.

The inference device 320 applies backward inference under the "background knowledge" and the "objective state" shown in FIG. 14, and sets the derived action predicate as the priority rule 204. FIG. 15 is a diagram showing an example of the priority rule 204 thus derived.

Next, an example of deriving the suppression rule 206 using the inference device 320 will be described.

The inference device 320 defines the "non-purpose state" shown in FIG. 16, and the action predicate leading to the definition is set as the suppression rule 206. In FIG. 16, since the condition of the branched portion is AND, it becomes a non-purpose state if all the conditions are satisfied.

FIG. 17 is a diagram showing an example of the suppression rule 206 thus derived. FIG. 17 shows three suppression rules. The first suppression rule shows that it is the suppression rule that goes to SW when it has red_mushroom and brown_mushroom and does not have a bowl. The same applies to the following two suppression rules.

As described above, the inferior 320 can dynamically create the priority rule 204 and the suppression rule 206 from the task rule 210.

The specific configuration of the present invention is not limited to the above-described embodiment, and is included in the present invention even if there is a change within the scope not departing from the gist of the present invention.

Although the present invention has been described above with reference to the embodiment (Example), the present invention is not limited to the above embodiment (Example). Various changes that can be understood by those skilled in the art can be made within the scope of the present invention in terms of the structure and details of the present invention.

Some or all of the above embodiments may also be described, but not limited to:

(Appendix 1) A high-level planner that determines a specific subgoal from N (N is an integer of 3 or more) subgoals from the starting point to reaching the goal; a low that determines the actual action according to the specific subgoal. With a level planner; the high level planner uses task knowledge to narrow down the N subgoals to M (M is an integer of 1 or more smaller than N) subgoal candidates, and the M subgoals. A subgoal determination unit for preferentially determining the specific subgoal from among the candidates is provided, and the task knowledge is knowledge determined based on a task rule to be satisfied from the start point to the arrival of the goal. Reinforcement learning device.

(Appendix 2) The task knowledge includes a priority rule that prioritizes subgoals that contribute to the achievement of the goal, which is obtained based on the task rule, and the subgoal determination unit determines the N subgoals according to the priority rule. The reinforcement learning device according to Appendix 1, which includes a priority selection unit that preferentially extracts and selects the M subgoal candidates from among the above.

(Appendix 3) The priority selection unit includes a subgoal candidate extraction unit that extracts the M subgoal candidates from the N subgoals according to the priority rule; and one preferentially one from the M subgoal candidates. The reinforcement learning device according to Appendix 2, which includes a subgoal selection unit that selects a subgoal and outputs the selected subgoal;

(Appendix 4) The task knowledge further includes a suppression rule for suppressing a subgoal that does not contribute to the achievement of the goal, which is obtained based on the task rule, and the subgoal determination unit is based on the suppression rule. The reinforcement learning device according to Appendix 3, further comprising a subgoal check unit for determining whether the selected subgoal is OK or NG as the specific subgoal.

(Appendix 5) The subgoal selection unit reselects the one subgoal from the M subgoal candidates with a predetermined probability when the subgoal check unit determines that the subgoal is NG. The described reinforcement learning device.

(Appendix 6) The high-level planner determines a specific subgoal from N (N is an integer of 3 or more) subgoals from the starting point to the goal, and the low-level planner follows the specific subgoal. It is a reinforcement learning method that determines the actual action, and the subgoal determination unit of the high-level planner uses task knowledge to convert the N subgoals into M (M is an integer greater than or equal to 1 smaller than N). A task that narrows down the candidates and preferentially determines the specific subgoal from the M subgoal candidates, and the task knowledge defines the rules to be satisfied from the start point to the goal. Reinforcement learning method, which is knowledge determined based on rules.

(Appendix 7) The task knowledge includes a priority rule that prioritizes a subgoal that contributes to the achievement of the goal, which is obtained based on the task rule, and the priority selection unit of the subgoal determination unit determines the priority rule according to the priority rule. The reinforcement learning method according to Appendix 6, wherein the M subgoal candidates are preferentially extracted and selected from the N subgoals.

(Appendix 8) The sub-goal candidate extraction unit of the priority selection unit extracts the M sub-goal candidates from the N sub-goals according to the priority rule, and the sub-goal selection unit of the priority selection unit has the M sub-goal candidates. The reinforcement learning method according to Appendix 7, wherein one subgoal is preferentially selected from the subgoal candidates and the selected subgoal is output.

(Appendix 9) The task knowledge further includes a suppression rule for suppressing a subgoal that does not contribute to the achievement of the goal, which is obtained based on the task rule, and the subgoal check unit of the subgoal determination unit controls the suppression rule. The reinforcement learning method according to Appendix 8, wherein it is determined whether the selected subgoal is OK or NG as the specific subgoal based on the above.

(Appendix 10) When the subgoal selection unit is determined to be NG by the subgoal check unit, the one subgoal is reselected from the M subgoal candidates with a predetermined probability. The described reinforcement learning method.

(Appendix 11) A high-level planner procedure for determining a specific subgoal from N (N is an integer of 3 or more) subgoals from the starting point to reaching the goal, and the actual action is determined according to the specific subgoal. A low-level planner procedure and an enhanced learning program recording medium that records an enhanced learning program that causes a computer to execute the procedure. The high-level planner procedure uses task knowledge to set the N subgoals to M (M is N). A subgoal determination procedure is provided in which the specific subgoal is preferentially determined from the M subgoal candidates by narrowing down the subgoal candidates (less than 1 or more integers), and the task knowledge is described from the start point. An enhanced learning program recording medium that is knowledge determined based on task rules that must be met before reaching a goal.

(Appendix 12) The task knowledge includes a priority rule that prioritizes subgoals that contribute to the achievement of the goal, which is obtained based on the task rule, and the subgoal determination procedure follows the N subgoals according to the priority rule. The reinforcement learning program recording medium according to Appendix 11, which includes a priority selection procedure for preferentially extracting and selecting the M subgoal candidates from among them.

(Appendix 13) The priority selection procedure includes a subgoal candidate extraction procedure for extracting the M subgoal candidates from the N subgoals and one preferentially one from the M subgoal candidates according to the priority rule. The reinforcement learning program recording medium according to Appendix 12, which includes a subgoal selection procedure for selecting a subgoal and outputting the selected subgoal.

(Appendix 14) The task knowledge further includes a suppression rule for suppressing a subgoal that does not contribute to the achievement of the goal, which is obtained based on the task rule, and the subgoal determination procedure is based on the suppression rule. The reinforcement learning program recording medium according to Appendix 13, further comprising a subgoal check procedure for determining whether the selected subgoal is OK or NG as the specific subgoal.

(Appendix 15) The subgoal selection procedure reselects the one subgoal from the M subgoal candidates with a predetermined probability when the subgoal check procedure determines that the subgoal is NG. The described reinforcement learning program recording medium.

The reinforcement learning device according to the present invention can be applied to applications such as a plant operation support system and an infrastructure operation support system.

100 Reinforcement learning device 101 Input device 102 Output device 104 Storage device 105 Data processing device 201 Program 202 Task knowledge 204 Priority rule 206 Suppression rule 210 Task rule 301 High level planner 302 Low level planner 303 Subgoal decision unit 305 Priority selection unit 307 Subgoal check Part 311 Subgoal candidate extraction part 313 Subgoal selection part 320 Inference device

Claims (10)

A high-level planner that determines a specific subgoal from N (N is an integer of 3 or more) subgoals from the start point to the goal.
With a low-level planner that decides the action according to the specific subgoal,
It has a storage device that stores task knowledge that represents the rules for selecting the subgoal in advance.
The high level planner, before using Kita risk knowledge, the N (the M N is smaller than an integer of 1 or more) of the subgoal M Search in pieces subgoal candidates, from among the M sub-goal candidate It has a subgoal determination unit that preferentially determines the specific subgoal.
Reinforcement learning device. The task knowledge includes a priority rule that prioritizes a subgoal that contributes to the achievement of the goal, which is obtained based on the rule.
The subgoal determination unit includes a priority selection unit that preferentially extracts and selects the M subgoal candidates from the N subgoals according to the priority rule.
The reinforcement learning device according to claim 1. The priority selection unit is
A subgoal candidate extraction unit that extracts the M subgoal candidates from the N subgoals according to the priority rule, and a subgoal candidate extraction unit.
A subgoal selection unit that preferentially selects one subgoal from the M subgoal candidates and outputs the selected subgoal.
2. The reinforcement learning device according to claim 2. The task knowledge further includes a suppression rule that suppresses a subgoal that does not contribute to the achievement of the goal, which is obtained based on the rule.
The subgoal determination unit further includes a subgoal check unit that determines whether the selected subgoal is OK or NG as the specific subgoal based on the suppression rule.
The reinforcement learning device according to claim 3. When the subgoal check unit determines that the subgoal is NG, the subgoal selection unit reselects the one subgoal from the M subgoal candidates with a predetermined probability.
The reinforcement learning device according to claim 4. The data processor determines a specific subgoal from N (N is an integer of 3 or more) subgoals from the start point to the goal.
A reinforcement learning method that determines actions according to the specific subgoal.
The task knowledge representing the rule for selecting the subgoal is stored in the storage device in advance.
Using the task knowledge, the data processing device narrows down the N subgoals to M (M is an integer of 1 or more smaller than N), and preferentially among the M subgoal candidates. Reinforcement learning method for determining the specific subgoal. A high-level planner procedure that determines a specific subgoal from N (N is an integer of 3 or more) subgoals from the start point to the goal.
A low-level planner procedure that determines actions according to the specific subgoal,
Is a reinforcement learning program that lets a computer execute
In the high-level planner procedure, the N subgoals are narrowed down to M (M is an integer of 1 or more smaller than N) subgoal candidates by using the task knowledge representing the rule for selecting the subgoals. A reinforcement learning program including a subgoal determination procedure for preferentially determining the specific subgoal from among subgoal candidates. A high level planner to determine the subgoal in either et part subgoal to reach the goal from the start point,
It is a reinforcement learning device equipped with a low-level planner that decides actions according to some of the subgoals mentioned above .
It has a storage device that stores the rules for selecting the subgoal in advance.
The high-level planner is a reinforcement learning device having a subgoal determination unit that determines a part of the subgoals in accordance with the rules . The data processing device determines some subgoals from the subgoals from the start point to the goal according to the rules for selecting the subgoals.
This is a reinforcement learning method that determines actions according to some of the subgoals mentioned above .
The rules for selecting the subgoal are stored in the storage device in advance.
A reinforcement learning method in which the data processing device determines a part of the subgoals according to the rules . By the data processing apparatus, and a high level planner function of determining the subgoal in either et part subgoal from the start point to reach the goal,
It is a reinforcement learning program that realizes a low-level planner function that decides actions according to some of the subgoals mentioned above .
The rules for selecting the subgoal are stored in the storage device in advance.
The high-level planner function is an reinforcement learning program that determines some of the subgoals according to the rules .

Images