JP2020009122A - Control program, control method and system - Google Patents

Abstract

To provide a control program, a control method, and a system capable of selecting and instructing an action according to a task at an appropriate time interval.SOLUTION: A control method includes the steps of: (A) in a search tree for a tree search for selecting an action and a time, in which an action to be executed next and a time until the further next action to be executed are associated with an edge between nodes, performing search processing of searching for a node with a value of a first evaluation expression while adding a new node during a time period associated with an edge to the node selected in the search processing previously executed; and (B) causing a control object to execute the action associated with the edge to the node selected in the search processing executed this time.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an optimal control technique.

  Monte Carlo Tree Search (MCTS) determines the best action to be taken in the current state by representing the decision space with an additional growing search tree. The search tree is updated with a simulated new state (node) and an action (edge or edge) by random simulation. By repeatedly adding nodes and edges, a transition is made from the current state to a terminal state. Each node of the search tree holds an expected reward when proceeding from the state of the node. The expected reward of a node represents the average performance of all simulations going through that node.

MCTS simulations are divided into four phases: selection, expansion, roll-out, and backpropagation. FIG. 1 schematically shows the four phases. (1) In the selection phase, nodes are selected from the root until reaching nodes that have not yet been fully deployed. In the example of FIG. 1, an arrow indicates that a node two levels below the root node has been selected. (2) In the development phase, one node is added to the nodes selected in the selection phase. (3) In the rollout phase, a simulation is performed according to the default policy of rollout. (4) In the back propagation phase, the result of the simulation is propagated in the reverse direction, that is, from the leaf node to the root node. In the example of FIG. 1, a state in which the result of the simulation is propagated in the opposite direction to the added node, the selected node, its parent node, and the root node is schematically illustrated.
As mentioned above, a node represents the state of a simulated task, and an edge corresponds to the action performed to transition from the state of the current node to the state of a child node.

  Each simulation of the MCTS starts with the root node corresponding to the current state. In the above-described selection phase, the search tree is traversed based on a selection policy defined for the value of the node and the number of visits. Then, when reaching a node where a child node can be added, the search tree is extended by adding a new leaf node to the node. This addition of the leaf node is the processing of the development phase. Then, in the rollout phase, the rollout policy is applied from the state of the new leaf node. The simulation of the rollout is repeated the maximum number of times allowed or until an end condition is reached. As this rollout policy, a rollout policy called a straight-forward random action is widely used. Finally, the result of the simulation is back-propagated to update the information managed in the subtree from the leaf node to the root node.

  The key in the selection phase of MCTS is to balance the use of promising actions and the search of decision space, and the Upper Confidence Bounds applied to Trees (UCT) algorithm has given a good solution to it. It is used regularly.

  A typical MCTS is applied to a finite number of sequential decision problems. However, if the decision space is very large, there is a problem that the view of the search tree becomes very shallow because all actions are searched at least once from a certain state. Such a case occurs in an environment where the decision space is infinite and continuous. For such a problem, an approach called Hierarchical Optimistic Optimization applied to Tree (HOOT) has been proposed. HOOT integrates the HOO algorithm into tree search planning to go beyond the limits of UCT. HOO constitutes a selection policy for the bandit problem, and the selection policy is to improve the range (regret bounds) lost in the selection of the arm with respect to the result of the previous simulation. . HOO forms a general topological representation of a decision space by decomposing the decision space as a tree.

  When an action is queried, the HOO algorithm performs a search along a path having a maximum score of B-value calculated at a node. At the leaf nodes, actions are sampled in a range in the decision space represented by the leaf nodes. Then, two child nodes are added to the leaf node, each child node covering half of the range in decision space represented by the leaf node's parent node.

  HOOT is similar to UCT, but HOOT places a HOO, a bandit algorithm of continuous actions, at each node of the search tree. HOO is used to sample the decision space for selection of actions because it exceeds the limit of discrete actions in UCT.

  In the conventional MCTS, the time interval between the selection of an action and the selection of the next action is fixed. Therefore, in an environment where the state changes every moment, it is possible to cope by shortening the time interval. However, when the time interval becomes short, there is a problem that the number of times the simulation is executed is limited and a reliable search tree cannot be generated.

  Further, in the conventional HOOT, one type of action is mostly handled, and there is no specific proposal on how to handle two or more types of actions.

Remi Coulom, "Efficient selectivity and backup operators in monte-carlo tree search" In Interna-tional conference on computers and games, pages p72-83, Springer, 2006 Christopher R. Mansley, Ari Weinstein, and Michael L. Littman, "Sample-based planning for continuous action markov decision processes", In Proceed-ings of the 21st International Conference on Automated Planning and Scheduling, 2011 Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer, "Finite-time analysis of the multiarmed bandit problem", Machine learning, 47 (2-3): 235-256, 2002 Levente Kocsis and Csaba Szepesvari, "Bandit based monte-carlo planning", In Eu-ropean conference on machine learning, pages 282-293, Springer, 2006 Kristof Van Moffaert, "Multi-Criteria Reinforcement Learning for Sequential Decision Making Problems", 2016 Uitgeverij VUBPRESS Brussels University Press

  Therefore, an object of the present invention is to provide, as one aspect, a technique for selecting and instructing an action according to a task at an appropriate time interval.

  According to the control method of the present invention, (A) a tree for selecting an action and a time, in which an action to be executed next and a time until a next action to be executed is selected are associated with an edge between nodes. In a search tree for search, a search process of searching for a node with the value of the first evaluation expression while adding a new node is associated with an edge to the node selected in the previously executed search process. And (B) causing the control object to execute the action associated with the side to the node selected in the search process executed this time.

  According to an aspect, an action according to a task can be selected and indicated at an appropriate time interval.

FIG. 1 is a diagram for explaining the MCTS. FIG. 2 is a diagram illustrating an overview of the entire system according to the embodiment. FIG. 3 is a diagram illustrating a time sequence example according to the embodiment. FIG. 4 is a diagram illustrating an example of a search tree of the MCTS according to the present embodiment. FIG. 5 is a diagram depicting a processing flow of a processing executed by the Monte Carlo tree search unit; FIG. 6 is a diagram depicting a processing flow of a processing executed by the 2DHOOT unit; FIG. 7A is a diagram illustrating a first phase in a specific example of the processing. FIG. 7B is a diagram illustrating a second phase in a specific example of the processing. FIG. 7C is a diagram illustrating a third phase in a specific example of the processing. FIG. 7D is a diagram illustrating a fourth phase in a specific example of the processing. FIG. 7E is a diagram illustrating a first phase in a specific example of the processing. FIG. 8 is a block diagram of the computer device.

  FIG. 2 shows an outline of a system according to the present embodiment. The system according to the present embodiment includes a control device 100 and a control target 200 that continuously executes a task. The control device 100 functions as a control device for the control target 200.

The control device 100 includes a Monte Carlo tree search unit 110, a 2DHOOT unit 120, a simulation unit 130, and an interface unit 140. Interface unit 140 receives the time tau _i until selecting an action a _i and the next action to be executed at time t _i from Monte-Carlo tree searching unit 110, the action a _i to the control object 200 during the time tau _i Is executed. On the other hand, the interface unit 140 receives the state S _i of the task at the time t _i from the control target 200 and outputs it to the Monte Carlo tree search unit 110.

  The simulation unit 130 is a simulation model that duplicates the behavior of a task executed by the control target 200. Therefore, if the Monte Carlo tree search unit 110 specifies the state S and instructs to execute the action a during the time τ, the simulation unit 130 predicts the expected state S and the reward r. Is returned to the Monte Carlo tree search unit 110.

2DHOOT unit 120 receives an average reward r _a state S and MCTS search tree of the node of interest from Monte-Carlo tree searching unit 110 performs the processing described below, and returns an action a and time τ corresponding to them.

The Monte Carlo tree search unit 110 performs the processing described below in cooperation with the 2DHOOT unit 120 and the simulation unit 130, and selects the action a _i to be executed next and the action to be executed next. Determine τ _i .

A specific time sequence will be described with reference to FIG. Time progresses from left to right, with the upper row representing transitions in actions and the lower row representing transitions in time intervals. That is, at time t _i , the state of the task is actually S _i , but the state S ′ _i is estimated, and the Monte Carlo tree search unit 110 selects the action a _i and the time τ _i at time t _i . . At the time τ _i during which the control target 200 is executing the action a _i , the Monte Carlo tree search unit 110 estimates the state S ′ _{i + 1} by performing the processing described below (in FIG. 3, the MCTS (Represented as S ′ _{i + 1} ), action a _{i + 1} and time τ _{i + 1} are selected. However, at time t _{i + 1} after time tau _i from time t _i, the actual state of the task is S _{i + 1.} Such a sequence is repeated. Note that τ _i = τ _{i + 1} may not be as shown in FIG. 3 or may be the same, but is selected each time.

An example of a search tree searched by the Monte Carlo tree search unit 110 in the present embodiment will be described with reference to FIG. In the present embodiment, a pair of an action and a time is associated with an edge between nodes. The state of the child node is a state estimated when the action associated with the edge is applied from the state of the parent node during the time associated with the edge. In the example of FIG. 4, in the state S _i , during the time τ _i during which the action a _i is being executed, the next state S ′ _{i + 1} is estimated, and the action and time of the first candidate are estimated therefrom. A node transitioning with a pair (a ¹ _{i + 1} , τ ¹ _{i + 1} ), a node transitioning with a second candidate action / time pair (a ² _{i + 1} , τ ² _{i + 1} ), There is a node that transitions with a pair of action and time of three candidates (a ³ _{i + 1} , τ ³ _{i + 1} ).

In the present embodiment, the Monte Carlo tree search unit 110 adopts the UCT described above and selects a node that maximizes a value called UCB1. UCB1 is represented as follows. Note that it represents a child node j.
UCB1 = Xa _j + C × (2 logn / n _j ) ^1/2

Here, Xa _j represents the average of all compensation simulation through the child nodes j. n represents the number of visits of the parent node of the child node j, and n _j represents the number of visits of the child node j. C is a positive constant, and is determined experimentally.

  In FIG. 4, assuming that the values assigned to the three nodes that transit in the pair of the action and time of the first to third candidates are the values of UCB1, as shown by the arrows, the value of UCB1 is the most significant. A first candidate action and time pair for the edge connected to the larger node will be selected.

  In the present embodiment, the conventional problem is solved by extending and applying HOOT to two dimensions. That is, a pair of time and action is effectively sampled using 2DHOOT in a two-dimensional space spanned by time and action. In 2DHOOT, the inclusion relation of a region in a two-dimensional space is hierarchically shown, and another search tree having a node corresponding to each region is used. For example, a root node represents the entire two-dimensional space, four child nodes of the root node represent four large regions obtained by dividing the two-dimensional space into four, and four grandchild nodes of the child nodes represent large regions. It represents four middle regions divided into four, and four great-grandchild nodes of the grandchild node represent four small regions obtained by dividing the middle region into four.

  However, the number of such pairs is limited using known progressive widening (PW). The PW causes the MCTS to initially focus on a limited number of pairs to avoid making the search tree shallow. As the number of simulations increases, the number of pairs to be considered increases so as to cover the decision space more widely. At some point, if the number of child nodes reaches the maximum number of pairs allowed in relation to the number of times the node has been visited, the node is considered fully expanded.

  2DHOOT functions to add a filter for action-time pairs to be evaluated during the simulation. In 2DHOOT sampling, the selection of action and time pairs to be added to the search tree is directed to regions where promising pairs are more likely to be selected. As a result, the efficiency of selecting the action-time pair by the MCTS is improved even with a limited number of simulations. Specific examples will be described later in detail.

  Next, processing contents of the control device 100 will be described with reference to FIGS. 5 to 7E.

  First, with reference to FIG. 5, a process related to the MCTS in a state given to a task will be described.

  The Monte Carlo tree search unit 110 sets a root node as a current node (that is, a node of interest) (FIG. 5: step S1). The construction of the search tree performed every time τ (τ varies) selected in combination with the previously selected action starts from the root node.

  Then, the Monte Carlo tree search unit 110 determines whether or not the current node has been sufficiently expanded (step S3). Whether or not the image has been sufficiently developed is determined based on, for example, PW. If it is determined that the expansion is sufficient, the Monte Carlo tree search unit 110 sets the child node having the largest evaluation value UCB1 as the current node (step S5). Then, the process returns to step S3. The definition of UCB1 is the same as the conventional one.

On the other hand, when it is not determined to have been fully deployed, Carlo tree searching unit 110 calls the 2DHOOT unit 120 passes the state S and the average reward r _{a, (a,} τ) is sampled, the search tree A leaf node having the side of (a, τ) is added (step S7). The processing of the 2DHOOT unit 120 will be described later. Here, a pointer to the node sampled in 2DHOOT is stored in the added leaf node.

  Then, the Monte Carlo tree search unit 110 executes the rollout by passing the sampled (a, τ) and the state S and causing the simulation unit 130 to simulate the task (step S9). The simulation is performed using a model imitating the task executed by the control target 200. Then, the simulation unit 130 returns the state S and the reward r as a simulation result.

  In addition, the Monte Carlo tree search unit 110 executes a backup process for back-propagating the simulation result from the added leaf node (step S11). The back-propagation in this step is the same as in the related art, and a description thereof is omitted.

Then, the Monte Carlo tree search unit 110 determines whether or not the time τ _{i-1i-1 associated} with the edge to the previously selected node has elapsed since the start of the current processing (step S13). If the time τ _i-1 has not elapsed since the start of the current process, the process returns to step S1.

On the other hand, if the time τ _i-1 has elapsed from the start of the processing, the Monte Carlo tree search unit 110 calculates the evaluation value UCB1 for each node, selects the node having the largest evaluation value UCB1, and sends the node to that node. (A _i , τ _i ) associated with the side is selected (step S15).

Then, the Monte Carlo tree search unit 110 outputs a _i and τ _i to the interface unit 140, and the interface unit 140 instructs the control target 200 to execute the action a _i during the time τ _i ( Step S17). Control object 200 performs the tasks according to the instructions come returns the state S _i as a processing result of the task.

  By repeating such processing, it becomes possible to cause the control target 200 to execute an appropriately selected action at each appropriately selected time interval.

  Next, the processing contents of the 2D HOOOT unit 120 called in step S7 will be described.

  In the present embodiment, each node of the search tree in the MCTS includes a search tree in 2DHOOT. In the search tree in 2DHOOT, the root node holds an optimistic estimate of the quality of the entire action space (decision space), and its child nodes hold more accurate estimates of a smaller area in the action space. I have. In general, the child nodes of a parent node represent different regions of the same size, which cover all the regions represented by the parent node. According to the 2DHOOT algorithm, a leaf node is selected, and a pair of an action a and a time τ is sampled from a region represented by the leaf node.

  More detailed processing contents will be described with reference to FIG.

  First, the 2DHOOT unit 120 selects a root node (FIG. 6: step S31). Then, the 2DHOOT unit 120 sets the selected node as the current node (that is, the node of interest) (Step S33). Thereafter, the 2DHOOT unit 120 determines whether the current node has a child node whose score or the like has not been updated (step S35). If the current node has a child node whose score or the like has not been updated (step S35: Yes route), the 2D HOOOT unit 120 selects a child node whose score or the like has not been updated (step S37). Then, the process returns to step S33. As described above, in the loop L311, a lower node whose score or the like is not updated is recursively searched from the root node.

  On the other hand, when the current node reaches a node having no child node whose score or the like has not been updated (step S35: No route), the 2DHOOT unit 120 updates the score or the like of the current node (step S39).

Expected reward actions and time pair to be selected from the search tree 2DHOOT depends on the average reward r _a simulation was conducted in MCTS. Also, the value of a node in the 2DHOOT search tree is affected by all nodes belonging to the subtree. In other words, estimated compensation R ^e _i of the node i is updated repeatedly by the following equation.

Incidentally, j represents the child node of the node i, n _i is the target number of visits node i, n _j is the number of visits node j, R ^e _j is estimated compensation node j Represents

Also, U-Value is defined as follows based on the estimated compensation R ^e _i node i.

Here, n represents the number of repetitions of the processing in FIG. 6, and the constants v ₁ and ρ are v ₁ > 0 and 0 <ρ <1, and appropriate values are set experimentally. h _i represents the depth of the node i in the search tree.

The score B-value (also referred to as score B) is defined as follows.

Note that B _j is the score B-value of the child node j of the node i. Equation (3) expresses the maximum value of B _j and the minimum value of U _i for all the child nodes j, B _i . In addition, infinity is set for B _i and U _i of nodes that have not been sampled yet.

  In step S39, the values such as the score defined in this way are updated.

  Then, the 2DHOOT unit 120 determines whether or not the current node is the root node (Step S41). If the current node is not the root node, the 2DHOOT unit 120 selects a parent node (Step S43). Then, the process proceeds to step S33.

  As described above, the values such as the score are updated in the direction from the leaf node to the root node by the processing in the loop L31.

  On the other hand, when the current node is the root node, the updating of the values such as the scores has been completed, and the 2DHOOT unit 120 determines whether the current node is a leaf node (step S45). When the process first proceeds to step S45, it is determined that the current node is not a leaf node because the current node is the root node.

  If the current node is not a leaf node, the 2D HOOT unit 120 selects a child node having the largest score B-value and sets it as the current node (step S47). Also, the number of times the selected node is visited is incremented by one. Then, the process proceeds to step S45. Such a loop L33 is a process of searching for a path that follows the node having the largest score B-value to the leaf node.

On the other hand, when the current node is a leaf node, the 2DHOOT unit 120 samples (a, τ) in an area covered by the leaf node (step S49). For action a, values in the area [a _min , a _max ] covered by the selected leaf node are sampled. As for the time τ, values in the area [τ _min , τ _max ] covered by the selected leaf node are sampled.

  Then, the 2DHOOT unit 120 performs a process of expanding the search tree of the 2DHOOT (step S51). Specifically, four child nodes are added to the selected leaf node. The four child nodes correspond to each of the regions obtained by equally dividing the region covered by the selected leaf node into four. Note that, since the space is a two-dimensional space of the action a and the time τ, the division is performed by a power of the division number (= 2) for one dimension = 4, with the number of dimensions (= 2) as an index. Note that the number of divisions for one dimension is a natural number of 2 or more, and is determined by settings of a user or the like.

  Note that a 2DHOOT search tree backup is performed to update the number of times the selected node from the leaf node to the root node is visited. That is, the number of times visited is held for each node, and the number is updated.

  Then, the 2DHOOT unit 120 outputs the pointer to the leaf node and the sampled (a, τ) to the Monte Carlo tree search unit 110 (Step S53). This samples the more promising action and time pairs.

  A specific example of the processing will be described with reference to FIGS. 7A to 7E.

When the node p in the search tree of the MCTS is visited for the first time (upper part of (a) of FIG. 7A), the root node 1 of the search tree of 2DHOOT is selected (upper part of (b) of FIG. 7A), and the root node 1 is selected. The point 1 (a ¹ , τ ¹ ) is sampled from the entire region of the decision space defined by the action a and the time τ represented by (c) in FIG. 7A.

  In the present embodiment, both the action a and the time τ are continuous values within a finite range, but are shown here normalized to [0, 1].

  Then, expansion of the 2DHOOT search tree, that is, addition of four child nodes, is performed (lower part of (b) of FIG. 7A). These four child nodes correspond to any of the regions A to D having the same area without overlapping each other, as shown in FIG. 7C.

The sampled (a ¹ , τ ¹ ) is used to extend the MCTS search tree at node a. That is, as shown in the lower part of (a) of FIG. 7A, the side from the node p to the node a is associated with (a ¹ , τ ¹ ). When the value of the node a is updated, the value of the node 1 is also updated.

Next, when the node p in the search tree of the MCTS is expanded again (the upper part of (a) in FIG. 7B), the child node 2 of the root node 1 is selected in the search tree of 2DHOOT ((in FIG. 7B). b) (upper), point 2 (a ² , τ ² ) is sampled from area B represented by node 2 ((c) in FIG. 7B).

  Then, the 2DHOOT search tree is expanded, that is, four child nodes are added (lower part of (b) of FIG. 7B). These four child nodes correspond to any of the regions B1 to B4 which do not overlap each other and have the same area as shown in FIG. 7C.

The sampled (a ² , τ ² ) is used to extend the MCTS search tree at node b. That is, as shown in the lower part of (a) of FIG. 7B, the side from the node p to the node b is associated with (a ² , τ ² ).

Next, when the node p in the search tree of the MCTS is expanded again (the upper part of (a) of FIG. 7C), the child node 3 of the root node 1 is selected in the search tree of 2DHOOT ((FIG. b) (upper), point 3 (a ³ , τ ³ ) is sampled from area A represented by node 3 ((c) in FIG. 7C).

  Then, the 2DHOOT search tree is expanded, that is, four child nodes are added (lower part of (b) of FIG. 7C). As shown in FIG. 7C, the four child nodes correspond to any of the regions A1 to A4 which do not overlap each other and have the same area.

The sampled (a ³ , τ ³ ) is used to extend the MCTS search tree at node c. That is, as shown in the lower part of FIG. 7A, the side from the node p to the node c is associated with (a ³ , τ ³ ).

Such a process is repeated, and after adding five nodes a to e to the MCTS search tree, if the node p in the MCTS search tree is expanded again (the upper part of (a) in FIG. 7D) In the 2DHOOT search tree, the child node 6 of the node 2 is selected (the upper part of (b) of FIG. 7D), and the point 6 (a ⁶ , τ ⁶ ) is sampled from the area B1 represented by the node 6. ((C) of FIG. 7D).

  Then, expansion of the 2DHOOT search tree, that is, addition of four child nodes, is performed (lower part of (b) of FIG. 7D). These four child nodes correspond to any of the regions B11 to B14 which do not overlap each other and have the same area as shown in (c) of FIG. 7D.

The sampled (a ⁶ , τ ⁶ ) is used to extend the MCTS search tree at node f. That is, as shown in the lower part of (a) of FIG. 7D, the side from the node p to the node f is associated with (a ⁶ , τ ⁶ ).

  Up to this point, node a is associated with node 1, node b is associated with node 2, node c is associated with node 3, node d is associated with node 4, node e is associated with node 5, and node e is associated with node e. f and the node 6 are associated with each other.

  Note that in the 2DHOOT search tree, all child nodes of the root node are updated before going to the grandchild node hierarchy. Generally, all children of the node are updated before going from the node to the next lower level.

When the process is repeated and ten nodes a to l are added to the MCTS search tree, and the node p in the MCTS search tree is expanded again (the upper part of (a) in FIG. 7E). In the 2DHOOT search tree, the child node 11 of the node 7 is selected (the upper part of (b) of FIG. 7E), and the point 11 (a ¹¹ , τ ¹¹ ) is sampled from the area B23 represented by the node 11. ((C) of FIG. 7E).

  Then, the search tree of 2DHOOT is expanded, that is, four child nodes are added (lower part of (b) of FIG. 7E). These four child nodes correspond to any of the regions B231 to B234 having the same area without overlapping each other, as shown in FIG. 7E (c).

The sampled (a ¹¹ , τ ¹¹ ) is used to extend the MCTS search tree at node k. That is, as shown in the lower part of (a) of FIG. 7E, the side from the node p to the node k is associated with (a ¹¹ , τ ¹¹ ).

  Thus, the nodes in the 2DHOOT search tree are associated with the nodes in the MCTS search tree. When the value of the node in the MCTS search tree is updated, the value of the node in the 2DHOOT search tree is also updated.

  Each node added in the MCTS search tree holds a pointer to the selected leaf node in the 2DHOOT search tree for the parent node, and initializes a new 2DHOOT search tree. That is, the search tree of the MCTS includes a 2DHOOT search tree for selecting an action and time pair for each node, and a leaf node where the action and time pair is sampled in the 2DHOOT search tree for the parent node. Holding a pointer.

Pointer to the leaf node is used to update the average reward r _a. (1) As shown by the formula, the calculation of the estimated compensation R ^e _i of the node i in the search tree of 2DHOOT, average reward r _a is used. By using such an average reward r _a , the node selection in the 2DHOOT search tree is performed in the decision space toward an area such as an action that is expected to be executed well over a long period of time.

  Although the present embodiment has been described above, the present invention is not limited to this. For example, the functional block configuration example is merely an example, and may not match the program module configuration. As for the processing flow, as long as the processing result does not change, the processing order may be changed or a plurality of steps may be executed in parallel.

Further, in the example described above, the two-dimensional space is handled by the action a and the time τ, but it is possible to handle more dimensions by HOOT. That is, the _n- dimensional space may be handled by the actions a _{1 to an -1} and the time τ. The space may be an n-dimensional space excluding the time τ. In this case, when expanding the search tree, n-dimensional HOOT adds n number of child nodes to the number of divisions (natural numbers of 2 or more) specified for one dimension.

  Note that the control target unit 200 may be any device, but is a device that executes, for example, automatic operation of a vehicle and other continuous tasks.

The contents of the processing shown in FIG. 6 are represented by pseudo code as follows.
1: procedure UCTSEARCH (root, τ)
2: while time <τ do // Step S13
3: front = TreeSearch (root) // Step S1
4: reward = DefaultPolicy (front.state, front.horizon)
// Step S9
5: BackUp (front, reward) // Step S11
6: return BestChild (root, 0) // Select the best (a, τ)

7: procedure TREESEARCH (node)
8: while not node.state.terminal () do: // Repeat until it reaches the end state
9: if not node.fully.expanded () then: // Step S3
10: return Expand (node) // Step S7
11: else:
12: node = BestChild (node, scale) // Step S5
13: return node

14: procedure DEFAULTPOLICY (state) // Details of step S9
15: reward = state.reward
16: done = state.terminal ()
17: steps = h
18: while done == false && steps <MAXSTEPS do:
19: a = sample.actionSpace ()
20: obs, rew, done = env.step (a)
21: reward = reward + rew
22: steps = steps + 1
23: return reward

24: procedure EXPAND (node)
25: a, τ, Hlead = HOOSearch (node.Hroot) // Step S7: (a, τ) sampling
26: addedNode = node.AddChild (a, τ, Hleaf) // Step S7: Add node
27: return addedNode // Step S7: Expansion of search tree

28: procedure BESTCHILD (node, C) // Select the node with the maximum UCB1
29: bestchildren = []
30: UCB1 _max = −∞
31: x do included in for node.children
32: UCB1 = x.reward / x.visits + C × {(2log (node.visits) /x.visits} ^0.5
33: if UCB1 == UCB1 _max then
34: bestchildren.append (x)
35: if UCB1> UCB1 _max then
36: bestchildren = [x]
37: UCB1 _max = UCB1
38: return random.choice (bestchildren) // Random for multiple candidates

39: procedure BACKUP (Node, reward) // Details of step S11
40: while node do
41: node.visits = node.visits + 1 // increments the number of visits
42: node.reward = node.reward + reward // Update reward
43: node.Hleaf.reward = node.reward // Update 2DHOOT node value
44: node = node.parent

The contents of the processing shown in FIG. 7 are represented by pseudo code as follows.
1: procedure HOOSEARCH (root)
2: HOO-Update (root, root.visits) // Loop L31 tree update
3: leaf = HOOPolicy (root) // Loop L33 Search for path with maximum score
4: a, τ = sample.leaf.region () // Step S49
5: leaf.expand () // Step S51
6: return a, τ, leaf // Return (a, τ) and pointer

7: procedure HOO-UPDATE (node, N) // Details of recursive loop L31
8: childB _max = 0
9: x do included in for node.children // loop L311 recursive child node update
10: HOO-Update (x, N)
11: x do included in for node.children // Step S39
12: node.reward = node.reward + x.reward
13: if x.Bvalue> childB _max then
14: childB _max = x.Bvalue // used for Bvalue update
15: Uvalue = node.reward / node.visits + {2log (N) /node.visits} ^0.5
+ v ₁ ρ ^h // Step S39 U-value update
16: if Uvalue <childB _max then // Step S39 B-value update
17: node.Bvalue = Uvalue
18: else
19: node.Bvalue = childB _max

20: procedure HOOPOLICY (node)
21: while node.children do // loop L33
22: node.visits = node.visits + 1 // increment of node visit count
23: bestchildren = []
24: Bvalue _max = -∞
25: x do included in for node.children
26: if x.Bvalue == Bvalue _max then
27: bestchildren.append (x)
28: if x.Bvalue> Bvalue _max then
29: bestchildren = [x]
30: Bvalue _max = x.Bvalue
31: node = random.choice (bestchildren) // Random for multiple candidates
32: node.visits = node.vistis + 1 // increment of the number of visits
33: return node // returns the selected leaf node

  The control device 100 described above is a computer device, and as shown in FIG. 8, a memory 2501, a CPU (Central Processing Unit) 2503, a hard disk drive (HDD: Hard Disk Drive) 2505, and a display device 2509. , A drive 2513 for the removable disk 2511, an input device 2515, a communication controller 2517 for connecting to a network, and a speaker 2518 are connected by a bus 2519. The HDD may be a storage device such as a solid state drive (SSD). An operating system (OS) and an application program for executing the processing in the present embodiment are stored in the HDD 2505, and are read from the HDD 2505 to the memory 2501 when executed by the CPU 2503. The CPU 2503 controls the display control unit 2507, the communication control unit 2517, and the drive device 2513 in accordance with the processing content of the application program to perform a predetermined operation. Although the data being processed is mainly stored in the memory 2501, the data may be stored in the HDD 2505. In the embodiment of the present technology, an application program for performing the above-described process is stored in a computer-readable removable disk 2511 and distributed, and is installed from the drive device 2513 to the HDD 2505. It may be installed in the HDD 2505 via a network such as the Internet and the communication control unit 2517. Such a computer device realizes the above-described various functions by the hardware such as the CPU 2503 and the memory 2501 described above and the programs such as the OS and the application program cooperate organically. .

  The data used by executing the above-described processing is stored in a storage device such as the memory 2501 or the HDD 2505 irrespective of whether the data is being processed or is a processing result. For example, data representing a search tree and node values such as the number of times visited, an evaluation value, and a score are stored in the memory 2501 or the like.

  The embodiments described above are summarized as follows.

  The control method according to the present embodiment includes: (A) selecting an action and a time, in which an action to be executed next and a time until a next action to be executed are selected are associated with an edge between nodes; In a search tree for tree search, a search process for searching for a node with the value of the first evaluation expression (for example, the value of UCB1 is maximum) while adding a new node is selected in the search process executed last time. And (B) causing the control object to execute an action associated with the edge to the node selected in the search process executed this time, and a step to be executed during the time associated with the edge to the selected node. Steps.

  As described above, using the new search tree, an action according to the task executed by the control target can be selected and indicated at an appropriate time interval. The search processing of the tree search is based on, for example, MCTS.

  The next action to be executed corresponds to one of the finite continuous values, and the time until the further next action to be executed is one of the finite continuous times. It can be a value. These values are selected with appropriate sampling.

  Further, the process of adding the new node included in the above-described search process is performed by: (a1) In a space spanned by an action to be executed next and time, the inclusion relation of the region included in the space is hierarchically determined. In the second search tree having nodes corresponding to the respective regions, nodes are selected from the root node of the second search tree to the leaf nodes by the value of the second evaluation expression (for example, the score B is the largest). (A2) selecting a point in the area corresponding to the selected leaf node; and (a3) adding a node connected by an action and time associated with the selected point to the search tree. And the step of performing

  As described above, in order to appropriately sample the action and the time as a pair, a process using the second search tree as described above may be adopted. The processing using the second search tree is based on, for example, HOOT extended in a multi-dimensional manner.

  Further, the process of adding the new node included in the above-described search process includes the step of: (a4) setting the area corresponding to the selected leaf node to a predetermined number (two or more The method may include a step of adding, to the second search tree, a leaf node corresponding to a new region generated by dividing into a power of (natural number). This makes it possible to appropriately divide the space and appropriately express the relationship between the regions by the second search tree.

  A program for causing a computer to execute the control method described above can be created, and the program is stored in various storage media.

  Further, the control device that executes the control method as described above may be realized by a single computer, or may be realized by a plurality of computers. It shall be called a system.

REFERENCE SIGNS LIST 100 Control device 110 Monte Carlo tree search unit 120 2DHOOT unit 130 Simulation unit 140 Interface unit 200 Control target

Claims (6)

In a search tree for searching a tree for selecting an action and a time, in which a next action to be performed and a time until the next action to be further performed are selected, a new node Executing a search process for searching for a node with the value of the first evaluation expression during the time period associated with the edge to the node selected in the search process executed last time,
Causing the control object to execute an action associated with the side to the node selected in the search processing executed this time;
For causing one or more processors to execute The next action to be performed corresponds to one of a finite number of continuous values,
The control program according to claim 1, wherein the time until the action to be executed next is selected is one value within a finite continuous time. The process of adding the new node included in the search process includes:
In a space spanned by the next action to be executed and the time, the inclusion relation of the regions included in the space is hierarchically represented, and in a second search tree having a node corresponding to each region, Selecting a node from a root node of the second search tree to a leaf node with a value of an evaluation expression;
Selecting a point in an area corresponding to the selected leaf node;
Adding, in the search tree, a node connected by the action and time associated with the selected point,
The control program according to claim 2, comprising: The process of adding the new node included in the search process includes:
A leaf node corresponding to a new region generated by dividing an area corresponding to the selected leaf node into powers of a predetermined number (a natural number of 2 or more) whose exponent is the dimension number of the space, The control program according to claim 3, further comprising: adding to the second search tree. In a search tree for searching a tree for selecting an action and a time, in which a next action to be performed and a time until the next action to be further performed are selected, a new node Executing a search process for searching for a node with the value of the first evaluation expression during the time period associated with the edge to the node selected in the search process executed last time,
Causing the control object to execute an action associated with the side to the node selected in the search processing executed this time;
And a control method executed by one or more processors. In a search tree for searching a tree for selecting an action and a time, in which a next action to be performed and a time until the next action to be further performed are selected, a new node A search unit that executes a search process for searching for a node with the value of the first evaluation expression during a time period associated with an edge to the node selected in the search process executed last time,
An instruction unit for causing the control object to execute the action associated with the side to the node selected in the search process executed this time;
A system having:

Images