JP6189784B2 - Behavior control device, method and program - Google Patents

Description

  The present invention relates to a technique for controlling actions of a plurality of control objects. For example, the present invention relates to a robot cooperative control technique for obtaining an action plan for each robot for moving a plurality of robots in a coordinated manner from a formation state at a start position, avoiding an obstacle, and forming a formation at a target position.

  In recent years, research has been actively conducted to efficiently control a large number of autonomous mobile robots. Their missions vary, such as monitoring places where people can't enter, transporting goods, etc., but technology is being sought for efficient formation of platoons through the coordinated operation of many robots. (For example, refer nonpatent literature 1.). In order to realize efficient formation of a formation by a large number of robots, it is important to plan the arrangement and operation order of each robot in advance. In such a plan, as a matter of course, it is necessary to sufficiently consider the presence of obstacles and the shape of a route in an actual environment where a plurality of robots operate.

  As an effective method for performing such a plan calculation, there are a dynamic programming method and a reinforcement learning method in a Markov decision process, and various studies have been conducted (for example, see Non-Patent Document 2). .)

M. Shimizu, A. Ishiguro, T. Kawakatsu, Y. Masubuchi, “Coherent Swarming from Local Interaction by Exploiting Molecular Dynamics and Stokesian Dynamics Methods”, Proceeaings of the 2003 IEE / RSJ International Conference on intelligent Robots and Systems, Las Veqas, pp.1614-1619, October 2003. Y.Wang, CWde Silva, “Multi-Robot Box-pushing: Single-Agent Q-Learning vs. Team Q-Learning”, Proceedings of the 2006 IEEE / RSJ International Conference on Intelligent Robots and Systems, Beijing, China, pp .3694-3699, October 2006.

  However, in the method of Non-Patent Document 1, the operation of the group robot is controlled using a method of incorporating the hydrodynamic characteristics into the robot operation, and there is an advantage that enables control with a low calculation load. It is not always possible to form a formation of any shape.

  In addition, as in the method of Non-Patent Document 2, when trying to perform such a plan using dynamic programming or reinforcement learning in the Markov decision process, a plurality of robots are used compared to the case of using a single robot. When used, the time required for the calculation and the storage capacity of the computer increase exponentially with respect to the number of robots. The main cause is the enormous increase in the number of states in the Markov state space for search computation. Non-Patent Document 2 does not show a solution to the explosion problem in the Markov state space in which the verified reinforcement learning method increases the computational load exponentially as the number of robots increases.

  As described above, a method that enables control of a formation having an arbitrary shape and a low calculation load has not yet been realized.

  In view of such a current situation, an object of the present invention is to provide a behavior control apparatus, method, and program that enable a column control of an arbitrary shape and that has a lower computational load than conventional ones.

The behavior control device according to one aspect of the present invention is a behavior control device that performs behavior control for moving a plurality of control objects to a set of target positions including a predetermined entrance position, and the plurality of control objects are: Learned by constructing the Markov state space with only one state variable of the control object, which represents the appropriateness when the control object takes each action a at the current position L of the control object, Assuming that behavior control is performed based on one value function that takes a state variable of one control object as an argument , (1) a position determination unit that determines whether each control object is located at a target position; 2) When it is determined that each control object is not located at the target position, the value function is updated based on the current position of each control object with the ideal condition that the control object goes to the entrance position. In the position where each control object can move A non-target area action determination unit that determines the action to move to the position with the largest value function value after the update as the action of each control object, and (3) When it is determined that each control object is located at the target position The value function is updated based on the current position of the void with the ideal state that the void, which is a virtual entity whose position changes with the movement of the controlled object, moves toward the entrance position. The position where the action that maximizes the value of the updated value function is the action that moves to the position L among the void positions that are movable to the current position L of each control object is the candidate void position. The value of the updated value function corresponding to the action to be maximized is set as the candidate void Q function value, and the position of each controlled object is movable within the candidate void position, and the candidate void Q function value is the minimum. Each action to move to a position A behavior allocating unit including a behavior determining unit within the target area that is determined as a behavior of the target object, a position updating unit that updates the position of each control target based on the determined behavior, a behavior allocating unit, and a position updating unit And a control unit that controls to repeatedly perform the process.

  Arbitrary-shaped formation control becomes possible, and calculation load can be made lower than before.

The block diagram for demonstrating the example of an action control apparatus. The block diagram for demonstrating the example of a learning part. The block diagram for demonstrating the example of an i-th allocation part. The block diagram for demonstrating the example of a non-target area | region action determination part. The block diagram for demonstrating the example of the action determination part in a target area | region. The block diagram for demonstrating the example of a scheduling part. The figure for demonstrating the problem which this invention solves. The figure for demonstrating the action selection by an inclusion structure. The flowchart for demonstrating the example of the learning step of a behavior control method. The flowchart for demonstrating the example of the action schedule step of an action control method.

[Theoretical background]
First, the theoretical background of the behavior control apparatus and method will be described. Hereinafter, a case where the control target that is the target of behavior control is a robot will be described as an example, but the control target may be other than the robot as long as it can be a target of control.

  A number of robots move from the formation state at the start position in cooperation with each other, and the task of forming the formation at the target position is, for example, from the start position in the room separated by walls as illustrated in FIG. This is realized by moving a plurality of robots.

  The number of robots performing the mission is N (for example, N ≧ 50), and each robot is movable in the X-axis direction and the Y-axis direction on the two-dimensional plane. That is, in this example, each robot can move in four directions, up, down, left, and right with respect to the paper surface of FIG. Each grid in FIG. 7 indicates the position of each robot. There can be only one robot in each grid. Each robot is assumed to be stationary if there are obstacles or other robots in the direction of movement.

  In FIG. 7, a grid in which R is described indicates a position where the robot is present, a grid in which O is described indicates a position where an obstacle is present, and a grid in which F is described indicates a target position. A region surrounded by a thick broken line indicates a start position, and a region surrounded by a thick line indicates an entrance position described later. As described above, in FIG. 7, the formation shape at the start position of the robot is substantially rectangular, and the formation shape at the target position is substantially star shape.

  When the initial position of each robot i (i represents the robot number) is (Xr0 [i], Yr0 [i]) and the target position is (Xre [i], Yre [i]) Consider the problem of finding an action plan for a placed robot to move to a target position.

  When simply applying the Markov state transition model to such a problem, the Markov state space is the position of the robot i (Xr [i], Yr [i]), i It consists of i actions a [i]. Each state (robot position and action) is represented by discrete values. When a room is represented by a two-dimensional plane composed of an orthogonal coordinate system of X and Y, each position is represented by a discrete representation of the X axis and the Y axis. That is, as shown in FIG. 7, the room (two-dimensional plane) is divided by a grid, and each grid corresponds to each position. In each grid, “present / none” of the obstacle is set in advance. As described above, in FIG. 7, the lattice with an obstacle is indicated by O.

In this example, the action subject that is the control target is each robot arranged in the room. The action a [i] ∈D [i] of the robot i (i is a robot number) takes one of five types, that is, stationary and movement of one lattice in the vertical and horizontal directions. That is, each action is defined as follows, for example, with D [i] ε {0,1,2,3,4}.
0: stationary
1: Move one grid to the right on the 2D plane
2: Move one grid upward on a two-dimensional plane
3: Move one grid to the left on the 2D plane
4: Move downward by one grid on a two-dimensional plane The Markov state space in such a mission environment has the number of robots x 2 dimensions, and the number of actions that can be selected is the robot action ( = 5 ways). For example, if the number of robots is 50 and the number of grids in the vertical and horizontal directions of the room is 20, the number of states becomes 20 to the 100th power, and the amount of resources required for the search calculation becomes enormous. Further, every time the number of robots increases, the number of states increases 400 times, which is a big problem when using multiple robots.

  Therefore, in the present invention, in order to avoid such an explosion of the state space, the Markov state space used for learning is composed of only one robot state variable. That is, state variables and behavior variables are defined as follows.

State variable L = (Xr, Yr), action variable a∈ {0,1,2,3,4}
All N robots share one value function Q (L, a) with this state variable as an argument. That is, in updating the value function Q (L, a) at each time step, the N robots update the same value function according to their experiences (that is, N updates are performed at one time step). The update formula is as follows.

  Here, α in equation (1) is a predetermined constant called a learning rate, and 0 <α <1. In the equation (1), ← means that the value on the left side is updated with the value on the right side.

  For the i-th robot, the position L [i] of the i-th robot is substituted into L on the right side of the expressions (1) and (2), and the i-th robot is substituted for a in the expressions (1) and (2). The value function and the policy are updated by substituting the action variable a [i] and executing Expression (1) and Expression (2). This is repeated for each i = 1, 2,.

  Also at the time of action selection, each robot performs action selection using the policy function π (L) derived by one value function Q (L, a). In other words, for example, a control object that is a robot performs action control based on a single value function that represents appropriateness when the control object takes each action a at the current position L of the control object. Let's say. As a result, no matter how much the number of robots increases, the number of state spaces used for learning will be the same as the number of states in the state space for one robot, and the size of the state space will not depend on the number of robots. .

  Each robot can measure its position, know whether there is another robot at the next position, and whether there is an obstacle at the next position. To do.

  Problems that occur when learning is performed using such a single value function Q (L, a) will be described below. For example, in learning, a high reward is given to the robot at each target position. First, Q (L, a) describes what action a single robot can select from the start position to reach the target position with the shortest time step number. , Π (L), for example, when trying to avoid an obstacle on the way to the target position, there is a tendency to try to pass the same position corresponding to the corner of the obstacle. That is, a large number of robots rush to the same route, causing literal traffic jams. It is also possible that a robot that reaches the target position early stops at that position and blocks the path of the robot trying to reach the target position later. As a result, it cannot be guaranteed that all robots properly reach the target position. In order to avoid this, there is a method that considers the starting position of each robot and assigns a distant target position to a robot that reaches the target position earlier, but for that purpose, It is necessary to incorporate the position into the Markov state space by the number of robots, and when the number of robots is large, the state space is seriously increased.

  Therefore, in order not to cause such a thing, two methods are mainly proposed. The first is an action selection method using an inclusion structure, and the second is void control at a target position.

  FIG. 8 shows a conceptual diagram of an example of an action selection method using an inclusion structure. A module in which s is drawn in a circle in FIG. 8 (hereinafter referred to as an inclusion module) is an important key part in the inclusion structure. The inclusion module ignores the input from the upper module if there is an input from the lower module that outputs the signal input from the upper module as long as there is no signal input from the lower module. Output module input.

  Each layer module is composed of a Qxth (x = 1, 2, 3, 4) module and a Stopper module. The lowermost Qxth module is the Q1st module, the second layer is the Q2nd module, the third layer is the Q3rd, and the fourth layer is the Q4th module. The top layer consists of StayCom modules.

  The Q1st module receives the current robot position (xr [i], yr [i]) as an input value, and maximizes the value of Q (L, a) at L = (xr [i], yr [i]) Is output as a candidate for action a [i] of robot i. Similarly, the Q2nd module receives the current robot position (xri, yri) as an input value, and the value of Q (s, a) is the second largest in L = (xr [i], yr [i]) The value a is output as a candidate for action a [i] of robot i. Furthermore, the Q3rd module receives the current robot position (xr [i], yr [i]) as an input value, and the value of Q (L, a) at L = (xr [i], yr [i]) Is output as a candidate for action a [i] of robot i. Similarly, the Q4th module receives the current robot position (xr [i], yr [i]) as an input value, and at L = (xr [i], yr [i]), the Q (L, a) The value of a having the fourth largest value is output as a candidate for action a [i] of robot i.

  Each Qxth module does not include a [i] = 0 (still) as a candidate action to be output. The Stopper module is located next to the robot (xr [i] + 1, yr [i]), (xr [i], yr [i] +1) at the position (xr [i], yr [i]) ), (Xr [i] -1, yr [i]), and (xr [i], yr [i] -1) If another robot exists at the position to which the robot moves, no action value is output. Otherwise, the input action value is output as it is. The StayCom module always outputs a static action a [i] = 0.

  The behavior selection method described here is, for example, when the robot selects the behavior that maximizes the Q value at the position L, and another robot has already existed in the lattice that is moved by the behavior. In addition, if the robot is not stationary and does not move, it is not optimal, but the next desired action is selected and the robot is instructed to move to a grid not occupied by other robots. It is.

  This gives the robot the property that even if a fluid hits an obstacle, it does not stop there and flows in a direction that does not deviate from the mainstream direction while avoiding the obstacle.

  The module shown in FIG. 8 is composed of four layers (first to fourth). In this example, the robot can take four types of actions (a = 1, 2,3,4). In general, if there are M types of actions (including stillness), the module in FIG. 8 has M−1 layers.

Next, the principle of void control will be described. First, the target position of each robot is not strictly allocated here, and the set of the entire target position is defined as a target platoon area G. That is,
(Xre [i], Yre [i]) ∈G… (3)
Assuming that each robot can freely set all the positions in G as target positions. In other words, G is treated just like a device for pouring fluid. That is, each robot can enter G from any position on the boundary of G. However, once a robot enters G, it cannot take an action to exit G. As for the reward setting at the time of reinforcement learning, only one entry point Pe is set on the boundary of G, r = 1 which is a high reward is given only when the robot enters G from Pe, and other than that For all experiences, r = 0. The position of Pe is inside G and can be anywhere as long as it is on the boundary of G. However, selecting a position close to the start position of the robot is effective for smooth operation of the robot. The position of Pe is called the entrance position.

  In G, the experience of the robot is not used as it is for updating the value function Q (L, a), but is treated as a “void” motion that moves with the robot's behavior, and Q (L, a) is updated. I will use it. A void is a gap that changes from L ′ to L at the same time when the robot changes from position L to L ′. That is, when one robot enters G, one void goes out of G at the same time. Since Pe is set as the entry point of G, the robot tends to enter G from Pe, but at the same time, the void goes out of G via Pe. That is, when the void goes out of Pe from G, a high reward is given. Then, the movement of the robot to realize such a movement of the void inevitably moves into the G from the Pe and then moves away from the Pe aiming at a point away from the Pe. The Q value calculation and action selection method for this purpose will be described below. In G, when the robot changes from position L to L ′, the Q value is updated by the following equation (4).

Q _max (L) is the maximum value of the Q function defined by the expression (1) at the position L. That is, Q _max (L) = max _a Q (L, a) Further, γ is a constant with a predetermined value called a learning rate, and 0 <γ <1. Further, a ^-1 represents an action in the opposite direction of action a. An example of the relationship between a and a ^-1 is as follows.

a ^-1 = 0 when a = 0
When a = 1, a ^-1 = 3
When a = 2, a ^-1 = 4
When a = 3, a ^-1 = 1
When a = 4, a ^-1 = 2
Equation (4) is equivalent to the inverse of the time flow by exchanging L and L ′ in Equation (1), which is an update equation for Q (L, a) in normal Q learning. To avoid inconsistency between the Q values inside and outside G, if L is outside G and L 'is inside G, the value of Q _max (L) should be set to 0. To do. The action policy π (the function that returns the action that maximizes the Q value) derived from the Q value updated in this way returns the optimum action of the void. The robot moves so that the void can take such actions.

  Next, action selection in G is described. In G, as well as outside of G, behavior selection by inclusion structure is performed, but the operation of the Qxth module is different inside G. Inside G, the Qxth module first places the position (xr [i] + 1, yr [i]), (xr [i], yr [i] +1), (xr [i]- 1, yr [i]), (xr [i], yr [i] -1) For each void, the action that maximizes the Q value will cause the void to point to the current robot position. Are selected as candidate voids. Next, one of the candidate voids that has the Qmax value that is xth smallest is selected as a target void, and an action value that directs the robot to the selected target void is output. By doing so, it is possible to move the robot to properly guide the void to the position of Pe, and to ensure that the robot that always tries to enter G later has a free entrance.

  There are several options for the design of the Qxth module. For example, as a method for selecting a target void from candidate voids, one that increases the value of Qmax x-th among candidate voids is selected as a target void. There are various methods, such as a method of selecting one, and outputting a behavior with a small action number as an action of the robot from actions of directing the robot to the candidate void.

  In addition, although the entrance position of G is set to one point, if Pe is occupied by the robot due to the effect of action selection by the inclusion structure, the robot enters G from a point near Pe other than Pe Thus, the robot's behavior is controlled so that the robot trying to enter G does not concentrate on one point of Pe and cause a traffic jam.

[Action Control Apparatus and Method]
An example of a behavior control device and method will be described with reference to the drawings. This behavior control apparatus and method performs behavior control for moving a plurality of control objects to a set of target positions including a predetermined entrance position.

  As shown in FIG. 1, the behavior control apparatus includes a learning unit 1, a storage unit 2, and a scheduling unit 3, for example.

  As illustrated in FIG. 2, the learning unit 1 includes, for example, an input unit 11, an action assignment unit 12, a position update unit 13, and a control unit 14.

  As shown in FIG. 6, the scheduling unit 3 includes, for example, an initial state input unit 31, an action assignment unit 32, a position update unit 33, and a target position arrival determination unit 34.

  Hereinafter, a case where the control target to be controlled is a robot will be described as an example. Of course, the control object may be other than the robot as long as it can be a control target.

  First, the process of the learning step by the learning unit 1 of the behavior control device will be described. An example of the processing flow of the learning step is shown in FIG.

<Input unit 11>
The input unit 11 receives initial positions (xr0 [i], yr0 [i]) and target positions (Xre [i], Yre [i]) of the N robots. Here, i = 1, 2,. The set of N target positions is G = {(Xre [1], Yre [1]), (Xre [2], Yre [2]), ..., (Xre [N], Yre [N])} As stored in the storage unit 2.

  For each of the N robots, the initial position L [i] = (xr0 [i], yr0 [i]) of the i-th robot is set using the input initial position information, and the i-th robot Are stored in the storage unit 2.

  The target position includes a predetermined entrance position. Information about the entrance position is also input from the input unit 11 and stored in the storage unit 2.

<Storage unit 2>
The storage unit 2 stores the Q function Q (L, a) for each of the combinations of the position L and a∈ {0,1,2,3,4}, and the measures π (L) and Qmax ( Assume that the initial value of L) is stored. The range that L can take is all the coordinates in the region on the target two-dimensional plane. These initial values may be set to random values. However, when L matches the obstacle position, Q (L, a) = 0 may be set. For π (L) and Qmax (L), the maximum value among the initial values Q (L, a) of the Q function for each L may be set.

  It is assumed that the reward r (L) at each position L is also stored in the storage unit 2. Information about the reward r (L) at each position L is input from the input unit 11, for example.

<Action allocation unit 12>
The action assignment processing by the action assigning unit 12 is sequentially executed for each robot. The action assigning unit 12 includes, for example, a first assigning unit 12-1, an i-th assigning unit 12-i,..., An N-th assigning unit 12-N.

  Assume that i = 1, 2,..., N, and the i-th assignment unit 12-i performs the action assignment process for the i-th robot, for example. An example of the configuration of the i-th allocation unit 12-i is shown in FIG.

<< Position Determination Unit 12-i-1 >>
The position determination unit 12-i-1 reads the position (xr [i], yr [i]) of the i-th robot from the storage unit 2, and the read position (xr [i], yr [i]) is the target. It is determined whether or not it is included in the position set G. In other words, the position determination unit 12-i-1 determines whether the robot is located at the target position (step A1).

  If the position (xr [i], yr [i]) is not included in the target position set G, the position determination unit 12-i-1 performs the next process When (xr [i], yr [i]) is included in the set G of target positions, control is performed so that the action determining unit 12-i-3 in the target area executes the following process.

  In addition, when the position (xr [i], yr [i]) is not included in the set G of target positions, the position determination unit 12-i-1 When the action value to be output is input to the position update unit 13 and (xr [i], yr [i]) is included in the set G of the target position, the action determining unit 12-i-3 in the target area Control is performed so that the action value to be output is input to the position update unit 13.

≪Non-target area action determination unit 12-i-2≫
An example of the configuration of the non-target area action determining unit 12-i-2 is shown in FIG.

  The non-target area action determination unit 12-i-2 determines an action based on the action selection method of FIG. In other words, when it is determined that the robot is not located at the target position, the behavior determination unit 12-i-2 outside the target area is based on the current position of the robot with an ideal state where the robot heads toward the entrance position. Then, the value function is updated, and the action to move to the position where the updated value function value is the largest among the positions where the robot can move is determined as the action of the robot (step A2).

[Out-of-region Q function update unit 12-i-21]
The out-of-region Q function updating unit 12-i-21 sets the position of the i-th robot one hour before as L = (xr [i], yr [i]) and sets the current position of the i-th robot to L By referring to Q (L, a) and Qmax (L ') stored in the storage unit 2 as' = (xr '[i], yr' [i]), the robot action a one hour before the step a , Q (L, a) is obtained by the equation (1), and the value of Q (L, a) stored in the storage unit 2 is updated with the obtained value of Q (L, a). Further, the out-of-region Q function updating unit 12-i-21 outputs the value of Q (L, a) before the update and the value of Q (L, a) after the update to the control unit 14.

  Further, the out-of-region Q function updating unit 12-i-21 uses the updated value of Q (L, a) to obtain the policy π (L) according to the equation (2), and the obtained π (L) The policy π (L) stored in the storage unit 2 is updated with the value.

[First Region Outside Action Candidate Determining Unit 12-i-22]
The first out-of-region action candidate determination unit sets Q (L, 1), Q (L, 2), Q (L, L) stored in the storage unit 2 as L = (xr [i], yr [i]). 3) The value of a that takes the maximum value among Q (L, 4) is output as the first out-of-region action candidate value. Also, the value of Qmax (L) stored in the storage unit 2 is updated with the value of the maximum Q function.

[First Region Inclusion Control Unit 12-i-23]
The first out-of-region inclusion control unit 12-i-23 assumes that the i-th robot moves according to the first out-of-region action candidate value determined by the first out-of-region action candidate determination unit 12-i-22. It is determined whether another robot exists at the position after movement (xr ′ [i], yr ′ [i]). That is, it is determined whether or not there exists j where (xr ′ [i], yr ′ [i]) = (xr [j], yr [j]) (i ≠ j). Further, it is determined whether or not an obstacle exists at the position (xr ′ [i], yr ′ [i]).

  The first region outer inclusion control unit 12-i-23 determines that the first region inclusion control unit 12-i-23 has no other robot at the moved position (xr '[i], yr' [i]) or has an obstacle. Control is performed so that the 2-out-of-region action candidate determination unit 12-i-24 executes the following process.

  The first out-of-region inclusion control unit 12-i-23 performs the first out-of-region action when there are no other robots and obstacles at the moved position (xr ′ [i], yr ′ [i]). Candidate values are output as “action values”.

[Second Out-of-Region Action Candidate Determination Unit 12-i-24]
The second out-of-region action candidate determination unit 12-i-24 sets Q (L, 1), Q (L, 2) stored in the storage unit 2 as L = (xr [i], yr [i]). , Q (L, 3), Q (L, 4), the value of a taking the second largest value is output as the second out-of-region action candidate value.

[Second Area Outer Inclusion Control Unit 12-i-25]
The second region inclusion control unit 12-i-25 assumes that the i-th robot moves according to the second region outside action candidate value determined by the second region outside action candidate determination unit 12-i-24. It is determined whether another robot exists at the position after movement (xr ′ [i], yr ′ [i]). That is, it is determined whether or not there exists j where (xr ′ [i], yr ′ [i]) = (xr [j], yr [j]) (i ≠ j). Further, it is determined whether or not an obstacle exists at the position (xr ′ [i], yr ′ [i]).

  The second region inclusion control unit 12-i-25, when there is another robot at the moved position (xr '[i], yr' [i]) or there is an obstacle, The out-of-region action candidate determination unit 12-i-26 is controlled to execute the next process.

  The second out-of-region inclusion control unit 12-i-25 performs the second out-of-region action when there is no other robot or obstacle at the moved position (xr '[i], yr' [i]). Candidate values are output as “action values”.

[Third Outside Region Action Candidate Determination Unit 12-i-26]
The third region outside action candidate determination unit 12-i-26 stores Q (L, 1) and Q (L, 2) stored in the storage unit 2 as L = (xr [i], yr [i]). , Q (L, 3), Q (L, 4), the value of a that takes the third largest value is output as a third out-of-region action candidate value.

[Third region outer inclusion control unit 12-i-27]
The third region outside inclusion control unit 12-i-27 assumes that the i-th robot moves according to the third region outside action candidate determination value determined by the third region outside action candidate determination unit 12-i-26. It is determined whether another robot exists at the position after movement (xr ′ [i], yr ′ [i]). That is, it is determined whether or not there exists j where (xr ′ [i], yr ′ [i]) = (xr [j], yr [j]) (i ≠ j). Further, it is determined whether or not an obstacle exists at the position (xr ′ [i], yr ′ [i]).

  The third region inclusion control unit 12-i-27, when there is another robot at the moved position (xr '[i], yr' [i]) or when there is an obstacle, The 4-outside area action candidate determination unit 12-i-28 performs control so as to execute the following process.

  If there is no other robot or obstacle at the moved position (xr ′ [i], yr ′ [i]), the third region outer inclusion control unit 12-i-27 Candidate values are output as “action values”.

[Fourth Region Action Candidate Determination Unit 12-i-28]
The fourth region outside action candidate determination unit 12-i-28 stores Q (L, 1), Q (L, 2) stored in the storage unit 2 as L = (xr [i], yr [i]). , Q (L, 3), Q (L, 4), the value of a that takes the fourth largest value (that is, takes the minimum value) is output as the fourth out-of-region action candidate value.

[Fourth Region Inclusion Control Unit 12-i-29]
The fourth region outer inclusion control unit 12-i-29 assumes that the i-th robot moves according to the fourth region outer action candidate determination value determined by the fourth region outer action candidate determination unit 12-i-28. It is determined whether another robot exists at the position after movement (xr ′ [i], yr ′ [i]). That is, it is determined whether or not there exists j where (xr ′ [i], yr ′ [i]) = (xr [j], yr [j]) (i ≠ j). Further, it is determined whether or not an obstacle exists at the position (xr ′ [i], yr ′ [i]).

  The fourth region outer inclusion control unit 12-i-29 is configured when another robot and an obstacle exist at the position (xr '[i], yr' [i]) after the movement or when an obstacle exists. Outputs a = 0 (stillness) as an “action value”.

  When there is no other robot at the moved position (xr ′ [i], yr ′ [i]), the fourth region outer inclusion control unit 12-i-29 sets the fourth region outer action candidate value. Output as “action value”.

<< Destination Area Action Determination Unit 12-i-3 >>
FIG. 5 shows a detailed configuration of the in-target area action determination unit 12-i-3.

  Similar to the non-target-area action determining unit 12-i-2, the in-target-area action determining unit 12-i-3 determines an action based on the action selection method of FIG. However, the point which uses Formula (4) instead of Formula (1) as a Q function differs from the non-target area | region action determination part 12-i-2. That is, the action determination unit 12-i-3 in the target area performs a process incorporating void control.

  That is, if it is determined that the robot is positioned at the target position, the action determining unit 12-i-3 within the target area receives a void that is a virtual entity whose position is switched with the robot as the robot moves. The value function is updated based on the current position of the void with the ideal state of going to the position, and the value of the updated value function is changed among the void positions that are movable to the current position L of the robot. The position in which the action to be maximized is the action to move to position L is set as a candidate void position, and the value of the updated value function corresponding to the action to be maximized is set as a candidate void Q function value. Is determined as the robot action (step A3). The action of moving to the position where the candidate void Q function value is minimum is determined.

[Intra-region Q function update unit 12-i-31]
The in-region Q function updating unit 12-i-31 sets the position (xr [i], yr [i]) of the i-th robot one step before as L, and sets the current i-th robot position as the void position L ′. Referring to Q (L ′, a) and Qmax (L) stored in the storage unit 2, a ⁻¹ is obtained from the action of the robot one step before, and Q (L ′, a a ⁻¹ ) is obtained, and the value of Q (L ′, a ⁻¹ ) stored in the storage unit 2 is updated with the obtained value of Q (L ′, a ⁻¹ ).

  Here, L ′ is the position after movement when the action a is selected at the position L = (xr [i], yr [i]) one step before. The relationship between a and L ′ is, for example, as follows.

If a = 1, L '= (xr [i] +1, yr [i])
If a = 2, L '= (xr [i], yr [i] +1)
If a = 3, L '= (xr [i] -1, yr [i])
If a = 4, L '= (xr [i], yr [i] -1)
Region Q function update unit 12-i-31 is updated before the Q (L ', a ^-1) and the updated Q (L', a ^-1) and outputs to the controller 14. Further, the value of the policy π (L ′) stored in the storage unit 2 is updated with the maximum value of the Q function at this time.

[Candidate Void Set Generation Unit 12-i-32]
The candidate void set generation unit 12-i-32 performs the following processes (1) to (3).

(1) Positions (xr [i] +1, yr [i]), (xr [i], yr [i] +1) adjacent to the position (xr [i], yr [i]) of the i-th robot ), (Xr [i] -1, y [ri]), (xr [i], yr [i] -1) as void positions L ′, and Q (L ′, a ^-1 ) Q (L ′, a ⁻¹ ) having the maximum value among [a ⁻¹ = 0,1,2,3,4] is determined as a “candidate void Q function value”. The value of a ⁻¹ at this time is determined as “candidate void behavior in L ′”.

  (2) Of the candidate void actions in each L ′ obtained in (1) above, the position after the movement when assuming that the void has moved from L ′ according to the candidate void action is the position of the i-th robot (xr [ The set of L ′ as i], yr [i]) is obtained as a “candidate void position set”.

  (3) A set comprising a set of each candidate void position L ′ included in the “candidate void position set” obtained in (2) above, a candidate void Q function value in L ′, and a candidate void action in L ′ It outputs to 1st area | region action candidate determination part 12-i-33 as a "candidate void set."

[First Area Action Candidate Determination Unit 12-i-33]
The first region action candidate determination unit 12-i-33 sets the candidate void position L ′ corresponding to the candidate void Q function value having the smallest candidate void Q function value from the “candidate void set” as the “first target position”. Determine as.

  The action of moving from the position (xr [i], yr [i]) of the i-th robot to the first target position determined in (2) above is output as a first region action candidate value.

[First Area Inclusion Control Unit 12-i-34]
The first region inclusion control unit 12-i-34 assumes that the i-th robot moves according to the first region action candidate value determined by the first region action candidate determination unit 12-i-33. It is determined whether another robot exists at the position after movement (xr ′ [i], yr ′ [i]). That is, it is determined whether or not there exists j where (xr ′ [i], yr ′ [i]) = (xr [j], yr [j]) (i ≠ j). Further, it is determined whether there is an obstacle at the position (xr '[i], yr' [i]) or whether the position (xr '[i], yr' [i]) is outside G. .

  The first region inclusion control unit 12-i-34 determines whether there is another robot at the moved position (xr '[i], yr' [i]) or the position (xr '[i], yr' If there is an obstacle in [i]) or the position (xr '[i], yr' [i]) is outside G, the second region action candidate determination unit executes the following process Control to do.

  The first region inclusion control unit 12-i-34 has no other robots and obstacles at the moved position (xr '[i], yr' [i]), and the position (xr '[ If i], yr ′ [i]) is not outside G, the first region action candidate value is output as the “action value”.

[Second Area Action Candidate Determination Unit 12-i-35]
The second region action candidate determination unit 12-i-35 determines the candidate void position L ′ corresponding to the candidate void Q function value having the second smallest candidate void Q function value from the “candidate void set” as the “first 2 target position ".

  The action of moving from the position of the i-th robot (xr [i], yr [i]) to the second target position determined in (2) above is output as a second area action candidate value.

[Second Area Inclusion Control Unit 12-i-36]
The second region inclusion control unit 12-i-36 assumes that the i-th robot moves according to the second region action candidate value determined by the second region action candidate determination unit 12-i-35. It is determined whether another robot exists at the position after movement (xr ′ [i], yr ′ [i]). That is, it is determined whether or not there exists j where (xr ′ [i], yr ′ [i]) = (xr [j], yr [j]) (i ≠ j). Further, it is determined whether there is an obstacle at the position (xr '[i], yr' [i]) or whether the position (xr '[i], yr' [i]) is outside G. .

  The second region inclusion control unit 12-i-36 determines whether there is another robot at the moved position (xr '[i], yr' [i]) or the position (xr '[i], yr' If an obstacle exists in [i]) or the position (xr ′ [i], yr ′ [i]) is outside G, the third region action candidate determination unit 12-i-37 Control to execute the following processing.

  The second region inclusion control unit 12-i-36 has no other robot or obstacle at the moved position (xr '[i], yr' [i]), and the position (xr '[ If i], yr '[i]) is not outside G, the action candidate value in the second region is output as the “action value”.

[Third Area Action Candidate Determination Unit 12-i-37]
The first region action candidate determination unit 12-i-37 determines the candidate void position L ′ corresponding to the candidate void Q function value having the third smallest candidate void Q function value from the “candidate void set” as the “first 3 target position ".

  The action of moving from the position (xr [i], yr [i]) of the i-th robot to the third target position determined in (2) above is output as an action candidate value in the third region.

[Third Region Inclusion Control Unit 12-i-38]
The third region inclusion control unit 12-i-38 assumes that the i-th robot moves according to the third region action candidate determination value determined by the third region action candidate determination unit 12-i-37. It is determined whether another robot exists at the position after movement (xr ′ [i], yr ′ [i]). That is, it is determined whether or not there exists j where (xr ′ [i], yr ′ [i]) = (xr [j], yr [j]) (i ≠ j). Further, it is determined whether there is an obstacle at the position (xr '[i], yr' [i]) or whether the position (xr '[i], yr' [i]) is outside G. .

  The third region inclusion control unit 12-i-38 determines whether there is another robot at the moved position (xr '[i], yr' [i]) or the position (xr '[i], yr' If there is an obstacle in [i]) or the position (xr ′ [i], yr ′ [i]) is outside G, the fourth region action candidate determination unit 12-i-39 Control to execute the following processing.

  The third region inclusion control unit 12-i-38 has no other robot or obstacle at the moved position (xr '[i], yr' [i]), and the position (xr '[ If i], yr ′ [i]) is not outside G, the action candidate value in the third region is output as the “action value”.

[Fourth Area Action Candidate Determination Unit 12-i-39]
The fourth region action candidate determination unit 12-i-39 determines the candidate void position L ′ corresponding to the candidate void Q function value having the fourth smallest candidate void Q function value from the “candidate void set” as the “first 4 target position ".

  The action of moving from the position of the i-th robot (xr [i], yr [i]) to the fourth target position determined in (2) above is output as the action candidate value in the fourth region.

[Fourth Region Inclusion Control Unit 12-i-310]
The fourth region inclusion control unit 12-i-310 assumes that the i-th robot moves according to the fourth region action candidate value determined by the fourth region action candidate determination unit 12-i-39. It is determined whether another robot exists at the position after movement (xr ′ [i], yr ′ [i]). That is, it is determined whether or not there exists j where (xr ′ [i], yr ′ [i]) = (xr [j], yr [j]) (i ≠ j). Further, it is determined whether there is an obstacle at the position (xr '[i], yr' [i]) or whether the position (xr '[i], yr' [i]) is outside G. .

  The fourth region inclusion control unit 12-i-310 determines whether there is another robot at the moved position (xr '[i], yr' [i]) or the position (xr '[i], yr' If there is an obstacle in [i]) or the position (xr '[i], yr' [i]) is outside G, a = 0 (still) is output as the “action value” .

  The fourth region inclusion control unit 12-i-310 has no other robot and obstacle at the moved position (xr '[i], yr' [i]), and the position (xr '[ If i], yr ′ [i]) is not outside G, the fourth region action candidate value is output as “action value a [i]”.

  Through the above processing, the i-th assigning unit 12-i-310 transmits an action value a [which is a value corresponding to the action selected by the i-th robot at the current position (xr [j], yr [j]). i] ∈ {0,1,2,3,4} is output. Therefore, the action assignment unit 12 outputs action values a [i] selected by the N robots at the current position.

<Location update unit 13>
For each i = 1, 2,..., N, the position updating unit 13 performs the action value a output from the action assigning unit 12 at the current position (xr [j], yr [j]) of the i-th robot. The position (xr '[i], yr' [i]) after the movement of the robot when the action corresponding to [i] is taken is calculated, and the calculated (xr '[i], yr '[i]) updates the position of the i-th robot stored in the storage unit 2. In other words, the position update unit 13 updates the position of each control target that is, for example, a robot based on the action determined by the action assignment unit 12 (step A4). Sequence of updated positions {(xr '[1], yr' [1]), (xr '[2], yr' [2]), ..., (xr '[N], yr' [N]) } Is input to the control unit 14.

<Control unit 14>
The control unit 14 performs control so as to repeatedly perform the processes of the action assignment unit and the position update unit (step A5).

  The control unit 14 performs control so that the processes of the action assignment unit and the position update unit are repeatedly performed until a predetermined end condition is satisfied. For example, the control unit 14 is composed of all the pre-update Q functions and the post-update Q functions output from the non-target region action determining unit 12-i-2 or the in-target region action determining unit 12-i-3. Control is performed so that the processing of the action assignment unit 12 and the position update unit 13 is executed until the difference between the value of the pre-update Q function and the value of the post-update Q function is equal to or less than a predetermined threshold. The termination condition in this case is that the difference between the value of the pre-update Q function and the value of the post-update Q function is equal to or less than a predetermined threshold value.

  When the difference between the value of the pre-update Q function and the value of the post-update Q function is less than or equal to a predetermined threshold for a set composed of all pre-update Q functions and post-update Q functions, learning of the behavior control device The process of the learning step by the unit 1 ends.

  Next, the process of the action schedule step by the scheduling unit 3 of the action control device will be described. Hereinafter, the description will focus on the parts that are different from the learning unit 1, and redundant description of the same parts as the learning unit 1 will be omitted.

  An example of the process flow of the action schedule step is shown in FIG.

<Scheduling unit 3>
The scheduling unit 3 determines an action plan of each robot for the N actual robots to form a target formation from the initial position, using the Q function and the policy obtained by the processing of the learning unit 1 described above. A detailed configuration of the scheduling unit is shown in FIG.

<< Initial state input unit 31 >>
The initial state input unit 31 receives initial positions (xr0 [i], yr0 [i]) [i = 1, 2,..., N] of N robots.

≪Action allocation unit 32≫
The process of the action assigning unit 32 is the same as that of the action assigning unit 12 of the learning unit 1. As i = 1, 2,..., N1, the i-th allocation unit 32-i is the same as the i-th allocation unit 12-i of the behavior allocation unit 12 of the learning unit 1.

However, the action assigning unit 32 stores the action a [i] determined for each i in the storage unit as the action a _t [i] selected by the i-th robot at the current time t. Thereby, the storage unit 2 stores a series of actions (action series) A [i] = {a ₁ [i], a ₂ [i], ..., a selected by the i-th robot at each time up to time t. _t−1 [i]} is stored.

  In addition, the behavior allocation unit 12 of the learning unit 1 not only determines a [i], but also updates the value of the Q function and the value of the policy, but the behavior allocation unit 32 of the scheduling unit 3 updates the Q function. There is no need to update the value and policy value.

  When the value of the Q function is not updated, the position determination unit 32-i-1 of the action assignment unit 32 determines whether the robot is positioned at the target position (step B1), and the target area of the action assignment unit 32 When it is determined that the robot is not located at the target position, the external action determination unit 32-i-2 determines the action of moving the robot to the position where the value of the value function is the largest among the positions where the robot can move. When the action is determined as an action (step B2) and the action determining unit 32-i-3 in the target area determines that the robot is located at the target position, the void is a position that can move to the current position L of the robot. The position where the action that maximizes the value of the value function in the position is the action that moves to the position L is set as the candidate void position, and the value of the value function corresponding to the action to be maximized is set as the candidate void Q function value. In the void position The action of moving to a position where the bot can move and the candidate void Q function value is minimum is determined as the action of the robot (step B3).

≪Location update unit 33≫
The processing of the position update unit 33 is the same as that of the position update unit 13 of the learning unit 1. That is, the position update unit 33 updates the position of each control target that is, for example, a robot based on the behavior determined by the behavior allocating unit 32 (step B4).

≪Target position arrival determination unit 34≫
The target position arrival determination unit 34, for each i = 1, 2,..., N, is the updated position (xr ′ [i], yr ′ [i]) ∈G output from the position update unit 33. If it is (xr ′ [i], yr ′ [i]) ∈G for all i, the action sequence A [i] = {a ₁ currently stored in the storage unit 2 _{[i], a 2 [i} ], ..., a t-1 [i], and outputs the scheduling result to a _t [i]}. If (xr ′ [i], yr ′ [i]) εG is not satisfied for at least one i, the behavior assigning unit 32 and the position updating unit 33 are controlled to be executed again (step B5).

[Modifications, etc.]
The non-target-area action determining unit 12-i-2 is configured with four layers (first to fourth). In the above example, the action that can be taken by the robot other than stationary (a = 0) is four. This is because the type (a = 1, 2, 3, 4) is assumed. In general, if there are M types of actions (including stillness), the non-target area action determining unit 12-i-2 has M-1 layers. The same applies to the action determining unit 12-i-3 within the target area, the action determining unit 32-i-1 outside the target area, and the action determining unit 32-i-3 within the target area.

  The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention. In addition, the processing described in the above embodiment may be executed not only in time series according to the order of description but also in parallel or individually as required by the processing capability of the apparatus that executes the processing. .

  Further, when the processing functions in the hardware entity described in the above embodiment are realized by a computer, the processing contents of the functions that the hardware entity should have are described by a program. Then, by executing this program on a computer, the processing functions in the hardware entity are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used. Specifically, for example, as a magnetic recording device, a hard disk device, a flexible disk, a magnetic tape or the like, and as an optical disk, a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read Only). Memory), CD-R (Recordable) / RW (ReWritable), etc., magneto-optical recording medium, MO (Magneto-Optical disc), etc., semiconductor memory, EEP-ROM (Electronically Erasable and Programmable-Read Only Memory), etc. Can be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In this embodiment, a hardware entity is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

DESCRIPTION OF SYMBOLS 1 Learning part 2 Memory | storage part 3 Scheduling part 11 Input part 12 Action allocation part 12-i-1 Position determination part 12-i-2 Out-of-target-area action determination part 12-i-3 In-target-area action determination part 13 Position update part 14 control part 31 initial state input part 32 action assignment part 32 action assignment part 32-i-1 position determination part 32-i-2 non-target area action determination part 32-i-3 in-target area action determination part 33 position update part 34 Target position arrival determination unit

Claims (3)

A behavior control device that performs behavior control for moving a plurality of control objects to a set of target positions including a predetermined entrance position,
The plurality of control objects are represented by a Markov state space that represents the appropriateness when the control object takes each action a at the current position L of the control object. As behavior control is performed based on one value function that takes as an argument a state variable of one control object learned by configuring ,
(1) a position determination unit that determines whether each control object is located at the target position; and (2) if it is determined that each control object is not located at the target position, the control object is The value function is updated based on the current position of each control object with the ideal direction to the entrance position, and the updated value function in a position where each control object can move A non-target area action determining unit that determines an action to move to a position having the largest value as the action of each control object, and (3) when it is determined that each control object is located at the target position. The value function is updated based on the current position of the void in an ideal state where the void, which is a virtual entity whose position is switched with the position of the control object as the control object moves, is directed to the entrance position. And the current of each control object The position where the action that maximizes the value of the updated value function among the void positions that can be moved to the position L is the action that moves to the position L as the candidate void position, and the action that maximizes the action The value of the corresponding updated value function is set as a candidate void Q function value, and the control object moves to a position where each of the controlled objects can move and has a minimum candidate void Q function value. An action allocating unit including an action determining unit within a target area that determines an action as an action of each of the control objects;
A position update unit that updates the position of each control object based on the determined action;
A control unit that performs control so as to repeatedly perform the processes of the action assigning unit and the position updating unit;
A behavior control device including: A behavior control method for performing behavior control for moving a plurality of control objects to a set of target positions including a predetermined entrance position,
The plurality of control objects are represented by a Markov state space that represents the appropriateness when the control object takes each action a at the current position L of the control object. As behavior control is performed based on one value function that takes as an argument a state variable of one control object learned by configuring ,
(1) a position determination step in which the position determination unit determines whether each of the control objects is located at the target position; and (2) an out-of-target-area action determination unit determines that each control object is If it is determined that the control object is not in the ideal state, the value function is updated based on the current position of each control object, and the control object is Out-of-target-area action determining step for determining the action to move to the position where the value function value after the update is the largest among the movable positions as the action of each control object, and (3) In-target action determination When the control unit determines that each of the control objects is located at the target position, a void that is a virtual existence in which the position of the control object interchanges with the movement of the control object is present at the entrance position. Going to the ideal state An action that updates the value function based on the current position of the void and maximizes the value of the updated value function among the void positions that are movable to the current position L of each control object. Is a position that is an action that moves to the position L as a candidate void position, and the value of the updated value function corresponding to the action to be maximized is a candidate void Q function value. An action allocating step including: an action determining step in a target area that determines an action of moving to a position where the control object is movable and the candidate void Q function value is minimum as the action of each control object;
A position update unit, wherein the position update unit updates the position of each control object based on the determined action;
Control unit, and a control step of controlling to perform repeated process between the behavior assigning step and the location update step,
A behavior control method including:   The program for functioning a computer as each part of the action control apparatus of Claim 1.

Images