JP6285849B2 - Behavior control system, method and program thereof - Google Patents

Description

  The present invention relates to a technique for controlling actions of a plurality of control objects. For example, the present invention relates to a robot cooperative control technique for obtaining an action plan for each robot for moving a plurality of robots in a coordinated manner from a formation state at a start position, avoiding an obstacle, and forming a formation at a target position.

  In recent years, research has been actively conducted to efficiently control a large number of autonomous mobile robots. Their missions vary, such as monitoring places where people can't enter, transporting goods, etc., but technology is being sought for efficient formation of platoons through the coordinated operation of many robots. (For example, refer nonpatent literature 1). In order to realize efficient formation of a formation by a large number of robots, it is important to plan the arrangement and operation order of each robot in advance. In such a plan, as a matter of course, it is necessary to sufficiently consider the presence of obstacles and the shape of a route in an actual environment where a plurality of robots operate.

  As an effective method for performing such a plan calculation, there are a dynamic programming method and a reinforcement learning method in a Markov decision process, and various studies have been conducted (for example, see Non-Patent Document 2). ).

  Also, in the robot row control, the robot row control under the assumption that the robot moves as a whole in a state where the robots are in contact with each other, the relative positional relationship between the robots The advantages of being able to determine the absolute position of each robot and the advantage of not requiring additional position measurement equipment are being studied. For example, in the non-patent document 3, the row control from an arbitrary rectangular shape row to another rectangular shape row is shown.

  In addition, in a series of studies leading to the research shown in Non-Patent Document 4 and Non-Patent Document 5, the formation control that changes from one formation to another formation is shown.

M. Shimizu, A. Ishiguro, T. Kawakatsu, Y. Masubuchi, “Coherent Swarming from Local Interaction by Exploiting Molecular Dynamics and Stokesian Dynamics Methods”, Proceeaings of the 2003 IEE / RSJ International Conference on intelligent Robots and Systems, Las Veqas, pp.1614-1619, October 2003. Y.Wang, CWde Silva, “Multi-Robot Box-pushing: Single-Agent Q-Learning vs. Team Q-Learning”, Proceedings of the 2006 IEEE / RSJ International Conference on Intelligent Robots and Systems, Beijing, China, pp .3694-3699, October 2006. A. Becker, G. Habibi, J. Werfel, M. Rubenstein, and J. McLurkin, “Massive Uniform Manipulation: Controlling Large Populations of Simple Robots with a Common Input Signal”, Proceedings of the IEEE / RSJ International Conference on Intelligent Robots and Systems, Japan, pp.520-527, November, 2013. Stanton Wong1 and Jennifer Walter ”Deterministic Distributed Algorithm for Self-Reconfiguration of Modular Robots from Arbitrary to Straight Chain Configurations”, Proceedings of the 2013 IEEE International Conference on Robotics and Automation (ICRA), Karlsruhe, Germany, pp.537-543, May 6-10, 2013. Michael Rubenstein, Alejandro Cornejo, Radhika Nagpal, “Programmable self-assembly in a thousand-robot swarm”, SCIENCE, 2014, Vol. 345, Issue 6198, pp.795-799.

  However, in the method of Non-Patent Document 1, the operation of the group robot is controlled using a method of incorporating the hydrodynamic characteristics into the robot operation, and there is an advantage that enables control with a low calculation load. It is not always possible to form a formation of any shape.

  In addition, as in the method of Non-Patent Document 2, when trying to perform such a plan using dynamic programming or reinforcement learning in the Markov decision process, a plurality of robots are used compared to the case of using a single robot. When used, the time required for the calculation and the storage capacity of the computer increase exponentially with respect to the number of robots. The main cause is the enormous increase in the number of states in the Markov state space for search computation. Non-Patent Document 2 does not show a solution to the explosion problem in the Markov state space in which the verified reinforcement learning method increases the computational load exponentially as the number of robots increases.

  Further, both the methods of Non-Patent Documents 1 and 2 require additional equipment for position measurement.

  In Non-Patent Document 3, no additional equipment for position measurement is required in consideration of the condition of maintaining the state in which the robot is in contact, but the presence of an obstacle is required for its realization. Therefore, the amount of calculation required for the operation plan is proportional to the square of the number of robots, and increases rapidly as the number of robots increases.

  In the method of Non-Patent Document 4, conversion to a linear formation must be performed once, and there is a great restriction on the possible formation operation itself.

  In the method of Non-Patent Document 5, there are problems that the starting row and the target row must share a point, and that obstacles are not considered.

  In view of such a current situation, the present invention reduces the calculation time required for the planned calculation and the storage capacity of the computer to be as small as when handling a single robot, while considering the presence of a large number of robots. It is possible to perform a deformation operation in an environment where there is an obstacle from a formation formation state at an arbitrary start position to a formation formation state at any other target position while maintaining the state where the robots are in contact with each other. The purpose is to provide a robot coordination corps formation technology.

  In order to solve the above problems, according to one aspect of the present invention, the behavior control system moves a plurality of control objects arranged in a set of start positions to a set of target positions including a predetermined entrance position. To control the behavior. The action control system has a direction that is not parallel to the first direction as a second direction, a direction opposite to the first direction as a third direction, a direction opposite to the second direction as a fourth direction, Each start position and each target position are adjacent to other start positions and target positions in at least one of the first to fourth directions, and the set of target positions and the set of start positions are each a set of arbitrary Forming a shape, the control object is a first position adjacent in the first direction on the two-dimensional plane of the control object, a second position adjacent in the second direction, a third position adjacent in the third direction, And a fourth position adjacent in the fourth direction, a fifth position adjacent to the first position in the second direction, a sixth position adjacent to the second position in the third direction, and adjacent to the third position in the fourth direction. 7th and 4th positions Communication means for communicating with another control object existing in the eighth position adjacent in the first direction, when the control object takes each action a at the current position s of the control object It is controlled based on one value function representing appropriateness, and is controlled to move to any one of the first to fourth positions on the two-dimensional plane, and the value function is stored. And a virtual existence that occurs with the movement of the control object or moves in the direction opposite to the movement direction of the control object as a void, and uses the value function (G-1 ) When the control object exists at any of the target positions, control is performed so that the control object moves to another target position far from the predetermined entrance position, and (G-2) the control object is the target position. If the target object is not included in the set of (V-1) When the void is located at a position not included in the set of target positions or at a predetermined entrance position, the void is located at a target position far from the predetermined entrance position. (V-2) If the void is located at any of the target positions excluding the predetermined entrance position, the void moves toward the predetermined entrance position. The action selection part which controls is included. The action selection unit determines the first control object close to the entrance position, and generates a void by moving it under the control of (G-1) and (G-2). The second control is performed to move the void as long as it follows the control of (V-1) and (V-2).

  In order to solve the above problem, according to another aspect of the present invention, a behavior control method moves a plurality of control objects arranged in a set of start positions to a set of target positions including a predetermined entrance position. To control the behavior. In the behavior control method, the direction that is not parallel to the first direction is the second direction, the direction opposite to the first direction is the third direction, the direction opposite to the second direction is the fourth direction, Each start position and each target position are adjacent to other start positions and target positions in at least one of the first to fourth directions, and the set of target positions and the set of start positions are each a set of arbitrary Forming a shape, the control object is a first position adjacent in the first direction on the two-dimensional plane of the control object, a second position adjacent in the second direction, a third position adjacent in the third direction, And a fourth position adjacent in the fourth direction, a fifth position adjacent to the first position in the second direction, a sixth position adjacent to the second position in the third direction, and adjacent to the third position in the fourth direction. 7th and 4th positions Appropriateness when the control object has each action a at the current position s of the control object with a communication means for communicating with another control object existing at the eighth position adjacent in one direction It is controlled on the basis of one value function that represents and is controlled to move to any one of the first to fourth positions on the two-dimensional plane. A virtual existence that occurs with the movement of the object or moves in the direction opposite to the direction in which the control object moves is defined as a void, and (G-1) the control object is moved to the target position using the value function. The control object is controlled to move to another target position far from the predetermined entrance position, and (G-2) a position where the control object is not included in the set of target positions. The control object moves toward a predetermined entrance position. (V-1) If the void is located at a position that is not included in the target position set or at a predetermined entrance position, the void is not included in the target position set that is far from the predetermined entrance position. (V-2) When the void is located at any of the target positions excluding the predetermined inlet position, the action selection is controlled so that the void moves toward the predetermined inlet position. Includes steps. In the action selection step, the first control object that determines the head control object close to the entrance position and moves by the control of (G-1) and (G-2), and the generation of the first control result The second control is performed to move the void as long as it follows the control of (V-1) and (V-2).

  According to the present invention, as will be described in detail later, a Markov state space as much as necessary for one robot is prepared, and using it, a robot's action policy at each position is determined using dynamic programming. By calculating and using its action strategy, it is possible for a large number of robots to maintain a state where the robots are in contact with each other, corresponding to an arbitrary formation shape of the robot and an arbitrary obstacle shape in the mission environment. A formation algorithm can be obtained. That is, the self-position coordinate definition formation formation algorithm can be obtained with the planned calculation load of one robot without depending on the number of robots.

The figure for demonstrating the mission which many robots move from the formation formation state in a starting position in cooperation, and form formation at a target position. The figure for demonstrating the movement of an expansion-contraction type robot. The figure for demonstrating the movement of an invariant robot. The figure for demonstrating calculation of Q function when two or more routes from a starting position are considered. The figure for demonstrating the mission which a large number of expansion-contraction type robots move from the formation formation state in a start position in cooperation, and form formation in a target position. The figure for demonstrating the mission which a large number of expansion-contraction type robots move from the formation formation state in a start position in cooperation, and form formation in a target position. The figure for demonstrating the mission which a large number of expansion-contraction type robots move from the formation formation state in a start position in cooperation, and form formation in a target position. The figure for demonstrating the mission which a large number of expansion-contraction type robots move from the formation formation state in a start position in cooperation, and form formation in a target position. The figure for demonstrating the mission which a large number of expansion-contraction type robots move from the formation formation state in a start position in cooperation, and form formation in a target position. The figure for demonstrating the method of many robots moving in a formation using value function Q (s, a). The figure which shows the example of the processing flow for demonstrating 1st control in the case of controlling only an expansion-contraction type robot. The figure which shows the example of the processing flow for demonstrating 2nd control in the case of controlling only an expansion-contraction type robot. The figure for demonstrating the mission which many invariant type robots cooperate and move from the formation formation state in a starting position, and form a formation in a target position. The figure for demonstrating the mission which many invariant type robots cooperate and move from the formation formation state in a starting position, and form a formation in a target position. The figure for demonstrating the mission which many invariant type robots cooperate and move from the formation formation state in a starting position, and form a formation in a target position. The figure for demonstrating the mission which many invariant type robots cooperate and move from the formation formation state in a starting position, and form a formation in a target position. The figure which shows the example of the processing flow for demonstrating the 1st control in the case of controlling when controlling only an invariant robot, and when a telescopic robot and an invariant robot coexist. The figure which shows the example of the processing flow for demonstrating the 2nd control in the case of controlling when controlling only an invariant robot, and when a telescopic robot and an invariant robot coexist. The functional block diagram of the action control system which concerns on 1st embodiment. The figure which shows the example of the processing flow of the action control system which concerns on 1st embodiment. FIG. 21A is a diagram illustrating adjacent communicable positions of the invariable robot and the contracted telescopic robot, and FIG. 21B is a diagram illustrating adjacent communicable positions of the expanded telescopic robot. The figure which shows the example in case each lattice is a rhombus.

  Hereinafter, embodiments of the present invention will be described. In the drawings used for the following description, constituent parts having the same function and steps for performing the same process are denoted by the same reference numerals, and redundant description is omitted.

<First embodiment>
[Theoretical background]
First, the theoretical background of the behavior control system and method will be described. Hereinafter, a case where the control target that is the target of behavior control is a robot will be described as an example, but the control target may be other than the robot as long as it can be a target of control.

  A number of robots move from the formation state at the start position in cooperation with each other, and the task of forming the formation at the target position is, for example, from the start position in the room separated by walls as illustrated in FIG. This is realized by moving a plurality of robots.

  In FIG. 1, a grid in which R is described indicates a position where the robot exists, and a grid in which O is described indicates a position where an obstacle exists. An area surrounded by a thick broken line indicates a set of start positions, and an area surrounded by a thick dashed line is an area represented by a set G of target positions (hereinafter, this area is referred to as “target row area”). G ”), and an area surrounded by a thick solid line indicates an entrance position Pe of a target platoon area G to be described later. In this way, each start position and each target position are adjacent to other start positions and target positions in at least one of the vertical and horizontal directions, respectively, and the formation at the start position and the target position of the robot is a lump. Any shape.

  The robots that perform the mission are p units (p is an integer of 2 or more, for example, p ≧ 50), and each robot can move in the X-axis direction and the Y-axis direction on the two-dimensional plane. That is, in this example, each robot can move in four directions, up, down, left, and right with respect to the paper surface of FIG. However, each robot is adjacent to at least one other robot, and p robots form one group. Here, that a certain robot and another robot are adjacent means that a certain robot and another robot share one side or one vertex.

  Of the p robots, q robots (q is any integer between 0 and p) are telescopic robots, and (pq) robots of the p robots are invariant robots. . That is, when q = 0, only invariant robots are available, when q = p, only telescopic robots are available, and when q is an integer between 1 and less than p, immutable robots and telescopic robots are mixed. .

  The telescopic robot is, for example, a square robot. As shown in FIG. 2, the length of two sides parallel to the traveling direction is extended by the length of one side of the robot, and the extended side is then extended. Assume that one cell is moved by contracting to the original length. In other words, as shown in FIG. 2, the telescopic robot includes a contracted state that occupies only one position and an extended state that occupies two adjacent positions in any of the up, down, left, and right directions. Move from one position to another by transforming from a contracted state that occupies only a certain position to an expanded state that occupies a certain position and another position, and deforming to a contracted state that occupies only another position . The telescopic robot is either stationary or deformed from the contracted state to the extended state or deformed from the extended state to the contracted state by one action control.

  On the other hand, as shown in FIG. 3, the invariant robot does not deform and occupies only one position and either remains stationary or moves to an adjacent position in the up / down / left / right directions by one action control. Controlled to move.

  There can be only one robot in each grid in FIG. Each robot is assumed to be stationary if there are obstacles or other robots in the direction of movement. However, if the lattice becomes empty as a result of the movement of another robot, it can be moved to that lattice.

  Each robot is one of the surrounding masses of the robot (8 masses when the invariant robot and the telescopic robot are contracted, and 10 masses when the telescopic robot is expanded). It is assumed that the robot moves while maintaining the state where another robot exists. Also, it is assumed that any robot can communicate with other robots existing in the mass around the robot. This method has the advantage that one robot itself can accurately measure the amount of movement, and by measuring the relative position of neighboring robots that share one side or vertex, The position of each robot in the entire group of robots can be easily known. For example, it is possible to move as a whole by connecting a plurality of robots and propagating expansion and contraction from the leading robot.

  As shown in FIG. 1, the formation shape at the start position and the target position of the robot is a continuous arbitrary shape. In other words, each start position and each target position are adjacent to other start positions and target positions in at least one of the up, down, left, and right directions, respectively, and the entire target position and the entire start position are each in an arbitrary shape. .

  The initial position of each robot i (i represents the robot number, i = 0,1,2, ..., p-1) is (Xr0 [i], Yr0 [i]), and the target position is (Xre [i], Yre [i]), consider the problem of obtaining an action plan for the robot placed at the initial position to move to the target position. Note that G is a set of target positions (Xre [i], Yre [i]).

  When a Markov state transition model is simply applied to such a problem, the Markov state space depends on the position of robot i (Xr [i], Yr [i]) and the action a [i] of robot i. Composed. Each state (robot position and action) is represented by discrete values. When a room is represented by a two-dimensional plane composed of an orthogonal coordinate system of X and Y, each position is represented by a discrete representation of the X axis and the Y axis. That is, as shown in FIG. 1, the room (two-dimensional plane) is divided by a grid, and each grid corresponds to each position. In each grid, “present / none” of the obstacle is set in advance.

In this example, the action subject that is the control target is each robot arranged in the room. The action a [i] ∈D [i] of the robot i takes one of a total of five types, that is, stationary and movement of one lattice in the vertical and horizontal directions. That is, each action is defined as follows, for example, with D [i] ε {0,1,2,3,4}.
0: stationary
1: Move one grid to the right on the 2D plane
2: Move one grid upward on a two-dimensional plane
3: Move one grid to the left on the 2D plane
4: Move down one grid on a two-dimensional plane

  The Markov state space in such a mission environment has a state of the number of dimensions of the number of robots × 2, and the number of selectable actions exists by the number of robot actions (= 5). For example, if the number of robots is 50 and the number of grids in the vertical and horizontal directions of the room is 20, the number of states becomes 20 to the 100th power, and the amount of resources required for the search calculation becomes enormous. Furthermore, as the number of robots increases by one, the number of states will increase by 400 times. Even if the constraint condition that the robots are in contact with each other is incorporated, it is difficult to reduce the fundamental amount of computation, and multiple robots It has become a big problem when using.

  Therefore, in this embodiment, in order to avoid such an explosion of the state space, the Markov state space used for learning is configured only by the state variables of one robot. That is, state variables and behavior variables are defined as follows.

State variable s = (Xr, Yr), action variable a∈ {0,1,2,3,4}
All the p robots share a single value function Q (s, a) with this state variable as an argument, and make an action decision.

Note that the calculation of the value function Q (s, a) is performed in advance of the task using, for example, dynamic programming. First, the target position of each robot is not strictly allocated here, and the set of the entire target position is defined as a target platoon area G. That is,
(Xre [i], Yre [i]) ∈G… (3)
Assuming that each robot can freely set all the positions in G as target positions. In other words, G is treated just like a fluid pourer, and its inlet position is Pe. In other words, each robot can enter G from the entrance position Pe set at one point on the boundary of G, and once entered into G, the robot cannot take action to exit G . In the present embodiment, a point close to the start position of an arbitrary shape (star shape) composed of G is selected as the entrance position Pe, and a high reward value is given when the robot moves to the entrance position Pe. Others receive low rewards. The position of the entrance position Pe is inside G, and any point can be used as long as it is one point of an arbitrary shape of a lump formed by G. It is effective for smooth operation.

(About Q function calculation)
The state transition probability and reward value setting in the Markov state space necessary for calculating the Q function are determined by assuming that the state (position) has moved from s to s' by action a. Then, for example, the transition probability P (s ′ | s, a) (probability of transition from state s to state s ′ by action a) is set as follows.
(1) When s is not an obstacle and s' is the destination cell of action a,
P (s' | s, a) ← 1
(2) When s is not an obstacle and s' is not the destination cell of action a,
P (s' | s, a) ← 0
(3) When s is an obstacle and s' = s,
P (s' | s, a) ← 1
(4) When s is an obstacle and not s' = s,
P (s' | s, a) ← 0
Also, reward R (s ′, s, a) (reward when transitioning from state s to state s ′ by action a) is set as follows.
(1) If s '= Pe then R (s', s, a) ← r_plus
(When the state s ′ and the entrance position match, the reward R (s ′, s, a) is set to r_plus.)
(2) Else If s is in G and s 'is out of G then R (s', s, a) ← -r_minus
(That is, when the condition of (1) is not satisfied and the user wants to go out of the set of target positions, the reward R (s ′, s, a) is set to −r_minus.)
(3) thenR (s', s, a) =-r_minus where Else If s is outside G and s' is within G and s' = Pe is not
(In other words, when the conditions of (1) and (2) are not satisfied and the user tries to enter the set of target positions from a position other than the entrance position Pe, the reward R (s ′, s, a) is -r_minus)
(4) Else R (s', s, a) = 0
Here, it is desirable that r_minus and r_plus are both positive values and r_minus> 10 × r_plus. Note that the transition probability and reward setting method is not limited to this as long as the conditions that the robot does not enter the obstacle, cannot enter the G other than the entrance position Pe, and cannot exit the G are described. .

  In this way, the Q function is learned so that a high reward value can be obtained when the control object moves to the predetermined entrance position Pe at all positions.

(Calculation of Q function when two or more routes the robot group follows from the start position)
When considering the shortest path from a plurality of start positions to the entrance position Pe, there may be two or more shortest paths that the robot group follows depending on the setting of the plurality of start positions and the arrangement of obstacles. For example, in the case of FIG. 4, the shortest path for robot A is RA, but the shortest path for robot B is RB. In this case, if the paths of all the robots are determined by one Q function, there is a possibility that the robot does not form one group and splits. Then, the premise that a group of p robots forms one group breaks down, and it may be difficult to acquire self-position coordinates. Therefore, it is effective to learn the Q function as follows in the motion plan before starting the robot motion.
(1) First, search calculation by dynamic programming is performed for all positions by the method described above, and the value of the Q function at each position is obtained.
(2) Next, search calculation by dynamic programming is performed only on the set of start positions. At this time, in the value of the calculated Q function as a result of (1), the maximum Q value (Qmax_start (s)) of all actions at that position in the set of start positions. ) However, the value of the Q function at the maximum position is fixed, and the value is not updated. For example, in the case of FIG. 4, in the values of Qmax_start at each position belonging to the set start position, if the value of Qmax_start for the start position P _A of the robot A is the largest, and a fixed value of Q function for the position P _A The value is not updated. Furthermore, the transition probability P, the action a, prohibits action escapes out of the starting position without passing through the position P _A. For example, the transition probability P (s ′ | s, a) is set as follows.
(A-1) When s is not an obstacle, s 'is the destination cell of action a, and s' is included in the set of start positions,
P (s' | s, a) ← 1
(A-2) When s is not an obstacle, s 'is the destination cell of action a, and s' is not included in the set of starting positions (outside the starting line),
P (s' | s, a) ← 0
(A-3) When s is not an obstacle, s 'is not the destination cell of action a, and s' is included in the set of start positions,
P (s' | s, a) ← 0
(A-4) When s is not an obstacle, s 'is not the destination cell of action a, and s' is not included in the set of starting lines
If s '= s, P (s' | s, a) ← 1
Otherwise, P (s' | s, a) ← 0
(A-5) When s is an obstacle and s' = s,
P (s' | s, a) ← 1
(A-6) When s is an obstacle and not s' = s,
P (s' | s, a) ← 0

  Depending on the setting of multiple start positions and the placement of obstacles, there may be cases in which a group of robots that freely move without restraints is divided into two or more groups according to the action policy calculated by the dynamic calculation method. is there. In such a case, it is inconvenient when it is desired to maintain the connection between adjacent robots, and the search calculation as described above is performed in order to solve the problem. That is, the action policy is determined so that all robots aim at the position where the Qmax_start value is the highest in the set of start positions, and the robot comes out of the start position because the Qmax_start value is the most within the start position. Set to be only via high positions. In other words, the finally obtained value function Q is such that the value function of the action aiming at the position of the point with the largest Qmax_start value learned in (1) among all the start positions becomes higher (and (In other words, the robot moves through the position so that the robot moves out of the starting position from the starting position).

(About action selection methods and two policies)
Normally, when a robot in each state s selects an action using the value function Q (s, a), the action a that maximizes the value of the Q function is selected in each state s. However, if the Q function is calculated with the condition settings presented above, both the robot outside the target platoon area G and the robot inside the target platoon area G will move toward the entrance position Pe. . Therefore, action selection is performed according to the following rules. This action policy is defined as action policy Policy_G.

(Action Policy Policy_G)
(1) When s is in G, the robot is an action that does not move to the position of other robots or obstacles, and does not go out of G, and Q (s, a) <The action that minimizes Q (s, a) is selected from actions a that satisfy Q (s, 0). If there is no such action, action a = 0 (still) is selected.
(2) When s is out of G, the robot does not move to the position of other robots or obstacles and satisfies Q (s, a)> Q (s, 0) a Select the action that maximizes Q (s, a). If there is no such action, action a = 0 (still) is selected.

  By doing in this way, according to (1), once the robot enters G, it moves to a position in G that is as far away as possible from the entrance position Pe. On the other hand, according to (2), the robot outside G moves toward the entrance position Pe and can take an action to enter the G from the entrance position Pe. Also, collisions with other robots and obstacles can be avoided.

(Action Policy Policy_V)
The concept of controlling the position of the void is used. The term “void” as used herein refers to a gap that can be made to the original position after a certain robot moves to another position. In addition, when the telescopic robot expands from a contracted state that occupies only a certain position and deforms to an expanded state that occupies the originally occupied position and another position, a void is generated at the originally occupied position. Alternatively, it is assumed that the void moves to the position originally occupied (see FIG. 2). On the other hand, when the invariant robot moves from one position to another, a void is generated or moved at a certain position (see FIG. 3). One flock is maintained by controlling the movement of this void. The action policy Policy_V for void control, which will be explained later,
(3) When s is outside G or at the entrance position Pe, the void is an action in which the moving destination overlaps with the position of another robot and does not enter the G. In addition, an action that minimizes Q (s, a) is selected from actions a that satisfy Q (s, a) <Q (s, 0). If there is no such action, action a = 0 (still) is selected. By making such a selection, the void is controlled to move to a position not included in the set of target positions far from the entrance position Pe.
(4) When s is within G and not at the entrance position Pe, the void is an action whose destination overlaps the position of another robot, and Q (s, a)> Q (s, 0 The action that maximizes Q (s, a) is selected from the actions a that satisfy). If there is no such action, action a = 0 (still) is selected. By making such a selection, the void is controlled to move toward the entrance position Pe.

(Restraint condition)
In this embodiment, the robot can measure the absolute position of each robot based on the relative positional relationship between the robots, and whether there is another robot at an adjacent position or whether there is an obstacle. And whether it is on the target position. Since the constraint conditions for realizing this vary depending on the configuration of the robot to be controlled, each will be described below.

(Restriction condition when controlling only telescopic robot)
When controlling only a telescopic robot, the positional relationship and the constraint conditions related to movement are as follows: one of the grids around each robot (10 squares in the stretched state, 8 squares in the contracted state) The robot must exist and the constraint condition that the entire robot forms one set must be satisfied. Examples of operations for forming a formation at the target position while maintaining this constraint condition are shown in FIGS. Hereinafter, a method of platoon movement using the value function Q (s, a) while maintaining this condition will be described with reference to FIGS.

  All robots are placed at positions (also referred to as start row positions) included in the set of start positions, and the head movement flag flg ← 0 is set (s1).

  The value of the head movement flag flg is determined (s3). The head movement flag is information indicating whether or not the robot close to the entrance position (hereinafter also referred to as “head robot”) is to be moved. When flg = 0, control for moving the head robot (hereinafter, this process is referred to as “ (Also referred to as “first control”) (s5). When flg = 1, control (hereinafter, this process is also referred to as “second“ second control ”) (s6) for moving a robot other than the head robot is performed. Note that the first control is a control that generates a void by deciding the head robot and moving it according to the action policy Policy_G, and the second control is a control that moves the void generated as a result of the first control as long as it follows the action policy Policy_V. I can say that.

  Details of the first control will be described. In the first control, first, robot ranking, selection of the leading robot, generation of voids (setting), and behavior selection of the leading robot are performed based on the behavior policy Policy_G (s51).

For example, in s51, the robots are ranked as follows.
(1) For each robot i in G, the minimum value Qmin (i) of Q (s, a) in all actions a at the position s of the robot i is calculated.
(2) For each robot i in G, ranking is performed in ascending order of Qmin (i).
(3) For each robot i outside G, the maximum value Qmax (i) of Q (s, a) in all actions a at the position s of the robot i is calculated.
(4) For each robot i outside G, ranking is performed in descending order of Qmax (i).
(5) Add the number of robots in G to the ranking value of each robot i outside G.

  Next, for example, the top robot is selected as follows. Of the robots that select an action other than a = 0 (stationary) by the action policy Policy_G, the robot with the highest rank obtained in the above ranking is selected and is set as i_select.

  Next, for example, voids are generated (set) as follows. Set the position of robot i_select as the current void position (void_x, void_y).

  Thereafter, the action selection of the robot i_select is performed by the action policy Policy_G (s51 so far).

  In the process subsequent to the first control, the head movement flag flg is updated to flg ← 1, and the robot i_select selected based on the action selected by the action policy Policy_G is moved (s56). The robot i_select is deformed from the contracted state to the extended state.

  Next, the second control (s6) will be described.

  In the second control, the robot i_select is deformed from the extended state to the contracted state, and at the same time, the subsequent robot is extended to the current void position. For this purpose, the following processes s61 to s64 are performed. When the current void position (void_x, void_y) is s, the action of the void is selected by the action policy Policy_V (s61). It is determined whether or not the selected action a_void moves (a_void ≠ 0) (s62). When a_void ≠ 0, the void is moved by the action policy Policy_V, the robot at the position of the movement destination is selected, and the robot is set as the robot i_select2. The action of the robot i_select2 is determined to move in the direction opposite to the action a_void (s63). The robot i_select2 is deformed from the contracted state to the extended state, and at the same time, the robot i_select is contracted simultaneously. Subsequently, i_select is replaced with i_select2 (i_select ← i_select2). When a_void = 0 (when the void is stationary), flg ← 0 is set (s64).

  Based on the moved position, the information on the positions of all the robots is updated (s8).

  It is determined whether or not all the robots have entered G (s9). If entered, the behavior control process is terminated. If there is a robot that has not entered, the processing of s3 to s9 is repeated. The state where all the robots are in G means a state in which a part of the extended robot is not in G and a contracted robot is in G.

  The operation realized by the above method will be described. First, among the robots in the starting formation form, the robot closest to the entrance position Pe is selected from the formation and moved one step toward the entry position Pe (first control). In subsequent time steps, subsequent robots move step by step so as to fill in the void positions created in the previous step. This is repeated until the void cannot move to a position where the Q function value is lower than this (second control). When the void becomes impossible to move, the robot closest to the entrance position Pe is selected from the platoon (first control). As the first control and the second control are repeated, the leading robot reaches the entrance position Pe and enters G. After that, the leading robot will continue to move toward the position with the lowest Q function value in G, and the following robot will continue to move to fill the void created by the leading robot. .

  With the above operation, the group of robots approaches the target line.

(Restriction conditions when controlling only invariant robots)
Next, a constraint condition in the case of controlling only the invariant robot will be described. In this case, the robot moves as many other robots as possible in contact with the robot in the direction opposite to the direction in which the robot moves.

  In this case, it is necessary to satisfy the constraint that another robot exists in one of the eight grids around each robot, and that the entire robot forms one set. Examples of operations for forming a formation at the target position while maintaining this constraint condition are shown in FIGS. In the following, regarding the method of platoon movement using the value function Q (s, a) while maintaining this condition, the processing contents of the first control and the second control are mainly the case of controlling only the telescopic robot. Since this is different, this portion will be mainly described with reference to FIGS.

  In the first control, first, robot ranking, selection of the leading robot, generation of voids (setting), and behavior selection of the leading robot are performed based on the behavior policy Policy_G (s51). This s51 is the same as the case where only the telescopic robot is controlled. However, in this case where an invariant robot is used, the following processing s511 is performed following s51.

  The variable Ir ← 0 is set, and an array for storing the numbers of robots to be moved at a time in the first control is i_select_move. The robot i_select selected in s51 is i_select_move [Ir], the action selected by the robot i_select_move [Ir] is the same as the action selected by the robot i_select, and is a_select (s511).

  Subsequently, the action of the void is selected by the action policy Policy_V when the current void position (void_x, void_y) is s (s53). It is determined whether or not the selected action a_void is an action opposite to the action a_select (s54).

  If the action is not an action in the reverse direction, the head movement flag flg is updated to flg ← 1, and the robots i_select_move [0] to i_select_move [Ir] are moved in the direction of the action a_select (s56).

  In the case of a reverse action, the variable Ir is incremented and the void is moved by the action a_void. Further, the robot at the position of the destination void is selected, and the robot is set as i_select_move [Ir]. The behavior to be selected by the robot i_select_move [Ir] is the same as the behavior a_select obtained in s51 and s511. The process returns to s53 (s55).

  Steps s53 to s55 are repeated until an action other than a reverse action is performed in s54.

  In the second control, the action a_void is selected by the action policy Policy_V when the variable Ir ← 0 and the current void position (void_x, void_y) is s (s61). It is determined whether or not the selected action a_void moves (a_void ≠ 0) (s62). When a_void = 0 (when the void is stationary), the head movement flag flg is updated, flg ← 0 is set (s64), and the second control is terminated. If a_void ≠ 0, the void is moved by the action policy Policy_V, the robot at the position of the movement destination is selected, and the robot is set as robot i_select_move [Ir]. Thereafter, the action to be taken by the robot i_select_move [Ir] is set to an action opposite to the action a_void (s63). The moving direction of the robot i_select_move [Ir] at this time is a_select.

  Subsequently, the action a_void is selected by the action policy Policy_V when the current void position (void_x, void_y) after the movement is s (s66). It is determined whether or not the selected action a_void is an action opposite to the action a_select (s67). If the action is not in the reverse direction, the robot i_select_move [0] to i_select_move [Ir] selected based on the action a_select selected in s63 is moved. (S69). In the case of a reverse action, the variable Ir is incremented and the void is moved by the action a_void. Further, the robot at the position of the destination void is selected, and the robot is set as i_select_move [Ir]. The action to be selected by the robot i_select_move [Ir] is the same as the action a_select obtained in s63. The process returns to s66 (s68).

  S66-s68 are repeated until it becomes other than a reverse action in S67.

  In the above process, when a subsequent invariant robot adjacent in the direction opposite to the movement direction of a certain previous invariant robot moves a void generated by the movement of the preceding invariant robot according to the control of the action policy Policy_V When moving in the direction opposite to the moving direction, as long as the moving direction is the same as the moving direction of the preceding invariant robot, it is controlled to move simultaneously. Specifically, first, the robot closest to the entrance position Pe is selected from the formation among the robots in the starting formation form, and the action to go to the entrance position Pe is selected. As far as possible, the robot abutting on the opposite side of the moving direction of the robot (the action a_void of the void selected by the action policy Policy_V in s53 and the action a_select of the robot selected by the action policy Policy_G in s511) (As long as and are the movements in the opposite direction), it moves one step (first control). In subsequent time steps, it can move in the same direction so as to fill the position of the void created in the previous step (the movement direction of action a_void selected by action policy Policy_V in s63 and s66 is the same) The robot moves one step at a time. The process is repeated until the void cannot move to a position where the Q function value is lower than this (second control). When the void becomes impossible to move, the robot closest to the entrance position Pe is selected from the formation (first control). As the first control and the second control are repeated, the leading robot reaches the entrance position Pe and enters G. After that, this time, the leading robot will continue to move with the following robot aiming at the position with the lowest Q function value in G, and the following robot will continue to fill the void created by the leading robot To move the following robot.

  With the above operation, the group of robots approaches the target line.

(Restriction condition when controlling when telescopic robot and immutable robot are mixed)
Furthermore, the case where a telescopic robot and an immutable robot are mixed will be described.

  In the following, there are other robots in either of the grids around each robot (10 squares in the stretched state of the extendable robot, 8 squares in the contracted state of the telescopic robot and 8 squares in the case of the invariant robot). The constraint condition that the entire robot forms one set must be satisfied. In the following, regarding the method of platoon movement using the value function Q (s, a) while maintaining this condition, the control contents of the invariant type robot and the processing contents of the first control and the second control are mainly described. Since this is different, this portion will be mainly described with reference to FIGS. As a new variable, compress [i] is defined as a variable indicating whether each robot i is a telescopic robot or an invariant robot. When compress [i] = 1, the robot i is a contractive robot, and when compress [i] = 0, the robot i is an invariant robot.

  In the first control, the following processes s51 to s56 are performed. Since s51, s511, and s53 to 56 are the same as in the case of controlling only the invariant robot, the description will be simplified.

  The robot is ranked, the head robot is selected, the void is generated (set), and the head robot is selected based on the action policy Policy_G (s51), and the moving robot array and action are set (s511).

  It is determined whether or not the selected robot is an immutable robot (compress [i] = 0?) (S52).

  In the case of the telescopic robot, the head movement flag flg is updated, flg ← 1, and the robots i_select_move [0] to i_select_move [Ir] selected based on the action a_select selected by the action policy Policy_G are moved. (S56). Here, if the robots included in the robots i_select_move [0] to i_select_move [Ir] are immutable robots, the robots are moved to the adjacent grid without being deformed, and if they are telescopic robots, the robots are contracted. It is deformed from the extended state.

  In the case of an invariant robot, s53 (action selection of a void by action policy Policy_V) and s54 described in the case of controlling only an invariant robot and s54 (whether the movement direction selected by the selected robot is opposite to the movement direction of the void) Whether or not).

  If the action is other than the reverse action, the head movement flag flg is updated, flg ← 1, and the robots i_select_move [0] to i_select_move [Ir] selected based on the action a_select selected by the action policy Policy_G are moved. (S56). In the case of an action in the opposite direction, the void is moved, the robot at the destination of the void is selected, and the action is set to move in the same direction as the previous robot (s55).

  Steps s52 to s55 are repeated until the telescopic robot is selected (s52) or until the selected moving direction of the selected robot is different from the moving direction of the void (s54).

  In the second control described later, when the second control is terminated because the telescopic robot is selected, the telescopic robot is deformed from the expanded state to the contracted state simultaneously with the above-described processes s51 to s56. Let

  In the second control, the following processes s61 to s69 are performed. Since s61 to s64 and s66 to 69 are the same as the case of controlling only the invariant robot, the description will be simplified.

  Action a_void is selected by action policy Policy_V (s61). It is determined whether or not the selected action a_void moves (a_void ≠ 0) (s62). If a_void ≠ 0, the void is moved, the robot at the destination position is selected, and the robot i_select [Ir The action to be taken is the action opposite to the action a_void (s63). If a_void = 0 (when the void is stationary), the head movement flag flg is updated, flg ← 0 is set (s64), and the second control is terminated.

  It is determined whether or not the selected robot is an immutable robot (compress [i] = 0?) (S65).

  In the case of the telescopic robot, the robots i_select_move [0] to i_select_move [Ir] selected based on the action a_select selected in s63 are moved (s69). In the case of a telescopic robot, the robot i_select [Ir] is deformed from the contracted state to the extended state.

  If the result of the determination in s65 is an invariant robot, the action a_void is subsequently selected by the action policy Policy_V (s66). It is determined whether or not the selected action a_void is an action opposite to the action a_select (s67). If the action a_void is not the opposite action, the robot i_select_move [selected based on the action a_select selected in s63 is determined. [0] to i_select_move [Ir] are moved (s69). In the case of a backward action, the void is moved, the robot at the position of the destination void is selected, and the action to be selected by the robot is the same as the action a_select obtained in s63 (s68).

  Steps s65 to s68 are repeated until the telescopic robot is selected (s65) or until the selected moving direction of the selected robot and the moving direction of the void are not in opposite directions (s67).

  When the first control is terminated because the telescopic robot is selected in the first control, the telescopic robot is deformed from the expanded state to the contracted state simultaneously with the above-described processes s61 to s69.

  In the above process, the movement of the telescopic robot is controlled, and the subsequent invariant robot adjacent in the direction opposite to the direction of movement of a certain preceding invariant robot has a void generated by the movement of the preceding invariant robot. When moving according to the control of the action policy Policy_V, the movement direction is controlled to move simultaneously as long as the movement direction is the same as the movement direction of the preceding invariant robot. Specifically, first, the robot closest to the entrance position Pe is selected from the formation among the robots in the starting formation form, and the action to go to the entrance position Pe is selected. Then, the robot in contact with the robot in the direction opposite to the moving direction of the robot moves and moves one step. If there is a retractable robot in the robot, the robot moves with the retractable robot as the rear end. One control). In subsequent time steps, subsequent robots that move in the same direction so as to fill in the void positions created in the previous step move one step at a time. If there is a robot that can be moved, the robot moves with the retractable robot as the rear end. The process is repeated until the void cannot move to a position where the Q function value is lower than this (second control). When the void becomes impossible to move, the robot closest to the entrance position Pe is selected from the platoon and performs the first control. As the first control and the second control are repeated, the leading robot reaches the entrance position Pe and enters G. After that, this time, the leading robot will continue to move with the following robot aiming at the position with the lowest Q function value in G, and the following robot will continue to fill the void created by the leading robot To move the following robot.

  With the above operation, the group of robots approaches the target line.

Hereinafter, a robot movement control method and processing for realizing the constraint conditions will be described.
<Action control system 100 according to the first embodiment>
FIG. 19 is a functional block diagram of the behavior control system 100 according to the first embodiment, and FIG. 20 shows an example of the processing flow. As illustrated in FIG. 19, the behavior control system 100 includes an operation planning unit 110, a behavior selection unit 120, an adjacent state determination unit 121, a position update unit 125, a position determination unit 126, a storage unit 140, and communication. Part 150 and input part 160.

  Hereinafter, a case where the control target to be controlled is a robot will be described as an example. Of course, the control object may be other than the robot as long as it can be a control target.

  In the present embodiment, the behavior control system 100 controls the behavior of p robots and is mounted on one of the p robots. Of the p robots, q robots (q is any integer between 0 and p) are telescopic robots, and (pq) robots of the p robots are invariant robots. . The robot on which the behavior control system 100 is mounted may be an invariant robot or a telescopic robot. Note that the p-1 robot on which the behavior control system 100 is not mounted also includes the communication unit 150 and the adjacent state determination unit 121.

<Operation Planning Unit 110>
The motion planning unit 110 calculates the value of the value function Q (s, a) in advance by dynamic programming before starting the mission action of the robot (S110) and stores it in the storage unit 140. Here, the calculation of the motion planning unit 110 may be replaced with Q learning using a single robot. If the value function Q (s, a) is calculated by a separate device and stored in the storage unit 140 in advance before starting the robot's mission behavior, the behavior control system 100 includes the motion planning unit 110. Not necessary.

  The calculation of the value function Q (s, a) is as described in (Calculation of the Q function) and (Calculation of the Q function when two or more routes of the robot group from the start position are considered). is there.

<Input unit 160>
The input unit 160 includes an initial position (Xr0 [i], Xr0 [i]) and a set of p target positions G = {(Xre [0], Yre [0]), (Xre [1], Yre [1]),..., (Xre [p-1], Yre [p-1])} are input and stored in the storage unit 140.

  Note that the target position includes a predetermined entrance position Pe. Information about the entrance position Pe is also input from the input unit 160 and stored in the storage unit 140.

<Storage unit 140>
It is assumed that the storage unit 140 stores a value function Q (s, a) for each of the combinations of the position s and aε {0,1,2,3,4}. The range that s can take is all the coordinates where the robot i in the region on the target two-dimensional plane can exist.

  It is assumed that the reward r (s) at each position s is also stored in the storage unit 140. Information about the reward r (s) at each position s is input from the input unit 160, for example.

<Communication unit 150>
All robots including the robot in which the behavior control system 100 is mounted are connected to the upper, upper right, right, lower right, lower, lower left, left, upper left directions (hereinafter “ It is also possible to communicate with other adjacent robots in the “8 directions”. The satin portion in FIG. 21A shows the communicable position of the invariable robot and the retractable telescopic robot adjacent to each other, and the satin portion in FIG. 21B shows the communicable position of the stretchable telescopic robot adjacent to each other.

<Action selection unit 120>
The action selection unit 120 extracts the value function Q from the storage unit 140. The action selection unit 120 performs control when (constraint condition when controlling only the telescopic robot), (restraint condition when controlling only the invariant robot), and (extractable robot and immutable robot are mixed). P robots are controlled by the method described in the section (Restrictive conditions in case) (s120). Specifically, s1 (initial setting), s3 (determination of the head movement flag), s5 (first control), and s6 (second control) in FIG. 10 are performed. In addition, in the action policy Policy_G and the action policy Policy_V, the position of each robot and the position of an obstacle are required, but the initial position (Xr0 [i], Xr0 [i]) of each robot is as described above. The information input via the input unit 160 and stored in the storage unit 140 is used, and the position of each robot after movement is obtained by the position determination unit 126 described later and stored in the storage unit 140. do it. As for the position of the obstacle, the position of the obstacle used when learning the value function Q (s, a) may be used, or obtained using the determination result determined by the adjacent state determination unit 121 described later. You may use what is available.

<Location update unit 125>
For each i = 0, 1,..., P−1, the position update unit 125 performs the action determined by the action selection unit 120 at the current position (Xr [i], Yr [i]) of the i-th robot. Calculates the position (Xr '[i], Yr' [i]) after movement (after action) of the robot when executed, and stores the calculated (Xr '[i], Yr' [i]) The position of the i-th robot stored in 140 is updated (S125). In other words, the position update unit 125 calculates a position assumed when the robot behaves (hereinafter also referred to as “assumed position”) based on the action determined by the action selection unit 120 and updates the position of the robot. Stored in the storage unit 140. Specifically, s8 (update of position) in FIG. 10 is performed.

<Adjacent state determination unit 121>
The adjacency state determination unit 121 determines whether an obstacle or another robot exists in the upper, lower, left, and right adjacent positions on the two-dimensional plane of the robot, and the upper right, upper left, lower left on the two-dimensional plane of the robot. Then, it is determined whether another robot is present at the lower right adjacent position (S121), and the determination result is stored in the storage unit 140.

  As described above, the p-1 robot on which the behavior control system 100 is not mounted also includes the communication unit 150 and the adjacent state determination unit 121. Therefore, each robot i determines whether there is an obstacle in its up / down / left / right direction in the adjacent state determination unit 121 and whether there is another robot in the surrounding eight directions, and determines the result of the determination via the communication unit 150. It can be output to the control system 100. The behavior control system 100 receives the determination result from each robot i via the communication unit 150 and stores it in the storage unit 140 together with the determination result of the adjacent state determination unit 121 included in the behavior control system 100. Note that the p robots always have other robots in adjacent positions (eight directions) of each robot, and the group formed by the adjacent robots is a lump, so each robot i is connected via the communication unit 150. The p-1 determination results can be transmitted to the behavior control system 100 directly or via another robot. In addition, the behavior control system 100 can transmit a control signal so as to execute the selected behavior to each robot i directly or via another robot via the communication unit 150. In addition, other information can be transmitted and received between the p robots.

  Even if the robot is controlled to move, it may not be able to move as controlled by the action selection unit 120 due to some trouble (the presence of an obstacle that could not be detected, equipment failure, etc.). On the other hand, the position of the robot that did not move is the same as before the control. Therefore, using the determination result by the adjacent state determination unit 121 based on the position of the robot that has not moved, the position of the robot that has been controlled to move (hereinafter also referred to as “post-action position”). ) (Xr "[i], Yr" [i]). When the invariant robots are arranged in a line with respect to the traveling direction, since all the robots move in the same direction, there is no reference robot. In such a case, the state is detected, one or more robots are moved by one square in a direction orthogonal to the traveling direction, and the other robots are originally moved with reference to the moved robots. What is necessary is just to move toward a direction.

<Position determination unit 126>
The position determination unit 126 obtains the post-behavior position using the determination result of the adjacent state determination unit 121, and the post-behavior position (Xr "[i], Yr" [i]) and the assumed position (Xr '[i], It is determined whether or not Yr ′ [i]) matches (S126). If they do not match, it is considered that the robot controlled to move could not move as controlled due to some trouble. In this case, at least one of the post-behavior position (Xr "[i], Yr" [i]) and the assumed position (Xr '[i], Yr' [i]) may be corrected. Various methods can be considered as the correction method. For example, all the robots that have moved may be instructed to return to the pre-control position, and the post-behavior position (Xr "[i], Yr" [i]) may be corrected, or the assumed position (Xr '[i], Yr' [i]) may be corrected according to the post-action position (Xr "[i], Yr" [i]).

  At each time step, it is determined whether all robots are in G (S127, corresponding to s9 in FIG. 10). If all lots are in G, the mission is terminated. If not, continue the mission.

  For example, in the target position arrival determination unit (not shown), the post-action position (Xr "[i], Yr" [i]) ∈ output from the position determination unit 126 for each i = 0, 1,. It is determined whether or not G, and if (Xr "[i], Yr" [i]) ∈ G for all i, the mission is terminated. If (Xr ″ [i], Yr ″ [i]) εG is not satisfied for at least one i, the behavior selection unit 120 is controlled to be executed again.

  The processes s120 to s127 described above are performed for each time step.

<Effect>
With such a configuration, the Markov state space required for one robot is prepared, and using it, the robot's action policy at each position is calculated using dynamic programming, and the action policy By using, we obtain a formation algorithm for a large number of robots while maintaining a state where the robots are in contact with each other, corresponding to an arbitrary formation shape of the robot and an arbitrary obstacle shape in the mission environment. be able to. That is, the self-position coordinate definition formation formation algorithm can be obtained with the planned calculation load of one robot without depending on the number of robots. Moreover, since an absolute position can be acquired by determining a relative position with respect to a stationary robot, no additional position measurement equipment is required.

  As described above, the telescopic robot moves by stretching and contracting like a scale insect. Such movement is known to be advantageous for movement on rough terrain and underground compared to movement by feet and wheels, and is particularly effective when used at a disaster site or the like. In the present embodiment, a control method for controlling only such a telescopic robot or a combination with an invariant robot is presented.

<Modification>
In the present embodiment, each lattice (mass) is a square, but may have other shapes. The lattice is continuously arranged in the left-right direction and the up-down direction. Each lattice is adjacent to the other two lattices in the left-right direction and adjacent to the other two lattices in the vertical direction. In other words, each grid is adjacent to other grids only in the same direction that the robot can move. Each lattice may have any shape as long as this condition is satisfied. Further, the term “orthogonal” does not necessarily mean strictly “vertically intersecting”. For example, as shown in FIG. 22, each lattice may be a rhombus, and each lattice is the other two lattices. One of the adjacent directions may be the vertical direction and the other is the horizontal direction. In this case, the vertical direction and the horizontal direction are orthogonal to each other.

  In other words, the control object is in the first direction (for example, the right direction), the second direction (for example, the upward direction) that is not parallel to the first direction, and the first direction on the two-dimensional plane. On the other hand, it is possible to move in the third direction (for example, the left direction) opposite to the second direction, and in the fourth direction (for example, the downward direction) opposite to the second direction. Control is performed so that the current area is moved to one of the adjacent areas in the first direction, the second direction, the third direction, and the fourth direction from the grid. In this case, on the two-dimensional plane of the robot, the position adjacent in the first direction is the first position, the position adjacent in the second direction is the second position, the position adjacent in the third direction is the third position, Positions that are adjacent in the four directions are the fourth position, positions that are adjacent to the first position in the second direction are the fifth positions, positions that are adjacent to the second position in the third direction are the sixth positions, and positions that are adjacent to the third position are the fourth direction May be referred to as the seventh position, and the position adjacent to the fourth position in the first direction as the eighth position. For example, the first to eighth positions correspond to the positions “1” to “8” in FIG.

  In the present embodiment, the example in which the behavior control system 100 is mounted on one of the p robots has been shown, but behavior control may be performed on a control target on a computer. In that case, unless a trouble occurs intentionally, the controlled object moves as controlled, so the assumed position (Xr '[i], Yr' [i]) and the post-movement position (Xr "[i], Yr "[i]) matches. In such a state, an ideal action plan for each control object i may be performed. The actual robot is made to execute the action plan until the mission obtained as a result is completed. The real robot includes a communication unit 150 and an adjacent state determination unit 121. After finishing each action, the real robot determines an adjacent state, obtains a post-behavior position using the determination result, and determines a post-behavior position (Xr "[i] , Yr "[i]) and the assumed position (Xr '[i], Yr' [i]) are determined to match. If they do not match, it is considered that the robot controlled to move could not move as controlled due to some trouble. Subsequent processing may be the same as in the first embodiment, or other processing may be performed. With such a configuration, it is possible to detect at least that movement was not possible as controlled.

<Other variations>
The present invention is not limited to the above-described embodiments and modifications. For example, the various processes described above are not only executed in time series according to the description, but may also be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. In addition, it can change suitably in the range which does not deviate from the meaning of this invention.

<Program and recording medium>
In addition, various processing functions in each device described in the above embodiments and modifications may be realized by a computer. In that case, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on a computer, various processing functions in each of the above devices are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Further, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its storage unit. When executing the process, this computer reads the program stored in its own storage unit and executes the process according to the read program. As another embodiment of this program, a computer may read a program directly from a portable recording medium and execute processing according to the program. Further, each time a program is transferred from the server computer to the computer, processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program includes information provided for processing by the electronic computer and equivalent to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In addition, although each device is configured by executing a predetermined program on a computer, at least a part of these processing contents may be realized by hardware.

Claims (3)

A behavior control system for performing behavior control for moving a plurality of control objects arranged in a set of start positions to a set of target positions including a predetermined entrance position,
The direction not parallel to the first direction is the second direction, the opposite direction to the first direction is the third direction, the opposite direction to the second direction is the fourth direction, and each start position and each The target positions are adjacent to other start positions and target positions in at least one of the first direction to the fourth direction, respectively, and the set of the target positions and the set of the start positions each have an arbitrary shape in a lump. And
The control object includes a first position adjacent in the first direction on the two-dimensional plane of the control object, a second position adjacent in the second direction, a third position adjacent in the third direction, and a fourth position. A fourth position adjacent in the direction, a fifth position adjacent to the first position in the second direction, a sixth position adjacent to the second position in the third direction, a seventh position adjacent to the third position in the fourth direction, And a communication means for communicating with another control object existing in the eighth position adjacent to the fourth position in the first direction, and the control object is a behavior a at the current position s of the control object. It is controlled based on a single value function that represents the appropriateness when taking the measurement, and is controlled to move to any one of the first to fourth positions on the two-dimensional plane. ,
The Markov state space used for learning is composed of only one robot state variable, and all p robots share one value function with this state variable as an argument.
The value function first learns that a high reward value can be obtained when the control object moves to a predetermined entrance position at all positions, and then the value function It was obtained by learning so that the value function of the action aiming at the position of the point with the largest value becomes high,
A storage unit for storing the value function;
A virtual existence that occurs with the movement of the control object or moves in a direction opposite to the direction in which the control object moves is defined as a void, and using the value function, (G-1) If it exists at any of the target positions, the control object is controlled to move to another target position far from the predetermined entrance position, and (G-2) the control object is added to the set of target positions. If it exists at a position not included, the control object is controlled to move toward the predetermined entrance position, and (V-1) a position where a void is not included in the set of target positions or the predetermined position The void is moved to a position that is far from the predetermined inlet position and not included in the set of target positions, and (V-2) the void is excluded from the predetermined inlet position. If it is located at any of the target positions, the void is Comprises action selecting section for controlling to move towards the fixed inlet position,
The action selection unit determines a leading control object close to the entrance position, and generates a void by moving the control object according to the control of (G-1) and (G-2), and the result of the first control The second control for moving the generated void as long as it follows the control of (V-1) and (V-2) is executed.
Behavior control system. A behavior control method for performing behavior control for moving a plurality of control objects arranged in a set of start positions to a set of target positions including a predetermined entrance position,
The direction not parallel to the first direction is the second direction, the opposite direction to the first direction is the third direction, the opposite direction to the second direction is the fourth direction, and each start position and each The target positions are adjacent to other start positions and target positions in at least one of the first direction to the fourth direction, respectively, and the set of the target positions and the set of the start positions each have an arbitrary shape in a lump. And
The control object includes a first position adjacent in the first direction on the two-dimensional plane of the control object, a second position adjacent in the second direction, a third position adjacent in the third direction, and a fourth position. A fourth position adjacent in the direction, a fifth position adjacent to the first position in the second direction, a sixth position adjacent to the second position in the third direction, a seventh position adjacent to the third position in the fourth direction, And a communication means for communicating with another control object existing in the eighth position adjacent to the fourth position in the first direction, and the control object is a behavior a at the current position s of the control object. It is controlled based on a single value function that represents the appropriateness when taking the measurement, and is controlled to move to any one of the first to fourth positions on the two-dimensional plane. ,
The Markov state space used for learning is composed of only one robot state variable, and all p robots share one value function with this state variable as an argument.
The value function first learns that a high reward value can be obtained when the control object moves to a predetermined entrance position at all positions, and then the value function It was obtained by learning so that the value function of the action aiming at the position of the point with the largest value becomes high,
The action selection unit uses a virtual existence that occurs with the movement of the control object or moves in a direction opposite to the direction of movement of the control object as a void, and uses the value function (G-1 ) When the control object exists at any of the target positions, the control object is controlled to move to another target position far from the predetermined entrance position, and (G-2) the control object is If it exists at a position that is not included in the set of target positions, the control object is controlled to move toward the predetermined entrance position, and (V-1) void is included in the set of target positions. The void is moved to a position not included in the set of target positions far from the predetermined entrance position, and (V-2) the void is the predetermined position. If it is located at any of the target positions excluding the entrance position of Comprises action selecting step of controlling such voids are moved toward the predetermined inlet position,
In the action selection step, the first control object that determines the head control object close to the entrance position, and generates a void by being moved by the control of (G-1) and (G-2), and the result of the first control The second control for moving the generated void as long as it follows the control of (V-1) and (V-2) is executed.
Behavior control method. A program for causing a computer to function as the behavior control system according to claim 1 .