JP2014079819A - Robot cooperative conveyance planning device, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot cooperative conveyance planning technology capable of reducing a calculation time required for plan calculation or a memory capacity of a computer, while taking existence of a plurality of robots into consideration.SOLUTION: A Markov state space is constituted hierarchically (an article orbital calculation processing is defined as a first hierarchy, a changeover position determination processing is defined as a second hierarchy, and a moving route planning processing is defined as a third hierarchy), and search for a movement plan is performed from a hierarchy having a low change frequency (first hierarchy), and a search range in a hierarchy having a high change frequency is restricted by using a result of search calculation in a hierarchy having a low change frequency, and search in a lower rank hierarchy (a second hierarchy, a third hierarchy) having a high change frequency is performed in the restricted state space.

Description

  The present invention relates to a robot cooperative transfer planning technique, and more particularly, to a robot cooperative transfer planning technique for obtaining an action plan of each robot for moving a moving object from a start position to a target position in cooperation with a plurality of robots.

  In recent years, research on article transportation by autonomous mobile robots has been actively conducted. Among various types of articles to be transported and the robots that transport them, there is a need for technology to efficiently transport articles by cooperating multiple robots (non-patented). Reference 3). In order to realize efficient cooperative conveyance of articles by a plurality of robots, it is important to plan in advance the arrangement and operation order of the plurality of robots. In such a plan, as a matter of course, it is necessary to sufficiently consider the presence of obstacles and the shape of a route in an actual environment where a plurality of robots operate.

As an effective method for performing such a plan calculation, there are a dynamic programming method and a reinforcement learning method in the Markov decision process, and various studies have been conducted.
However, if such a plan is made using dynamic programming or reinforcement learning in the Markov decision process, the calculation is required when using a plurality of robots as compared to using a single robot. There is a problem that the time and the memory capacity of the computer increase exponentially with respect to the number of robots, and the main cause is the enormous increase in the number of states in the Markov state space for search calculation. It is. As one of the solutions to such a problem, for example, a method of constructing a state space for performing search calculation using dynamic programming or reinforcement learning using a hierarchical structure is also considered ( Non-patent document 1, Non-patent document 2).

Rajbara Makar, Sridhar Mahadevan, Mohammad Ghavamzaden, Hierarchical Multi-Agent Reinforcement Learning, AGENTS’01, pp246-253, 2001. Bernhard Hengst, Discovering Hierarchy in Reinforcement Learning with HEXQ, 19th International Conference on Machine Learning, pp.243-250. Natsuki Miyata, Jun Ota, Tamio Arai, and Hajime Asama, Cooperative Transport by Multiple Mobile Robots in Unknown Static Environments Associated With Real-Time Task Assignment, IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION, VOL.18, NO.5, pp.769- 780, OCTOBER 2002.

  The method of Non-Patent Document 1 has a problem that a human must devise the structure of the hierarchical state space itself. The method of Non-Patent Document 2 is a method of configuring a hierarchy in order of change frequency with a variable whose value change frequency is low among the state variables in the Markov state space as the lowest layer. Although it is possible to automatically construct a hierarchical state space, it is difficult to automatically construct a hierarchical state space for complex problems, and it is difficult to apply it to practical problems. In addition, most of these conventional technologies are considered when a single robot is used, and no effective hierarchical model construction method that considers the use of multiple robots has been proposed. is there.

  In view of such a current situation, the present invention provides a robot cooperative transport planning technology that can reduce the calculation time and the storage capacity of a computer necessary for the plan calculation while considering the presence of a plurality of robots. Objective.

  The robot cooperative transfer planning technique according to the present invention is as follows. This is a robot cooperative transfer planning technique for obtaining an action plan of each robot for moving a moving object (hereinafter referred to as an article) from a start position to a target position in cooperation with a plurality of robots, and using the position of the article as a state variable. An article trajectory calculation process for obtaining a chain of positions (hereinafter referred to as an optimum path point sequence) that forms a movement path of an article from a start position to a target position, based on a Markov state transition model in which the movement direction of the article is an action variable; An ideal robot that can take each action included in the union of all of the plurality of robots with the position of the article having each position included in the optimum path point sequence as a domain and at least the action that each robot can take ( Based on the Markov state transition model, in which a pair of the position of the ideal robot) is a state variable, and an action that the ideal robot can take is a behavior variable. , The action taken by the ideal robot when there is an article at each position included in the optimum route point sequence (hereinafter referred to as the optimum behavior sequence), and one ideal robot in the optimum route point sequence moves the article A position (hereinafter referred to as a robot switching position) that divides the optimal path point sequence into sections that can be used, and an action sequence (hereinafter referred to as the robot) that divides the optimal behavior sequence and divides the ideal robot by each switching position For each section that can be taken according to the robot switching position, the position of the article having each position included in the section as a domain, and the ideal robot And a state variable is a set of a position of an ideal robot (hereinafter referred to as a second ideal robot) that is not in charge of conveying articles in the section, and the second ideal robot Since the second ideal robot moves to a position where the article can be moved in the section after the section based on the Markov state transition model using the possible behavior as a behavior variable, the second ideal in the section The movement path planning process for obtaining a sequence of actions to be taken by the robot (hereinafter referred to as the second section optimum action sequence) and the first section optimum action sequence and the second section so that there is no contradiction among the robots. And a scheduling process for assigning an optimal action sequence.

  In the scheduling process, for each section that can be taken according to the robot switching position, the position of the article with the start point position of the section as the domain of definition and the article can be pushed at each position included in the section, or the article can be moved to each position. Actions or articles in which each robot moves an article with a state variable as a set of the position of each robot whose position is the pressed position and information for each robot to identify the action in the section immediately preceding the section Based on the Markov state transition model, where the behavior variable is the movement to the position where the robot can be moved or the standby behavior, the behavior of each robot corresponding to each variable included in the state variable is determined, and the optimal path point sequence By assigning actions corresponding to each variable to each robot in order from the beginning, the first section optimum action sequence and the second section optimum action are assigned to each robot. The may be allocated.

  In the switching position determination process, the robot switching position may be rewritten at a position where the robot switching position is further divided into sections where one robot included in the plurality of robots can move the article.

  Further, in the movement route planning process, the robot included in the plurality of robots waiting at the same position in each section also obtains an action for avoiding a collision with an article moved in the section. You may do it.

  According to the present invention, as will be described in detail later, the Markov state space is hierarchically configured (the article trajectory calculation process is the first hierarchy, the switching position determination process is the second hierarchy, and the movement route planning process is the third hierarchy. ), The search of the action plan is performed from the layer with the low change frequency (the first layer), and the search calculation result in the layer with the low change frequency is used to limit the search range in the layer with the high change frequency. In the limited state space, a search is performed in a lower hierarchy (second hierarchy, third hierarchy) having a high change frequency. That is, the number of variables included in the space increases as the number of hierarchies increases, but according to the present invention, the definition area of each variable is narrowed. For this reason, it is possible to reduce the calculation time required for calculating the operation plan and the storage capacity of the computer.

The figure which illustrates the moving direction of a robot, and an article conveyance possible direction. The figure for demonstrating the example of the shape of the articles | goods conveyed, and the position of the robot used as a necessary condition for moving an article | item in a moving direction. The figure which illustrates the environment of the goods conveyance mission by a some robot. The figure which shows the function structural example of a conveyance planning apparatus. The figure which shows the function structural example of an article | item trajectory calculation part. The figure which shows the function structural example of a switching position determination part. The figure which shows the function structural example of a movement route calculation part.

[Problem settings]
The task of a plurality of robots cooperating to transport a moving object from the start position to the target position is, for example, an article (moving object) in a room partitioned by walls as illustrated in FIG. 3 from the start position to the target position. It is transported by cooperative action of multiple robots. The robots that perform the mission are p units (p ≧ 2), and each robot can push and move an article only in the Y-axis direction on the two-dimensional plane (in this example, the vertical direction of the drawing). 1 (see FIG. 1 (a)), or a robot (see FIG. 1 (b)) that can push and move the article only in the X-axis direction (in this example, the left-right direction on the drawing) in the two-dimensional plane, or the article Is one of the robots (see FIG. 1 (c)) that can be pushed and moved in the XY-axis direction (in this example, the four directions up, down, left and right in the drawing) in the two-dimensional plane. Each robot is movable in the XY-axis directions on the two-dimensional plane (in this example, the four directions up, down, left and right in the drawing).

  Further, from the viewpoint of specifically describing the present invention, it is assumed that the shape of the article in a plan view is a shape in which a plurality of square blocks are combined as shown in FIG. The position of the article is defined by the grid position (article position reference point) of any one of the blocks constituting the article. In the present embodiment, as shown in FIG. 2, the block at the lower left corner among the blocks constituting the article is used as the article position reference point.

  In such a problem setting, in order to properly move the article to the target position, cooperative operation between the robots is indispensable, and obstacles exist. You have to plan exactly whether you want to move it to a "possible position".

  Each robot will be stationary if there is an obstacle in the direction of movement or if there is an obstacle in the direction of pushing the article and the article is pushed in that direction. Assume. It is assumed that a plurality of robots can exist at the same position, and there is no collision between the robots.

  When the initial position of each robot i (i represents the robot number) is (Xri0, Yri0), the start position of the article is (Xb0, Yb0), and the target position of the article transport destination is (Xbe, Ybe), This problem can be defined as a behavior plan for a robot placed at an initial position to move an article from a start position to a target position.

  When a Markov state transition model is simply applied to such a problem, the Markov state space is the position of robot i (Xri, Yri), the action di of robot i, the article, where i is the robot number. (Xb, Yb). Each state (robot position and action, article position) is represented by discrete values. When a room is represented by a two-dimensional plane composed of an orthogonal coordinate system of X and Y, each position is represented by a discrete representation of the X axis and the Y axis. That is, as shown in FIG. 3, the room (two-dimensional plane) is divided by a grid, and each grid corresponds to each position. In each grid, “present / none” of the obstacle is set in advance.

The action subject is each robot arranged in the room. The action hi of the robot i (i is a robot number) takes one of a total of five types: stationary, movement of one lattice in the vertical and horizontal directions. For example, hi hi {0,1,2,3,4}
0: stationary
1: Move one grid to the right on the 2D plane
2: Move one grid upward on a two-dimensional plane
3: Move one grid to the left on the 2D plane
4: Suppose that one grid moves downward on a two-dimensional plane.

  The Markov state space in such a mission environment has a state of dimensions of the number of articles x 2 + the number of robots x 2, and the number of selectable actions exists by the number of robots that is the number of robot actions (= 5). . For example, if the number of articles is 1, the number of robots is 2, and the number of grids in the vertical and horizontal directions of the room is 10, the number of states will be 10 × 10 × 10 × 10 × 10 × 10 = 1000000, and search calculation The amount of resources required for this is enormous. Further, every time the number of robots increases, the number of states increases 100 times, which is a big problem when using multiple robots.

  In addition, when a technique such as HEXQ in Non-Patent Document 2 is applied in such a problem, the variable (Xb, Yb) of the article position whose value change frequency is low constitutes the state space of the highest hierarchy. The variable (Xri, Yri) of the position of the robot with a high change frequency is hierarchized as a variable constituting the state space of the lower hierarchy. However, the range in which the robot can operate changes every moment as the article position changes. In such a case, according to HEXQ, the search is performed from the lowermost layer. However, when the robot motion plan is performed in the lowermost layer, the search is performed without considering the value of the article position variable that changes every moment. It is clear that only inappropriate results can be obtained. As described above, it is a general search problem that a variable in an upper hierarchy must be taken into consideration at the lowest layer or vice versa. A certain variable can be included in the highest hierarchy or in the lowest hierarchy, and it is not easy to simply assign a variable to each hierarchy.

[Handling of behavior in the present invention]
Originally, movement of an article is realized by performing a “pushing operation” after a robot located at a position away from the article moves to an appropriate position where the article can be pushed. In order to consider it perfectly, consider the relative positional relationship between the position of the article (Xb, Yb) and the surrounding robot i position (Xri, Yri), including the presence of obstacles. There must be. However, considering all of this from the beginning is not appropriate for the efficiency of search computation. Therefore, in the present invention, a method is adopted in which the range of action around the robot article is expanded and considered hierarchically. Hereinafter, a robot whose action range around the article is limited is also referred to as a virtual robot, and search calculation is performed on the assumption that the virtual robot presses the article at each level.

  First, in the first hierarchy (article trajectory calculation process), an action subject is an article, and the article is handled as if it were moving by itself. In other words, in the first hierarchy, only the position where the virtual robot can push the article (position where the virtual robot touches the article and pushes the article in an appropriate direction) among the positional relationship between the article and the virtual robot is considered. Then, an optimum movement route (optimum route point sequence) for the article to move from the start position to the target position is obtained.

  Next, in the second hierarchy (switching position determination process), after the article is pushed and moved by the virtual robot v, it is considered whether or not the virtual robot v can continue to move to a position where the next article can be moved. In other words, a case where the virtual robot v is in contact with the article but is not in a place where the article can be pushed in a specified direction is considered, and the action that the virtual robot v can take is “until the virtual robot reaches a position where the article can be pushed. Since the action “movement” is included, the action range around the article of the virtual robot v is expanded more than the highest hierarchy. Then, in the sequence of the optimal movement route of the article obtained in the first hierarchy, the action of the virtual robot v in the section in which one virtual robot v can continuously move the article and the other robot are not taken over. Find the position (switching position) that no longer gets.

  Further, in the third hierarchy (movement route planning process), after the article is moved several times by the virtual robot v, another virtual robot w can realize the next movement of the article at the destination. Consider whether the virtual robot w can move to a certain position. That is, a state in which the virtual robot w is located away from the article is considered. Then, when the article reaches the end point of the section in the section in which one robot can continuously move in the sequence of the optimal movement route of the article obtained in the first hierarchy (movement is a virtual robot v The action (movement route) of the virtual robot w for reaching the position where the virtual robot w can move the article to the start position of the next section is obtained. In addition, while the virtual robot v is pushing the article, the virtual robot w may have to wait at a predetermined position. In this case, the virtual robot w does not interfere with the movement of the article. An action for appropriately avoiding a collision with the moving article is also obtained in the third hierarchy.

  Thus, the positional distance between the virtual robot (v or w) and the article that the virtual robot (v or w) is about to move as the processing proceeds from the first hierarchy, the second hierarchy, and the third hierarchy. Will spread. However, the search in the second hierarchy is performed on the premise that the article moves only along the trajectory obtained in the first hierarchy, and the search in the third hierarchy is performed in the single hierarchy obtained in the second hierarchy. Since a search is performed in a section in which an article is continuously moved by one robot, the range of article positions that must be considered in the search is gradually narrowed. Then, in the second hierarchy, the behavior of the robot (virtual robot v) pushing the article is determined. In the third hierarchy, the robot (virtual robot w) taking over the virtual robot v and taking charge of the article movement is in a position where the article can be pushed. Determine actions to move. Finally, it is determined by scheduling processing which robot in the real environment performs in which order the behavior obtained as the virtual robot v and the virtual robot w.

[Outline of Embodiment of the Present Invention]
The configuration of the robot cooperative transfer planning apparatus A of this embodiment is shown in FIG. The robot cooperative transfer planning apparatus A includes an article trajectory calculation unit 100, a switching position determination unit 200, a movement route planning unit 300, and a scheduling unit 400.

  In the article trajectory calculation unit 100 (first layer), the switching position determination unit 200 (second layer), and the movement route planning unit (third layer), variables constituting the Markov state space used for the search are as follows. .

Article trajectory calculation unit 100 (first layer):
State variable s1 = (Xb, Yb), action variable a1∈ {0,1,2,3,4}

Switching position determination unit 200 (second layer):
State variable s2 = (Xb, Yb, Xr, Yr), action variable a2∈ {0,1,2,3,4}

Travel route planning department (third layer):
State variable s3 = (Xb, Yb, Xr, Yr), action variable a3∈ {0,1,2,3,4,5}

  The article trajectory calculation unit 100 considers only the state variable s1 representing the article position and the behavior variable a1 representing the movement direction of the article. That is, the action subject here is an article, and the article trajectory calculation unit 100 performs the calculation as if the article moves by itself, and does not consider the position of the robot pushing the article. It is assumed that the behavior variable representing the moving direction of the article can take any one of five types, that is, stationary and movement of one lattice in the vertical and horizontal directions. The article trajectory calculation unit 100 uses these variables to calculate the chain of positions of the article movement from the start position to the target position (without considering the position of the robot and the direction in which the robot can push the article), and the optimum path point sequence. (Ideal trajectory) s1_trj = (s1_trj [0], s1_trj [1], ..., s1_trj [e]).

  The switching position determination unit 200 considers a state variable s2 composed of an article position (Xb, Yb) and a virtual robot position (Xr, Yr), and an action variable a2 representing the direction in which the virtual robot pushes the article. The action subject here is one virtual robot v (a robot that pushes and moves an article).

  In the switching position determination unit 200, all the positions included in the optimum path point sequence s1_trj obtained by the article trajectory calculation unit 100 are within the range that can be taken by the article position coordinates (Xb, Yb) included in the state variable s2, and other than that. The article position is not considered. In addition, the direction in which the virtual robot v can push the article (the value that the action variable a2 can take) is a union of the directions in which each robot in the real environment can push the article.

  The switching position determination unit 200 calculates the maximum range in which the virtual robot v can continuously move in each position of the optimum route point sequence s1_trj. For example, when a robot that has moved an article from s1_trj [5] to s1_trj [6] tries to move it to a position for moving the article to s1_trj [7], it moves with an obstacle in the movement direction. If this is not possible, it is necessary to take over the action to another robot in another position. The switching position determination unit 200 determines a place (switching point) where the responsible robot needs to be switched in this way, among the positions of the optimum route point sequence s1_trj.

  The movement path planning unit 300 considers a state variable s3 composed of the article position (Xb, Yb) and the virtual robot position (Xr, Yr), and an action variable a3 representing the movement direction of the virtual robot. The action subject here is a virtual robot w that moves to a position where the article can be pushed (a robot that does not push the article).

  The movement path planning unit 300 uses the position coordinates within the individual ranges delimited by the switching points determined by the switching position determination unit 200 among the positions in s1_trj as the article position (Xb, Yb) included in the state variable s3. ), And other article positions are not considered. Further, the direction in which the virtual robot can move (values that the behavior variable a3 can take) is a3ε {0, 1, 2, 3, 4, 5}. Here, when a3 = 5, the virtual robot w stops and the virtual robot v pushing the article pushes the article only once (that is, the virtual robot w does not disturb the movement (action) of the robot v pushing the article or the article) It means to be stationary.

  When the movement path planning unit 300 finishes moving the article from the start position to the end position (switching point) of the section handled by a certain robot (virtual robot v), the virtual robot w reaches the position where the article can be continuously pressed. As shown, the action sequence for moving the virtual robot w is determined.

  Assuming that the article trajectory calculation process is the upper hierarchy (first hierarchy), and the switching position determination process and the movement route planning process are the second hierarchy and the third hierarchy in this order, the present invention is the upper hierarchy (first hierarchy). The above-mentioned non-patent point is that the variable to be used is a variable with a low change frequency (article position variable), and the variable to be used at the lower hierarchy (third hierarchy) is a variable with a high change frequency (robot position variable). It can be said that it is almost the same as the concept of Reference 2. However, the difference is that the search calculation is performed not from the layer with high change frequency (third layer) as in Non-Patent Document 2, but from the layer with low change frequency (first layer). Then, by using the search calculation result in the hierarchy with low change frequency, the search range in the hierarchy with high change frequency is limited, and the variable in the lower hierarchy with high change frequency is limited in a limited space (narrow space). Perform a search. In other words, as the hierarchy increases, the number of variables included in the space increases, but on the other hand, the definition area of each variable is narrowed.

[Specific Example of Embodiment]
Hereinafter, specific processing of search calculation in each unit will be described.

[Article Trajectory Calculation Unit; Step S1]
The article trajectory calculation unit 100 expresses a value function V ^π (s1) representing how much article position (state) s1 = (Xb, Yb) is likely to reach the target position by the following Markov state transition model. The optimum route point sequence indicating the shortest route for moving the article from the start position to the target position is output by using the dynamic programming method.

Here, the set S1 = {s1 ₀ of "a set of discrete states that could be taken of the environment" the article position _{s1 i = (Xb i, Yb} i) in Markov state transition _{model, s1 1, ..., s1 n} -1 }. For example, the target room is divided into M sections in the X-axis direction, and the index of each discrete section is set to 0, 1,. Similarly, it is divided into N sections in the Y-axis direction, and the indices of the discrete sections are represented by 0, 1,..., N−1. The position of each lattice is represented by an index (Xb _i , Yb _j ) of the X axis and the Y axis (0 ≦ Xb _i ≦ M−1, 0 ≦ Yb _j ≦ N−1).

Also, the action a1 that the action subject (article) can take is a1∈ {0,1,2,3,4},
0: stationary
1: The article moves to the right on the two-dimensional plane.
2: The article moves upward on a two-dimensional plane
3: The article moves to the left on the 2D plane
4: An article moves downward on a two-dimensional plane. A set of values A1 that a1 can take is A1 = {0, 1, 2, 3, 4}.

  In the Markov state space of the article trajectory calculation unit 100, which robot in the real environment (which robot 1 or robot 2) pushes the article is not considered. Also, the process of moving the robot to a position where the article can be pushed is not considered. However, if there is an obstacle around the position necessary for the robot to push the article, and there is not enough space for the robot pushing the article, the robot cannot move the article. Only that point is considered. Such a case is, for example, a case where the article is moved in the right direction, and the position where the article is in contact with the left side is occupied by an obstacle.

When the action a1εA1 is executed in the state s1, the state (article position) is assumed to transition to the position s1′εS1 in a probabilistic manner. The state transition probability P1
P1 (s1 '| s1, a1) = Pr (s1 _{t + 1} = s1' | s1 _t = s1, a1 _t = a1)
Is represented by t represents time. Then, the state transition probability P (s1 ′ | s1, a1) for each s1, a1, s1 ′ is set as follows:

However, <condition A> is a case where any one of the following conditions (1) to (5) is satisfied.
(1) a1 = 1 & Xb '= Xb + 1 &Yb' = Yb & Judge (s1, a1) = YES
(2) a1 = 2 & Xb '= Xb &Yb' = Yb + 1 & Judge (s1, a1) = YES
(3) a1 = 3 & Xb '= Xb-1 &Yb' = Yb & Judge (s1, a1) = YES
(4) a1 = 4 & Xb '= Xb &Yb' = Yb-1 & Judge (s1, a1) = YES
(5) a1 = 0 & Xb '= Xb &Yb' = Yb & Judge (s1, a1) = YES
Here, “&” indicates an AND condition.

Judge (s1, a1) is given by the following equation.

  Judge (s1, a1) is a function that returns YES if the article at position s1 can move in the direction according to action a1, and returns NO if it cannot move. Actually, in order for the article to move, the robot must push the article in that direction, so there is a space where the robot can enter the position where the article can be pushed (no obstacles). It becomes a necessary condition. Judge (s1, a1) is a function that checks whether or not this requirement is satisfied. As shown in FIG. 2, Ready_Pos_k (s1, a1) represents a set of k candidates for the position of the robot in contact with the article for moving the article at the article position s1 in the direction indicated by the action a1. Among these, when there is a robot position that does not overlap the obstacle (there is an element of Ready_Pos_k (s1, a1) that does not match the obstacle position), the robot moves the article at position s1 in the direction indicated by action a1. be able to. Otherwise it cannot be moved.

  In order to calculate the candidate position of Ready_Pos_k (s1, a1), for example, at the article position reference point ((Xb, Yb) = (0, 0)), to move the article in the direction indicated by the action a1. Pos_1 (a1), Pos_2 (a1),…, Pos_k (a1) are stored in advance, and each position Pos_j (a1) (j = 1,2,…, k) and the position of s1 It can be calculated by adding the coordinates for each element. If Pos_j (a1) = (XPos_j (a1), YPos_j (a1)), s1 = (Xb, Yb), (XPos_j (a1) + Xb, YPos_j (a1) + Yb) is calculated for all j. That's fine.

  The first three conditions in (1) to (5) of <Condition A> are based on whether or not the combination of the current article position s1 and the article movement direction a1 matches the article position s1 ′ after the state transition. This is a condition for determining whether or not. For example, in the case of (1) of <Condition A>, when a1 = 1 (the article moves in the right direction) and s1 = (Xb, Yb), the article position after the state transition is s1 ′ = (Xb + 1, Yb). The article cannot move to any other position.

  The above formula (1) indicates that the combination of the current article position s1 and the article movement direction a1 is aligned with the article position s1 ′ after the state transition, and the robot is at a position where the article can be pushed. When the space to enter is vacant (there is no obstacle), the state transition probability is set to 1. Moreover, since it cannot change to the state which does not satisfy | fill this condition, it means making a state transition probability 0.

Further, when the action a1εA1 is executed in the state s1 and the state (article position) transits to s1′εS1, a reward R1 given from the environment to the action subject is set by the following equation.

The article trajectory calculation unit 100 calculates the optimum value function V ^π1 (s1) and policy π1 by the dynamic programming method using the Markov state space (for example, Sadayoshi Mikami, Masaaki Minagawa) , RSSutton, AGBarto Original work “Reinforcement Learning” Morikita Publishing, 1998, pp.94-118).

  A detailed configuration of the article trajectory calculation unit 100 is shown in FIG. The article trajectory calculation unit 100 includes an article trajectory calculation state transition model setting unit 101, an article trajectory calculation value function initialization unit 102, an article trajectory calculation policy update unit 103, an article trajectory calculation value function update unit 104, and an article trajectory. A calculation value function convergence determination unit 105 and an optimum path point sequence calculation unit 106 are included.

  The state transition model setting unit 101 for article trajectory calculation uses the state transition probability of the above equation (1) for all combinations of each article position s1, the movement direction a1 of the article, and the article position s1 ′ after the state transition. The reward of the above formula (2) is set.

The article trajectory calculation value function initialization unit 102 sets an initial value of the value function V ^π1 (s1) for each s1εS1. An appropriate value may be set as the initial value, but in this embodiment, the initial value is set to V ^π1 (s1) = 0 for all s1.

The article trajectory calculation policy updating unit 103 calculates Expression (3) for all combinations of s1∈S1 and a1∈A1, and determines the action a1 having the maximum value of V ^π1 (s1) as the policy π1 (s1). Set to value.

Here, Σ represents the sum of all possible s1 ′, and γ is a constant that satisfies a predetermined 0 <γ <1.0. The value of V ^π1 (s1) on the right side of the equation (3) is the initial value set by the article trajectory calculation value function initialization unit 102 when the article trajectory calculation policy update unit 103 is executed for the first time. In the second and subsequent executions, the value of the value function updated by the following item trajectory calculation value function updating unit 104 is used.

The article trajectory calculation value function updating unit 104 updates the value function V ^π (s1) with respect to each s1εS1 according to the equation (4).

Article trajectory calculation value function convergence determining unit 105, and the following error is a predetermined threshold value and updating the previous V of V ^.pi.1 updated in article trajectory calculation value function update unit 104 (s1) ^π1 (s1) Until this time, the process returns to the article trajectory calculation policy updating unit 103, and the policy and value function update processing is repeated. When the value is equal to or less than the predetermined threshold, it is determined that the value function V ^π1 (s1) has converged, and the value function V ^π1 (s1) and the policy π1 (s1) are output to the optimum path point sequence calculation unit 106.

The value function V ^π1 (s1) thus obtained is an index representing how easily the article position s1 can reach the target position. As a result, a series of article positions indicating the shortest path of article movement from an arbitrary position in the room to the target position can be obtained.

The optimum path point sequence calculation unit 106 uses the value function V ^π1 (s1) and the policy π1 (s1) output from the article function calculation value function convergence determination unit 105 to determine the shortest path from the start position to the target position. A series of article positions (optimum route point sequence) is calculated. For example, when the start position is s1_trj [0] ∈S1, the article indicated by the value of π1 (s1_trj [0]) is moved and moved to the position s1_trj [1] ∈S1. Subsequently, in s1_trj [1], the article indicated by the value of π1 (s1_trj [1]) is moved to move to position s1_trj [2]. By repeating this until the target position s1_trj [e] ∈S1 is reached, a point sequence of the midpoints included in the shortest path of the article from the start position to the target position (optimum path point sequence): s1_trj = (s1_trj [0 ], s1_trj [1], ..., s1_trj [e]) (s1_trj [n] = (Xb_trj [n], Yb_trj [n]) (n = 0, 1, ..., e)). In addition, the point sequence a1_trj = (a1_trj [0], a1_trj [1], ..., a1_trj [e]) = (π1 (a1_trj [0]), π1 (a1_trj [1]) indicating the moving direction of the article at this time ,..., Π1 (a1_trj [e]), and the optimum route point sequence calculation unit 106 outputs the optimum route point sequence s1_trj and the movement direction sequence a1_trj.

[Switching position determination unit; step S2]
The switching position determination unit 200 can be continuously pushed by one robot in each position of the optimum route point sequence s1_trj = (s1_trj [0], s1_trj [1],..., S1_trj [e]). Which section the position is in is calculated by dynamic programming using a Markov state transition model.

  For example, if one robot can continuously move an article through each position from s1_trj [0] to s1_trj [4], s1_trj [0], s1_trj [1], s1_trj [ 2), s1_trj [3], and s1_trj [4] are better served by one robot. Conversely, for example, a robot that has moved an article from s1_trj [3] to s1_trj [4] is prevented by an obstacle from moving to a position where the article can be pushed to the next s1_trj [5] to move the article. If it is, it is better to take over the movement of the article to another robot in another position. That is, the switching position determination unit 200 switches positions of the robot in charge during the movement at each position of the optimum route point sequence s1_trj = (s1_trj [0], s1_trj [1],..., S1_trj [e]) The position where the robot must take over the movement) is calculated.

  Here, the “set of discrete states that the environment can take” in the Markov state transition model is defined as the set of each article position (Xb, Yb) and robot position (Xrv, Yrv) included in the optimal path point sequence s1_trj. A set of possible values. That is, when s2 = (Xb, Yb, Xrv, Yrv), (Xb, Yb) ∈ s1_trj. Article position values other than s1_trj are not considered in the search. Further, only one virtual robot that considers a set of actions of all robots in the real environment is considered as the action subject (robot).

For the above problem setting,
Since the robot 1... Pushes the article in the Y-axis (up and down) direction, or the stationary robot 2... Pushes the article in the X-axis (left and right) direction, or can take a stationary action. Assume that the subject (robot) can push the article in all four directions, up, down, left, and right. The action a2 that the action subject (robot) can take is a2∈ {0,1,2,3,4},
0: stationary
1: Robot moves to the right on the 2D plane
2: Robot moves upward on 2D plane
3: Robot moves to the left on the 2D plane
4: The robot moves downward on a two-dimensional plane. A set A2 of values that a2 can take is set to A2 = {0,1,2,3,4}.

When the action a2∈A2 is executed in the state s2∈S2, the state transition probability to transit to the state s2′∈S2 is
P2 (s2 '| s2, a2) = Pr (s2 _{t + 1} = s2' | s2 _t = s2, a2 _t = a2)
And the state transition probability P2 (s2 ′ | s2, a2) for each s2, a2, s2 ′ is set as follows:

However, <Condition B> is when the following conditions (1) to (6) are satisfied.
(1) Xb = Xb '& Yb = Yb'& a2 = 1 & Xrv '= Xrv + 1 &Yrv' = Yrv
(2) Xb = Xb '& Yb = Yb'& a2 = 2 & Xrv '= Xrv &Yrv' = Yrv + 1
(3) Xb = Xb '& Yb = Yb'& a2 = 3 & Xrv '= Xrv-1 &Yrv' = Yrv
(4) Xb = Xb '& Yb = Yb'& a2 = 4 & Xrv '= Xrv &Yrv' = Yrv-1
(5) Xb = Xb '& Yb = Yb'& a2 = 0 & Xrv '= Xrv &Yrv' = Yrv
(6) (Xb, Yb) = s1_trj [l + 1] & (Xb ', Yb') = s1_trj [l + 2] & a2 = a1_trj [l + 1] & (Xrv, Yrv) = Ready_Pos_k (s1_trj [ l + 1], a1_trj [l + 1]) && (Xrv ', Yrv') = position after moving by a1_trj [l + 1]

Further, when the action a2∈A2 is executed in the state s2∈S2 and the state (article position) transits to s2′∈S2, the reward R2 given to the action subject from the environment is set by the following equation.

Using the above defined P2 (s2 '| s2, a2) and R2 (s2', s2, a2) to find policy π2 (s2) by dynamic programming, π2 (s2) is , When the article is at position s1_trj [l + 1], a set of positions that can take action a1_trj [l + 1] to move the article from position s1_trj [l + 1] to position s1_trj [l + 2] ( Ready_Pos_k (s1_trj [l + 1], a1_trj [l + 1])) is an action policy (a policy for movement) of the virtual robot v for the virtual robot to reach at least one place. That is, when the value of the value function value V ^π2 (s2) in s2 = (Xb_trj [l + 1], Yb_trj [l + 1], Xrv, Yrv) is 0 or more, the position (Xrv, Yrv) indicated by the state s2 The virtual robot v in the position s1_trj _{l + 1} = (Xb_trj [l + 1], Yb_trj [l + 1]) indicated by the state s2 can push the article at the next position (Ready_Pos_k ( s1_trj [l + 1] and a1_trj [l + 1])). Further, the movement at that time may follow the value of the behavior variable indicated by π2 (s2).

Here, it arrives when the robot at the position Ready_Pos_k (s1_trj [l], a1_trj [l]) pushes the article at the position s1_trj [l] in the direction indicated by the action a1_trj [l] and moves itself There are a maximum of K positions, and the position that does not overlap the obstacle is Ready_Pos_k (s1_trj [l + 1], a1_trj [l]) = (Xrl [k], Yrl [k]) (k = 0,1, ..., K-1). Then, at the article position at the position s1_trj [l + 1], K points s2 [k] = (Xb_trj [l + 1], Yb_trj [l + 1], Xrl [k], Yrl [k]) (however, , K = 0,1,…, K-1, (Xrl [k], Yrl [k]) ∈Ready_Pos_k (s1_trj [l + 1], a1_trj [l]), value function in s2 [k] ∈S2) If all of the values V ^π2 (s2 [k]) = V ^π2 (Xb_trj [l + 1], Yb_trj [l + 1], Xrvl [k], Yrvl [k]) are greater than 0, the same robot Can continue to move the item to s1_trj [l + 2], otherwise the robot that moved the item to s1_trj [l + 1] will move the item further due to obstruction Since it is not possible, it turns out that it is necessary to switch to another robot.

  A detailed configuration of the switching position determination unit 200 is shown in FIG. The switching position determination unit 200 includes a switching position determination state transition model setting unit 201, a switching position determination value function initialization unit 202, a switching position determination policy update unit 203, a switching position determination value function update unit 204, and a switching position. A decision value function convergence determination unit 205, a switching position determination unit 206, and a policy division unit 207 are included.

  The switching position determination state transition model setting unit 201 sets the state transition probability of Expression (5) and the reward of Expression (6).

  Each process of the switching position determination value function initialization unit 202, the switching position determination policy update unit 203, the switching position determination value function update unit 204, and the switching position determination value function convergence determination unit 205 includes a state transition probability and a reward. Is the same as the processing described in the article trajectory calculation unit 100, except that is replaced by Equation (5) and Equation (6), respectively.

The switching position determination value function initialization unit 202 sets the initial value of the value function V ^π2 (s2) for each s2εS2 to zero.

The switching position determination policy updating unit 203 calculates the following expression for all combinations of s2∈S2 and a2∈A2, and sets the action a2 having the maximum value of V ^π2 (s2) to the value of the policy π2 (s2). Set.

Here, Σ represents the sum of all possible s2 ′, and γ is a predetermined constant of 0 <γ <1.0. In addition, as the value of V ^π2 (s2 ′) in Expression (7), the initial value set by the switching position determination value function initialization unit 202 is used when the switching position determination policy update unit 203 is first executed. In the second and subsequent executions, the value of the value function updated by the following switching position determination value function updating unit 204 is used.

The switching position determining value function updating unit 204 updates the value function V ^π2 (s2) according to the following equation for each s2εS2.

Value for switching the position determination function convergence determining unit 205, the error value and updates the previous ^V π2 (s2) of the V ^.pi.2 updated in the switching position determination value function update unit 204 (s2) and a predetermined threshold value or less Until that time, the process returns to the policy position updating unit 203 for determining the switching position, and the policy and value function update processing is repeated. When the value is equal to or less than the predetermined threshold, it is determined that the value function V ^π2 (s2) has converged, and the switching position determination value function convergence determination unit 205 determines the value function V ^π2 (s2) and the policy π2 (s2). Is output to the switching position determination unit 206.

The switching position determination unit 206 calculates the following equation for each l = 0, 1,..., E using the value function.

  Here, l is an index of the optimum route point sequence s1_trj. When the value of Partition [l] is 0, it indicates that the assigned robot must be changed after the article in s1_trj [l] is moved to s1_trj [l + 1].

  In the processing so far, the calculation is performed assuming a virtual robot that can push an article in all directions. However, the direction in which the robot in the real environment can move the article may be restricted (for example, in the setting of the present embodiment, each robot is restricted to the movement direction of the article shown in FIG. 1). ). When the direction in which the robot can move the article in the real environment is restricted, the following processing is further performed after obtaining the above Partition [l].

First, l = 0 is set, and one robot in an actual environment that can push the article is selected in the movement direction of the article indicated by the value of Partition [0]. Here, it is assumed that robot i is selected. Next, the value of l is incremented by 1 (l ← l + 1), and the movement direction of the article pointed to by Partition [l] is a set of article movement directions that can be performed by the robot i selected in Partition [0]. Since the robot i can continue to move the article only when it is included in the list, the value of Partition [l] is not changed. If not, the robot needs to be switched, so update Partition [l] = 0. By repeating this in order for l = 2, 3,..., E, the value of Partition [l] is updated. The value of Partition [l] obtained in this way indicates a section that can be continuously pressed by one robot. That is, a section divided by Partition [l] = 0 is a section that can be continuously pressed by one robot. In the following, it is assumed that the index value of the start point of each section is p ₀ , p ₁ , p _{2 ,.} Note that the start point of the section is a value of l at which l = 0 and Partition [l] = 0.

For example,
Partition [0] = 1,
Partition [1] = 1,
Partition [2] = 0,
Partition [3] = 2,
Partition [4] = 2,
Partition [5] = 3,
Partition [6] = 0,
Partition [7] = 4,
Partition [8] = 5,…
In this case, the end points of the section are p ₀ = 0, p ₁ = 2, and p ₂ = 6.

The policy division unit 207 divides the policy π2 (s2) obtained as described above for each section obtained by the switching position determination unit 206. Measures dividing unit 207 can take p _m, p _n combinations _{(p m, p n ∈ {} p 0, p 1, p 2, ..., p f}, p m <p n) for each, measures .pi.2 ( s2) is divided into π2 [p _m ] [p _n ] (s2). π2 [p _m ] [p _n ] (s2) corresponds to the optimal path point sequence s1_trj in the interval [p _m , p _n ] of the policy π2 (s2), and the robot in charge of moving the article is , Represents an action policy for continuously moving the article from position s1_trj [p _m +1] to s1_trj [p _n +1]. In the switching position determination unit 206, since the movement of the virtual robot is treated as the same action as the real robot, the calculated policy π2 [p _m ] [p _n ] (s2) is moved as it is in the real environment. It becomes the action policy of the robot in charge.

[Movement route planning unit; Step S3]
In the movement path planning unit 300, the search area is limited to the article position in each section calculated in the second hierarchy and the robot position in the entire room. That is, the action policy for guiding the next assigned robot when the assigned robot is switched is calculated by a dynamic programming method using a Markov state transition model. The robots other than the robot in charge for switching are treated as performing continuous article movement performed simultaneously or waiting while avoiding a collision with the article.

  The movement path planning unit 300 calculates, for each section divided by an index (value of l) in which the value of Partition [l] calculated by the switching position determination unit 206 is 0 in the series of the ideal trajectory s1_trj of the article. The robot in charge of this section calculates an action policy for moving to a position where the article at the initial position of the previous section can be pushed. In addition, after finishing the action of pushing the article, the robot z waiting at the same position as it is also calculates an action policy for avoiding a collision with the article pushed by the robot v.

First, calculation of an action policy for a robot in charge of a later section in time to move to a position where an article at an initial position in the later section can be pushed will be described. Here, while the other robot v is moving the article in the section [p _m , p _{m + 1} ], the virtual robot w is a section later in time than [p _m , p _{m + 1} ]. A sequence of actions (action policy π3 [p _m ] [p] that the virtual robot w should take in the section [p _s , p _n ] in order to move to a position where the article can be pushed in the section (starting from p _n ). _s ] [p _n ] (s3)). In other words, while the virtual robot v is moving the article from s1_trj [p _m +1] to s1_trj [p _{m + 1} +1], it waits after pushing and moving the article to s1_trj [p _s +1]. virtual robot w that is, to move the article in the s1_trj [p _n +1] s1_trj to [p _n +2] that can be moved to a position, a sequence of actions to be taken by the virtual robot w calculate. Here, p _m , p _s , p _n ∈ {p ₀ , p ₁ , p ₂ ,..., P _f }, p _s ≦ p _m, p _s <p _n , p _{m + 1} ≦ p _n .

For example, if p _m = 2, p _s = 1, and _pn = 5, π3 [2] [1] [5] (s3) indicates that the virtual robot v moves the article in the interval [2,3]. In the meantime, the virtual robot w that has been waiting after moving the article to the section [0, 1] is an action policy for moving to the start position where the article can be moved in the section [5, 6].

Note that in this setting, one section [p _m, p _{m + 1]} moving section starting at p _m of the virtual robot v in charge of an article moved by it to, the number of states in the scheduling layer to be described later There is also an effect that scheduling can be reduced and scheduling can be performed efficiently.

Let s3 = (Xb, Yb, Xrw, Yrw) be a set of discrete states that the environment can take in the Markov state transition model, and the action a3∈ {0,1,2, which the action subject (robot w) can take 3,4,5} (Markov chain). The behavior variable a3 = 0, 1, 2, 3, and 4 represents the stationary motion of the robot and the movement in the up / down / left / right directions, like the behavior variable a2. a3 = 5 indicates “standby”. The difference between “standby” and “stationary” is that “standby” takes into account the behavior of other robots moving the article. That is, a3 = 5 means that another robot is moving the article, and thus waits until the movement of the article is completed for one step. In addition, the article position (Xb, Yb) included in the state variable s3 is
(Xb, Yb) ∈ {s1_trj [p _m +1], s1_trj [p _m +2]…, s1_trj [p _{m + 1} +1]}
It is.

Further, the position (Xrw, Yrw) of the robot w included in the state variable s3 is defined as the entire room. Then, when the action a3∈A3 is selected in the state variable s3 = (s1_trj [l], Xrw, Yrw) ∈S3, the state transitions to the state s3 ′ = (s1_trj [l] ', Xrw', Yrw ') ∈S3 The state transition probability P3 [p _m ] [p _s ] [p _n ] to be set is set as follows.

However, <Condition C> is a case where any of the following is satisfied.
(1) a3 = 1 & Xrw '= Xrw + 1 &Yrw' = Yrw & s1_trj [l] '= s1_trj [l]
(2) a3 = 2 & Xrw '= Xrw &Yrw' = Yrw + 1 & s1_trj [l] '= s1_trj [l]
(3) a3 = 3 & Xrw '= Xrw-1 &Yrw' = Yrw & s1_trj [l] '= s1_trj [l]
(4) a3 = 4 & Xrw '= Xrw &Yrw' = Yrw-1 & s1_trj [l] '= s1_trj [l]
(5) a3 = 0 & Xrw '= Xrw &Yrw' = Yrw & s1_trj [l] '= s1_trj [l]
(6) a3 = 5 & Xrw '= Xrw &Yrw' = Yrw & s1_trj [l] '= s1_trj [l + 1]

Also, the reward R3 at this time is set as follows.

P3 [p _m ] [p _s ] [p _n ] (s3 ′ | s3, a3) and R3 [p _m ] [p _s ] [p _n ] (s3 ′, s3, a3) defined as above If we find the policy π3 [p _m ] [p _s ] [p _n ] (s3) using dynamic programming, π3 [p _m ] [p _s ] [p _n ] (s3) While the robot moves the article from s1_trj [p _m +1] to s1_trj [p _{m + 1} +1], the article at position s1_trj [p _n +1] is moved to a1_trj [p _n +1] Action policy (virtual robot w) for the virtual robot w to reach a position (Ready_Pos_k (s1_trj [p _n +1], a1_trj [p _n +1]) that can be pushed in the direction shown) Measures for movement). That is, while the other robot continuously moves the article, the action subject (virtual robot w) is moved to any position of Ready_Pos_k (s1_trj [p _n +1], a1_trj [p _n +1]) while moving the article continuously. I will do some operation plan. In fact, other robots are only evaluated here as moving with the article. The movement path planning unit 300 plans the movement of the robot considering the cooperation with other robots for the first time, but the operation of the other robots is determined by the switching position determination unit 200, and the other robots (moving articles) Therefore, the movement path planning unit 300 does not need to consider state variables describing the states of other robots. That is, the coordinates (Xrw, Yrw) of the robot included in the state variable s3 handle the coordinates of the robot that has not moved the article and moves to the next switching position.

The above processing can be performed for each possible p _m , p _s , p _n ∈ {p ₀ , p ₁ , p ₂ ,..., P _f }, p _s ≦ p _m , p _s <p _n , p _{m + 1} < performed for p _n, measures _{π3 [p m] [p s} ] [p n] Request (s3).

Next, after finishing the action of pushing the object to s1_trj [p _s +1], the robot z waiting at the same position is moved by the robot v and moves in the section [p _m , p _{m + 1} ]. The calculation of an action policy (π3 [p _m ] [p _s ] [p _s ] ( _{s 3} )) for avoiding a collision with the article being operated will be described. When the article continuously moves from the position s1_trj [p _m +1] to the position s1_trj [p _{m + 1} +1]), the robot z waits at the position after pushing the article to the position s1_trj [p _s +1]. The robot z may interfere with the movement of the article. At that time, it is necessary for the robot z to temporarily move to a position that does not hinder the movement of the article. The policy π3 [p _m ] [p _s ] [p _s ] ( _{s 3} ) represents the action (collision avoidance action) of the robot z for that purpose. The action strategy for collision avoidance behavior is in the form of π3 [p _m ] [p _s ] [p _s ] (s3), because the destination of robot z is the same as the starting point (p _s ). It can be described as a measure. However, the target position in the case of collision _{avoidance, Ready_Pos_k (s1_trj [p s +1} ], a1_trj [p s]) is any position.

The state transition probability uses Equation (9) as it is (where P3 [p _m ] [p _s ] [p _s ]), and the reward is the robot's target position in the reward setting of Equation (10) Ready_Pos_k Change to (s1_trj [p _s +1], a1_trj [p _s ]), that is, using the following equation, π3 [p _m ] [p _s ] [p _s ] ( Find s3).

The above processing is performed for each possible p _m , p _s ∈ {p ₀ , p ₁ , p ₂ ,…, p _f }, p _s ≦ p _m , and policy π3 [p _m ] [p _s ] [p _s ] (s3) is obtained.

  A detailed configuration of the movement route planning unit 300 is shown in FIG. The movement route planning unit 300 includes a state transition model setting unit 301 for the next assigned robot movement plan, a value function initializing unit 302 for the next assigned robot movement plan, a policy updating unit 303 for the next assigned robot movement plan, and a next assigned robot movement plan. A value function update unit 304, a value function convergence determination unit 305 for the next assigned robot movement plan, a collision avoidance state transition model setting unit 306, a collision avoidance value function initialization unit 307, a collision avoidance movement plan policy update unit 308, A collision avoidance movement planning value function update unit 309 and a collision avoidance movement plan value function convergence determination unit 310 are included.

  The state transition model setting unit 301 for the next assigned robot movement plan sets the state transition probability of the above equation (9) and the reward of the equation (10).

  The next assigned robot movement plan value function initialization unit 302, the next assigned robot movement plan policy update unit 303, the next assigned robot movement plan value function update unit 304, and the next assigned robot movement plan value function convergence determination unit 305. The processing is the same as in the case of the article trajectory calculation unit 100 except that the state transition probability and the reward are replaced with the equations (9) and (10), respectively.

The next assigned robot movement plan value function initialization unit 302 sets an initial value of the value function V ^π (s3) for each s3εS3. An appropriate value may be set for this.

The next robot movement plan policy updating unit 303 calculates the following equation for all combinations of s3∈S3 and a3∈A3, and the value of V ^π3 [p _m ] [p _s ] [p _n ] (s3) is calculated. The maximum action a2 is set to the value of policy π3 [p _m ] [p _s ] [p _n ] (s3).

Here, Σ represents the total sum of all possible s3 ′, and γ is a constant that satisfies a predetermined 0 <γ <1.0. In addition, the value of V ^π3 [p _m ] [p _s ] [p _n ] ( _{s 3} ′) in the expression (11) is the next assigned robot movement plan at the first execution time of the next assigned robot movement plan policy update unit 303. The initial value set by the value function initialization unit 302 is used, and the value of the value function updated by the following assigned robot movement plan value function update unit 304 is used in the second and subsequent executions.

The next assigned robot movement plan value function updating unit 304 updates the value function V ^π3 [p _m ] [p _s ] [p _n ] (s 3) for each s 3 ∈ S 3 according to the following equation.

The next assigned robot movement plan value function convergence determination unit 305 updates and updates the value of V ^π3 [p _m ] [p _s ] [p _n ] (s3) updated by the next assigned robot movement plan value function update unit 304. ^{Until the error from the} previous V ^π3 [p _m ] [p _s ] [p _n ] ( _{s 3} ) is less than or equal to a predetermined threshold value, the process returns to the policy update unit 303 for the next assigned robot movement plan, and the policy and value function Repeat the update process. When the value is equal to or less than a predetermined threshold, it is determined that the value function V ^π3 [p _m ] [p _s ] [p _n ] ( _{s 3} ) has converged, and the value function V ^π3 [p _m ] [p _s ] [ p _n ] (s3) and policy π3 [p _m ] [p _s ] [p _n ] (s3) are output.

  Similarly, the collision avoidance state transition model setting unit 306 sets the state transition probability of the above equation (9) and the reward of the equation (10.2).

The processes of the collision avoidance value function initialization unit 307, the collision avoidance policy update unit 308, the collision avoidance value function update unit 309, and the collision avoidance value function convergence determination unit 310 are V ^π3 [p _m ] [p _s ]. [p _n ] (s3) is ^replaced by V ^π3 [p _m ] [p _s ] [p _s ] (s3), and π3 [p _m ] [p _s ] [p _n ] (s3) is π3 [p _m ] [p _s ] [p _s ] (s3) is replaced, and the state function probabilities and rewards are replaced by equation (9) and equation (10.2), respectively. Since the processing is the same as that of the unit 302, the next assigned robot movement plan policy updating unit 303, the next assigned robot movement plan value function updating unit 304, and the next assigned robot movement plan value function convergence determining unit 305, description thereof is omitted.

Also in the movement route planning unit 300, the action value used in the Markov state space is the movement of the robot itself, so the calculated action policy π3 [p _m ] [p _s ] [p _n ] (s3) It can be used as an action policy for the virtual robot to move to the position where the virtual robot can press the article at the position where switching is performed. The difference from π2 [p _m ] [p _n ] (s2) is that an action (a3 = 5) of another robot that moves the article is defined.

[Modification]
In the above description, the possible values of the action variable a3 in the movement route planning unit 300 are six types of 0 to 5, but the virtual robot w and the virtual robot v move at the same time (that is, the virtual robot w takes 1 item). Action variables (a3 = 6, 7, 8, 9) are further increased to 10 kinds of action variables that mean that the action of step movement and the action of the virtual robot v moving in a predetermined direction are performed simultaneously, You can do the calculation, where
a3 = 6: The virtual robot moves to the right a3 = 7: The virtual robot moves to the left a3 = 8: The virtual robot moves upward a3 = 9: The virtual robot moves downward. In this case, the above <Condition C> is replaced with the above (1) to (6) plus the following (7) to (10).
(7) a3 = 6 & Xrw '= Xrw + 1 &Yrw' = Yrw & s1_trj [l] '= s1_trj [l + 1]
(8) a3 = 7 & Xrw '= Xrw &Yrw' = Yrw + 1 & s1_trj [l] '= s1_trj [l + 1]
(9) a3 = 8 & Xrw '= Xrw-1 &Yrw' = Yrw & s1_trj [l] '= s1_trj [l + 1]
(10) a3 = 9 & Xrw '= Xrw &Yrw' = Yrw-1 & s1_trj [l] '= s1_trj [l + 1]

The action variable a3∈ {0,1,2,3,4,5} indicates that either the virtual robot w or the virtual robot v pushing the article is moving or both are stationary. It is action. When an action is performed using this, even if the robot w and the robot v can actually act in parallel (simultaneously), the action plan is such that the action is performed sequentially. Since there are few, there exists an advantage that the calculation of a movement path | route plan part can be calculated quickly.
On the other hand, when a3∈ {0,1,2,3,4,5,6,7,8,9}, the robot w and the robot v are obtained by a3∈ {6,7,8,9}. Parallel actions can also be considered. Since the number of action variables increases, it takes time to calculate the movement route planning unit. However, there is an advantage that an action sequence that is actually planned is not wasted and a more appropriate action sequence (short required time) can be obtained.

[Scheduling unit; step S4]
With the above processing, an action policy π2 [p _m ] [p _n ] for robot operation when a robot i continuously moves an article from the trajectory section s1_trj [p _m +1] to s1_trj [p _n +1]. (s2) and while robot i moves the article from s1_trj_ [p _m +1] to s1_trj [p _{m + 1} +1], the robot j in charge of the next section The action policy π3 [p _m ] [p _s ] [p _n ] ( _{s 3} ) of the next robot j to move to the position where the article at the top position of the robot can be pushed) can be calculated. Thus, for example, while the robot 1 continuously moves the article according to π2 [p ₀ ] [p ₁ ] (s2), the robot 2 according to π3 [p ₀ ] [p ₀ ] [p ₁ ] (s3). Is moved to a position where s1_trj [p ₁ +1] can be moved to s1_trj [p ₁ +2], and the robot 3 moves s1_trj according to π3 [p ₀ ] [p ₀ ] [p ₂ ] (s3). move the article in the [p ₂ +1] to movable positioned s1_trj [p ₂ +2], after moving the robot 1 article s1_trj [p ₁ +1], the robot 2 action policy .pi.2 [ p ₁ ] [p ₂ ] (s2) to move the article, and then the robot 3 switches the action policy to π2 [p ₂ ] [p ₃ ] (s2) to move the article. Cooperative operation is feasible. After that, the scheduling unit 400 may obtain a combination of the action policy to be used and the robot number so that the switching of the action policy and the allocation of the robot match.

As a scheduling method, a known technique such as a graph search may be used. For example, the scheduling can be performed by dynamic programming in a Markov state space. Now, using Markov state space with p robots
s4 = (Cb, C1,…, Cp, CA1,…, CAp)
a4 = (d1,…, dp)
And

Here, Cb is the article position, and its domain is the end point of the section where Partition [l] = 0, that is, Cb∈ {p ₀ , p ₁ ,..., P _f }. Ci (i = 1, 2, ..., p) is the position of robot i, but not the actual robot X and Y coordinates, but the position of the article that the robot has pushed to that position (i), or from now on. This refers to the position (b) of the article that is about to be expressed, and is expressed by the end point number p _m as in Cb (the position of (b) or (b) can be specified if the position of the article is known, so it may be the end point number. ). There can be two actual robot positions even with the same Ci value. The actual robot position is either (A) or (B) depending on how the robot has reached the position of the article indicated by the Ci value. If the actual robot position is (Xri, Yri), then (Xri, Yri) ∈ {Ready_Pos_k (s1_trj [p _m +1], a1_trj [p _m ]) ∨Ready_Pos_k (s1_trj [p _m +1], a1_trj [ p _m +1])} (m = 0,1, ..., f). Here, Ready_Pos_k (s1_trj [p _m +1], a1_trj [p _m ]) is the position of the robot after pushing the article from s1_trj [p _m ] to the start position (s1_trj [p _m +1]) of the section Ready_Pos_k (s1_trj [p _m +1], a1_trj [p _m +1]) indicates the article at the end of the section (s1_trj [p _m +1]) in the direction indicated by a1_trj [p _m +1] It is a position that can be pressed. CAi is information for identifying this, and takes an identification value indicating whether the action di executed by the robot i one step before is the article pushing action π2 or the switching movement action π3. This identification value is, for example, a Boolean value (1 = article pushing behavior π2, 0 = switching movement behavior π3).

di (i = 1,..., p) represents an action policy selected by the robot i. The value of di is divided into a value indicating an article pushing action of π2, a movement action of π3, or waiting. Since the number of start points of the section in the optimum path point sequence s1_trj is f + 1, until di = 0, 1,..., F indicates the movement or waiting of π3 (waiting if the value of di indicates the current robot position). When di = f + 1, it indicates the selection of the article pushing action π2. When the waiting action is selected when CAi = 1, the collision avoidance action π3 calculated in the third layer is performed, and when it is selected when CAi = 0, nothing is done. When Cb = p _r , Ci = p _q and the value of di is 0, 1,…, f, the robot uses π3 [p _r ] [p _q ] [p _di ] (s3) as the action strategy to use. When the value of di is f + 1, the robot selects π2 [p _r ] (s2) as the action policy to be used. The value of di that can be selected depends on the robot state Ci, CAi and the article. There is a limit depending on the position Cb. The restriction rules are as follows.

(1) Only when Ci = Cb and CAi = 0, a value for selecting the article pushing action π2 can be selected as the di value.
(2) Only when CAi = 1, a value for selecting the moving action π3 as the di value can be selected.
(3) The standby action can be selected in any case.

  The restriction (1) indicates that the robot cannot select the action π2 that presses the article continuously, and the restriction (2) indicates that only the robot after pressing the article can select the movement action π3.

The above is the limit of the di value for a single robot, but further, as the limit of the entire action a4,
Any one of the elements (d1, ..., dp) must be the action π2 that pushes the article,
This also adds a restriction.

The state transition probability in the Markov state transition model is defined by the following conditions.

However, <Condition D> is a case where any of the following is satisfied.
Cb = p _r and the time _{Cb '= p r, Ci =} p and the Ci when _q' and = p _{q ',} when any of the following conditions
[1] di = π2 (push) & r '= r + 1 &q' = q + 1 & CAi '= 1
[2] di = π3 (move) & r '= r + 1 &q' = di & CAi '= 0
[3] di = π3 (standby) & r '= r + 1 &q' = q & CAi '= CAi

  Under the above conditions, [1] means that when the robot selects an article pushing action, both the robot and the article position move to the next section end point and change to CAi = 1 (article pushing action). To do. [2] means that when the moving action is selected, the position of the robot moves to the target indicated by the action, the article position moves to the next section end point, and changes to CAi = 0 (moving action). . [3] indicates that the robot positions Ci and CAi are not changed by the standby action, and only the article position moves to the next section end point.

Further, when the action a4εA4 is executed in the state s4 and the state transits to s4′εS4, a reward R4 given from the environment to the action subject is set by the following equation.

If the article reaches the start point (Cb '= p _f ) of the last section, the position of one of the robots is p _f , and the action that reaches there is a moving action (CAi' = 0), the robot Since it is determined that the pushing of the article can be started in the final section, it can be seen that the article can reach the target position. Only at that time, the value of the reward R4 is set to 1, and in other cases, the reward is set to 0. When policy π4 (s4) is obtained by dynamic programming using P4 (s4 ′ | s4, a4) and R4 (s4 ′, s4, a4) defined as above, policy π4 (s4) Indicates the action policy selected by each robot di at the end point Cb of the section. From this, in order from Cb = s1_trj [0], a policy π4 (s4) for the state variable s4 corresponding to the end point of the section is obtained, so that the action policy selected by each robot di at the end point (switching position) of each section is obtained. I want. The scheduling unit 400 outputs this as a scheduling result.

  When the action policy of the scheduling unit 400 is obtained, the dynamic planning method in the Markov model determination process (MDP) is used similarly to the article trajectory planning unit 100, the switching position determination unit 200, and the movement route calculation unit 300. However, using dynamic programming in the quasi-Markov decision process (SMDP), which is an extended method of MDP, can obtain scheduling results that can further reduce the time required in the real environment. SMDP differs from MDP in that the value function is calculated in consideration of the length of time required for each state transition.

  Let τ (s4, s4 ', a4) be the time required for state transition to s4' when action a4 is selected in state s4. The required time may be obtained by simulation or the like.

The scheduling value function initialization unit sets the initial value of the value function V ^π4 (s4) for each s4εS4 to zero.

The scheduling policy updater calculates the following equation for all combinations of s4εS4 and a4εA4, and sets the action a4 that maximizes the value of V ^π4 (s4) to the value of the policy π4 (s4).

Here, Σ represents the total sum of all possible s4 ′, and γ is a constant that satisfies a predetermined 0 <γ <1.0. In addition, the value of V ^π4 (s4 ′) in equation (13) uses the initial value set by the scheduling value function initialization unit at the first execution of the scheduling policy update unit, and the second and subsequent executions Then, the value of the value function updated by the following scheduling value function updating unit is used.

The scheduling value function updating unit updates the value function V ^π4 (s4) according to the following equation for each s4εS4.

[Error feedback unit; Step S5]
As described above, in the hierarchical operation plan calculation up to the article trajectory planning unit 100, the switching position determining unit 200, the movement route calculating unit 300, and the scheduling unit 400, the conditions ( In the optimum route point sequence, it may happen that an appropriate solution cannot be calculated in the operation plan in the lower hierarchy (switching position determination unit 200, movement route calculation unit 300, scheduling unit 400). In order to cope with such a case, an error feedback unit (not shown) may be further provided to perform recalculation.

The feedback unit uses the value of the value function in the switching operation of the assigned robot obtained by the movement route calculation unit 300. For example, in the state variable s3 = (Xb, Yb, Xrw, Yrw) of the value function V ^π3 [p _m ] [p _s ] [p _n ] (s3), (Xb, Yb) = s1_trj [p _m +1] And V ^π3 [p _m ] [p _s ] [p _n ] when (Xrw, Yrw) is set to a position after the robot w has pushed the article from position s1_trj [p _s ] to s1_trj [p _s +1]. The value of (s3) is checked for all s, m pairs where m <n, s <= m, and in all these cases the value of ^Vπ3 [p _m ] [p _s ] [p _n ] (s3) If is not greater than or equal to 0, it is impossible to switch the robot appropriately according to the policy π3 [p _m ] [p _s ] [p _n ] (s3). Then, it is impossible to switch the assigned robot at the article position s1_trj [p _n +1] in the article trajectory at the current time, so the movement from the article position s1_trj [p _n ] to s1_trj [p _n +1] It is not appropriate to include in s1_trj. Therefore, in the article trajectory calculation unit 300, the state transition probability P1 (s1_trj [p _n +1] | s1_trj [p _n ], a1) = 0 (that is, s1_trj [p _n ] to s1_trj [p _{n in} any action a1) +1]) and P1 (s1_trj [p _n ] | s1_trj [p _n ], a1) = 1 (the transition probability when staying at s1_trj [p _n -1] is 1) Re-plan after changing the value of the state transition probability P1 of the state transition model.

Thus, in the feedback unit, the value function V ^π3 [p _m ] [p _s ] [p corresponding to the policy π3 [p _m ] [p _s ] [p _n ] ( _s 3) obtained by the movement path calculation unit 300 is obtained. _n ] (s3), after (Xb, Yb) = s1_trj [p _m +1] and (Xrw, Yrw) push the article from position s1_trj [p _s ] to s1_trj [p _s +1] If the value of V ^π3 [p _m ] [p _s ] [p _n ] ( _{s 3} ) is not greater than or equal to 0, the first state transition model setting unit (not shown)
P1 (s1_trj [p _n +1] | s1_trj [p _n ], a1) = 0
P1 (s1_trj [p _n ] | s1_trj [p _n ], a1) = 1
Then, the optimal path point sequence is updated by performing processing of the article trajectory calculation unit 100 (from the article trajectory calculation value function initialization unit to the optimal path point sequence calculation unit). The process of the switching position determination unit 200, the movement route calculation unit 300, the scheduling unit 400, and the error feedback unit 500 is repeated using the updated optimum route point sequence.

[Variation: Adjustment of the number of assigned robots]
As described above, whether or not the assigned robot can be switched is determined using the value function V ³ [p _m ] [p _s ] [p _n ] ( _s ³ ). Instead of requesting recalculation of the article trajectory, a configuration may be adopted in which a responsible robot is added. For example, if the robot that exists at the position after pushing the article position from s1_trj [p _s ] to s1_trj [p _s + 1] cannot move to the position where the article at the position of s1_trj [p _n +1] can be pushed For example, the value function V ^π3 [p _m ] in the mission environment at any intermediate position while the article moves from the position s1_trj [p _m +1] to the position s1_trj [p _{m + 1} +1]. Add a new robot to the robot position where the value of [p _s ] [p _n ] (s3) can be positive, and move the new robot to a position where it can push the item at the position of s1_trj [p _n +1]. , S1_trj [p _n +1] is in charge of the previous article movement.

  It is also effective to use the action policy obtained by the above processing as an action policy for selecting a greedy action in reinforcement learning, and to make the learning itself more efficient.

<Hardware configuration example of robot cooperative transport planning device>
The robot cooperative transfer planning apparatus according to the above-described embodiment may include an input unit to which a keyboard or the like can be connected, an output unit to which a liquid crystal display or the like can be connected, a CPU (Central Processing Unit) [cache memory, or the like. ] RAM (Random Access Memory) or ROM (Read Only Memory) and external storage device as a hard disk, and data exchange between these input unit, output unit, CPU, RAM, ROM, and external storage device It has a bus that can be connected. If necessary, each device may be provided with a device (drive) that can read and write a storage medium such as a CD-ROM. A physical entity having such hardware resources includes a general-purpose computer.

  The external storage device of the robot cooperative transport planning device stores a program for transport planning and data necessary for processing of this program [not limited to the external storage device, for example, the program is read by a dedicated storage device. It may be stored in a certain ROM. ]. Data obtained by the processing of these programs is appropriately stored in a RAM or an external storage device. Hereinafter, a storage device that stores data, addresses of storage areas, and the like is simply referred to as a “storage unit”.

  In the robot cooperative transfer planning apparatus, the program stored in the storage unit and the data necessary for processing the program are read into the RAM as necessary, and are interpreted and executed by the CPU. As a result, the transportation plan is realized by the CPU realizing predetermined functions (article trajectory calculation unit, switching position determination unit, movement path planning unit, scheduling unit, error feedback unit).

<Supplementary note>
The present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the spirit of the present invention. In addition, the processing described in the above embodiment may be executed not only in time series according to the order of description but also in parallel or individually as required by the processing capability of the apparatus that executes the processing. .

  Further, when the processing functions in the hardware entity described in the above embodiment are realized by a computer, the processing contents of the functions that the hardware entity should have are described by a program. Then, by executing this program on a computer, the processing functions in the hardware entity are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used. Specifically, for example, as a magnetic recording device, a hard disk device, a flexible disk, a magnetic tape or the like, and as an optical disk, a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read Only). Memory), CD-R (Recordable) / RW (ReWritable), etc., magneto-optical recording medium, MO (Magneto-Optical disc), etc., semiconductor memory, EEP-ROM (Electronically Erasable and Programmable-Read Only Memory), etc. Can be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In this embodiment, a hardware entity is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

Claims (6)

A robot cooperative transfer planning apparatus for obtaining an action plan of each robot for moving a moving object (hereinafter referred to as an article) from a start position to a target position in cooperation with a plurality of robots,
Based on a Markov state transition model in which the position of the article is a state variable and the movement direction of the article is a behavior variable, a chain of positions (hereinafter referred to as optimal) that forms the movement path of the article from the start position to the target position. An article trajectory calculation unit for obtaining a route point sequence),
Ideal position that can take each action included in the union of all of the plurality of robots with the position of the article having each position included in the optimum path point sequence as a domain and at least the action that each robot can take Based on a Markov state transition model in which a pair of a position of a robot (hereinafter referred to as an ideal robot) is a state variable and an action that the ideal robot can take is an action variable, each position included in the optimal path point sequence is The action taken by the ideal robot when the article is present (hereinafter referred to as the optimum action sequence) is determined, and the optimum robot is moved to the section where the ideal robot can move the article in the optimum path point sequence. The ideal robot is divided according to each switching position by dividing the route point sequence (hereinafter referred to as the robot switching position) and the optimal action sequence. Action column (hereinafter referred to as a first interval optimal action sequence) corresponding to the section and a switching position determination unit and determines,
For each section that can be taken according to the robot switching position, the position of the article having each position included in the section as a definition area and the ideal robot that is not in charge of transporting the article in the section A section after the section based on a Markov state transition model in which the pair of the position of the robot (hereinafter referred to as the second ideal robot) is a state variable and the action that the second ideal robot can take is an action variable. In order to move the second ideal robot to a position where the article can be moved in step S2, a series of actions to be taken by the second ideal robot in the section (hereinafter referred to as a second section optimum action sequence) is obtained. A travel route planning department;
A robot cooperative transfer planning apparatus including a scheduling unit that assigns the first section optimum action sequence and the second section optimum action sequence to each robot so that there is no contradiction between the robots. The robot cooperative transfer planning device according to claim 1,
The scheduling unit
For each section that can be taken according to the robot switching position, the article can be pushed to the position or the position of the article with the starting point position of the section as the definition area and each position included in the section. The action of each robot moving the article, with a state variable as a set of the position of each robot whose position is defined as the position and information for identifying the action of each robot immediately before the section. Alternatively, based on the Markov state transition model, the behavior of each robot included in the state variable is determined based on a Markov state transition model, where the behavior variable is a movement to a position where the article can be moved or a standby behavior. By assigning to each robot an action corresponding to each of the variables in order from the beginning of the optimum path point sequence, each robot is assigned the first section maximum. Robot Cooperative Transportation Planning and wherein the assigning the action sequence and the second segment optimal action sequence. The robot cooperative transfer planning device according to claim 1 or 2,
The switching position determination unit
The robot switching position obtained by rewriting the robot switching position at a position that is further divided into sections where one robot included in the plurality of robots can move the article is used as the robot switching position. Cooperative transport planning device. The robot cooperative transfer planning device according to any one of claims 1 to 3,
The travel route planning department
A robot that is waiting at the same position in each of the sections, and that a robot included in the plurality of robots also obtains an action for avoiding a collision with the article moved in the section. Cooperative transport planning device. A robot cooperative transport planning method for obtaining an action plan of each robot for moving a moving object (hereinafter referred to as an article) from a start position to a target position in cooperation with a plurality of robots,
The position where the article trajectory calculation unit forms the movement path of the article from the start position to the target position based on a Markov state transition model in which the position of the article is a state variable and the movement direction of the article is a behavior variable. Article trajectory calculation step for obtaining a chain (hereinafter referred to as an optimal path point sequence) of
Each action included in the union of all of the plurality of robots of the position of the article whose definition area is each position included in the optimal path point sequence and the action that each robot can take Based on a Markov state transition model, the optimal path point sequence is based on a combination of the position of an ideal robot (hereinafter referred to as an ideal robot) that can take a state as a state variable, and an action that the ideal robot can take as a behavior variable. And determining an action (hereinafter referred to as an optimal action sequence) that the ideal robot takes when the article is at each position included in the position, and moving the article by one ideal robot in the optimal path point sequence. The ideal robot is switched by dividing the optimal action sequence by dividing the optimal route point sequence into sections where the optimal route point sequence is divided (hereinafter referred to as the robot switching position). Action column for each section is divided by the position (hereinafter, referred to as first zone optimal action sequence) and a switching position determination step of determining,
For each section that the movement path planning unit can take in accordance with the robot switching position, the position of the article having each position included in the section as a domain of definition, and the ideal robot that has the article in the section. Based on a Markov state transition model in which a pair of the position of a robot that is not in charge of conveyance (hereinafter referred to as a second ideal robot) is a state variable, and an action that the second ideal robot can take is an action variable. In order to move the second ideal robot to a position where the article can be moved in a section after the section, a sequence of actions to be taken by the second ideal robot in the section (hereinafter, second section optimum) A travel route planning step for obtaining an action sequence),
A robot cooperative transfer planning method, wherein the scheduling unit includes a scheduling step in which the first section optimum action sequence and the second section optimum action sequence are assigned to the robots so that there is no contradiction among the robots.       A program for causing a computer to function as the robot cooperative transfer planning apparatus according to any one of claims 1 to 4.