JP2023168996A - Movement control system, movement control method, movement controller, moving apparatus, movement control program, and program for moving apparatuses - Google Patents

Abstract

To provide a movement control system, movement control method, movement controller, moving apparatus, movement control program, and program for moving apparatuses capable of coping with an unexpected event while suppressing a cost of calculations.SOLUTION: A movement control system searches a moving route of a moving apparatus (S204), allows the moving apparatus to move along the moving route (S205), and acquires an ambient situation of the moving apparatus (S207). An area laid out around the moving apparatus is defined as a neighbor area. The moving apparatus acts on the basis of a measure associated with each condition pattern signifying whether entry to each of unit areas included in the neighbor area is permitted (S214).SELECTED DRAWING: Figure 20

Description

The present invention relates to a movement control system and a movement control method for controlling a mobile device that performs guided movement, autonomous movement, etc. along a route, a movement control device and a movement device used in such a movement control system, and a movement control device and a movement device thereof. The present invention relates to a movement control program and a movement device program for realizing such a movement control device and movement device.

BACKGROUND ART In facilities such as large-scale factories, mobile devices such as AGVs (Automated Guided Vehicles) and AMRs (Automated Mobile Robots) are used. In order to set a route for a mobile device to reach a target position within a moving range of a facility, etc., it is necessary to perform route planning. Various methods have been proposed for solving route planning problems to avoid collisions with obstacles such as walls and other moving devices, and to plan routes that reduce travel costs. For example, as a method for solving route planning problems, there are studies that use rule-based methods such as Dijkstra's algorithm (Non-patent Document 1), multi-agent nets (Non-patent Document 2), and Petri nets (Non-patent Document 3). Proposed. Furthermore, a method of solving a route planning problem using a reinforcement learning method has also been proposed (Non-Patent Document 4, Non-Patent Document 5).

K.A. Dowsland and A.M. Greaves, Collision avoidance in bi-directional AGV systems, Journal of the Operational Research Society, Vol.45, No.7, pp.817-826 (1994). Kunihiko Hiraishi, “Modeling multi-agent systems using Petri nets,” Transactions of the Institute of Systems, Control and Information Engineers, Vol. 45, No. 8, pp. 439-444 (2001). M. Dotoli and M.P. Fanti, Colored timed Petri net model for real-time control of automated guided vehicle systems, International Journal of Production Research, Vol.42, No.9, pp.1787-1814 (2004). Michiko Watanabe, Masashi Furukawa, Masahiro Kinoshita, Yunobo Kakazu, “Research on autonomous transportation of multiple AGVs using Q-learning,” Journal of the Japan Society for Precision Engineering, Vol. 67, No. 10, pp. 1609-1614 (2001). S.M. Jeon, K.H. Kim and H. Kopfer, Routing automated guided vehicles in container terminals through the Q-learning technique, Logistics Research, Vol.3, No.1, pp.19- 27 (2011).

However, the rule-based method for solving route planning problems has a problem in that when the movement range becomes large, the number of required simulations increases and the calculation cost increases. Furthermore, when using multiple mobile devices, the method of solving route planning based on rules can immediately obtain an almost optimal route plan as an alternative route in case of unexpected situations such as failure of other mobile devices. Problems may arise where it is difficult to do so.

Methods of solving route planning problems using reinforcement learning methods can flexibly plan routes in response to unexpected situations such as the occurrence of obstacles or delays in moving devices, but reinforcement learning requires trial and error. Since it is a method for finding the optimal solution, the computational cost required for processing such as trial and error and learning becomes a problem. Furthermore, the method of solving a route planning problem using reinforcement learning has a problem in that the optimality of the planned route is not guaranteed.

The present invention has been made in view of such circumstances, and its main purpose is to provide a movement control system that can cope with unexpected situations while suppressing calculation costs.

The present application also discloses a movement control method implemented in the movement control system according to the present invention. Further, the present application discloses a movement control device and a movement device used in the movement control system according to the present invention, and a movement control program and a movement device program for realizing these.

In order to solve the above problems, the present disclosure provides a movement control system that includes a movement device and controls the movement of the movement device, the movement control system controlling the movement path of the movement device from a movement start position to a target position. A route search means for searching, a situation acquisition means for acquiring the surrounding situation of the mobile device, and a unit area set based on the size of the mobile device for each unit movement distance that is a reference for the movement of the mobile device. A region arranged around the moving device is defined as a neighboring region, and the surrounding situation acquired by the situation obtaining means is applied to the neighboring region, and the moving device is determined for each unit region included in the neighboring region. an entry permission determining means for determining whether the device can enter; and a device determining whether the device can enter, based on a policy of the mobile device that is associated with each state pattern indicating whether it is possible to enter each unit area included in the nearby area. a policy acquisition unit that acquires a policy associated with a state pattern corresponding to the determination result of whether or not each unit area can be entered, and the route search unit searches based on the policy acquired by the policy acquisition unit; movement permission determining means for determining whether or not movement is possible along the travel route, and determining to take an action indicated by the policy acquired by the policy acquisition means when it is determined that movement along the travel route is not possible. The method is characterized by comprising a means for determining an action.

The movement control system further includes a movement control device capable of communicating with the mobile device, wherein the number of the movement devices is plural, and the movement control device includes the route search means, the situation acquisition means, and the approach permission determination. means, the policy acquisition means, the movement permission determining means, the action determining means, and the means for transmitting, to the corresponding mobile device, an action command for each of the mobile devices to perform the action determined by the action determining means, The entry permission determining means determines that one of the plurality of mobile devices cannot enter a unit area in which another mobile device is located and/or a unit area in which the other mobile device moves. Features.

Further, in the movement control system, there is a plurality of moving devices, and one of the plurality of moving devices includes at least the situation acquisition means, the approach permission determination means, the policy acquisition means, and the It is characterized by comprising a movement permission determining means and the action determining means, wherein the entry permission determining means determines that a unit area in which another mobile device is located and/or a unit area in which the other mobile device moves cannot be entered. do.

Further, in the movement control system, the route search means searches for a graph in which a movable range of the mobile device is defined by a unit area as a node and a movable route between unit areas as an edge. , a movement route from a start node corresponding to a movement start position of the mobile device to a goal node corresponding to a target position is searched using a graph search algorithm.

Further, in the movement control system, the nearby area is defined based on movement of two units of a unit movement distance.

Further, in the movement control system, the relationship between the state pattern and the policy acquired by the policy acquisition means is a learning model obtained by reinforcement learning.

Further, in the movement control system, the learning model indicates the respective action values of moving and stopping the moving device in the movement direction for each assumed state pattern, and the policy acquisition means is configured to It is characterized in that the strategy is to move or stop in the direction with the highest value.

Furthermore, the movement control method disclosed in the present application is a movement control method using a movement device and a movement control device that controls movement of the movement device, the movement control method including searching a movement route from a movement start position of the movement device to a target position. acquiring the surrounding situation of the mobile device; and determining the unit area set based on the size of the mobile device for each unit movement distance serving as a reference for the movement of the mobile device. defining an area arranged around the area as a nearby area, applying surrounding conditions to the nearby area, and determining whether or not the moving device can enter with respect to each unit area included in the nearby area; Based on the policy of the mobile device, which is associated with each status pattern indicating whether or not it is possible to enter each unit area included in the nearby area, the area is associated with a state pattern corresponding to a determination result of whether or not each unit area can be entered. a step of acquiring a policy; a step of determining whether movement along the movement route is possible based on the acquired policy; and a step of causing the mobile device to move along the movement route when it is determined that movement is possible. If it is determined that movement is not possible, the mobile device takes the action indicated by the policy.

Furthermore, the movement control device disclosed in the present application is a movement control device that controls the movement of a movement device, and includes means for searching a movement route from a movement start position of the movement device to a target position, and means for searching a movement route of the movement device from a movement start position to a target position. means for acquiring the situation, and a unit area set based on the size of the mobile device, an area arranged around the mobile device for each unit movement distance serving as a reference for movement of the mobile device, and a nearby area. means for determining whether or not the moving device can enter each unit area included in the nearby area by applying a surrounding situation to the nearby area; means for acquiring a policy associated with a state pattern corresponding to a determination result of whether or not each unit area can be entered, based on a policy of the mobile device that is associated with each state pattern indicating whether or not entry is allowed; and the obtained policy. means for determining whether or not movement is possible along the movement route based on the above, and an action instruction that causes the user to move along the movement route if it is determined that movement is possible, and to take the determined action if it is determined that movement is not possible. and means for transmitting the information to the mobile device.

Further, the mobile device disclosed in the present application is a mobile device that moves along a travel route, and includes a means for acquiring surrounding conditions and a unit area set based on the size of the mobile device. For each standard unit movement distance, define the surrounding area as a nearby area, apply the surrounding situation to the nearby area, and determine whether or not the own aircraft can enter for each unit area included in the nearby area. and a means for determining whether or not each unit area included in the neighborhood area can be entered into, and a state pattern corresponding to a determination result of whether or not each unit area can be entered, based on a measure that is associated with each state pattern indicating whether or not entry is possible to each unit area included in the neighboring area. a means for acquiring a policy that has been determined; a means for determining whether movement is possible along the movement route based on the acquired policy; a means for moving along the movement route when it is determined that movement is possible; and a means for moving along the movement route based on the acquired policy; and means for taking the action indicated by the acquired policy when it is determined that.

Furthermore, the movement control program disclosed in the present application is a movement control program that causes a computer equipped with communication means for communicating with a mobile device to control the movement of the mobile device, and the program causes the computer to move the mobile device from a movement start position to a target position. a step of searching for a travel route to the mobile device; a step of acquiring the surrounding situation of the mobile device; and a step of determining a unit area set based on the size of the mobile device, a unit movement distance serving as a reference for the movement of the mobile device. In each case, an area arranged around the mobile device is defined as a nearby area, and the surrounding situation is applied to the nearby area to determine whether or not the mobile device can enter into each unit area included in the nearby area. a step of determining, and a state corresponding to a determination result of whether or not each unit area can be entered, based on a strategy of the mobile device associated with each state pattern indicating whether or not each unit area included in the nearby area can be entered; a step of acquiring a policy associated with the pattern; a step of determining whether or not movement is possible along the movement route based on the acquired policy; and when it is determined that movement is possible, moving along the movement route; The mobile device is characterized in that, when it is determined that movement is not possible, transmitting an action command to the mobile device to cause the mobile device to take the determined action.

Furthermore, the mobile device program disclosed in the present application is a mobile device program that is executed by a mobile device moving along a travel route, and includes steps for the mobile device to obtain surrounding conditions, and a step for determining the size of the mobile device. The unit area set based on the unit area is defined as a neighboring area for each unit movement distance that is the reference for movement of the own aircraft, and the surrounding situation is applied to the neighboring area, and The step of determining whether or not the own aircraft can enter each unit area included in the unit area, and the step of determining whether or not the own aircraft can enter each unit area included in the neighboring area based on the policy associated with each state pattern indicating whether or not it is possible to enter each unit area included in the neighboring area. a step of acquiring a policy associated with a state pattern corresponding to a determination result of whether entry is possible; a step of determining whether movement is possible along the movement route based on the acquired policy; and a step when it is determined that movement is possible. , the step of moving along the movement route, and the step of taking the action indicated by the acquired policy when it is determined that movement is not possible.

The movement control system, etc. disclosed in the present application performs a movement of the mobile device based on a policy associated with a state pattern in which the surrounding situation of the mobile device is reflected in a neighborhood area in which a unit area is arranged around the mobile device. Determine. As a result, the movement control system and the like disclosed in the present application can achieve excellent effects, such as allowing a mobile device that moves along a movement route to capture changes in the surrounding situation as a state pattern and reflect them in its actions. .

In the movement control system disclosed in the present application, an example of a graph applied to the layout is shown. FIG. 2 is an explanatory diagram conceptually showing an example of a nearby area defined around a mobile device in the movement control system disclosed in the present application. In the movement control system disclosed in the present application, it is an explanatory diagram illustrating whether it is possible to enter a unit area included in a nearby area. FIG. 2 is an explanatory diagram conceptually showing an example of Q learning used to determine the behavior of a mobile device in the movement control system disclosed in the present application. FIG. 2 is an explanatory diagram showing an example of a nearby area of a moving device and a moving route superimposed on a layout in the movement control system disclosed in the present application. FIG. 2 is an explanatory diagram showing an example of the state of a nearby area around a moving device in the movement control system disclosed in the present application. FIG. 2 is an explanatory diagram showing an example of a nearby area of a moving device and a moving route superimposed on a layout in the movement control system disclosed in the present application. FIG. 2 is an explanatory diagram showing an example of the state of a nearby area around a moving device in the movement control system disclosed in the present application. FIG. 2 is an explanatory diagram showing an example of a nearby area of a moving device and a moving route superimposed on a layout in the movement control system disclosed in the present application. FIG. 2 is an explanatory diagram showing an example of the state of a nearby area around a moving device in the movement control system disclosed in the present application. FIG. 2 is an explanatory diagram conceptually showing an example of behavior updating by Q learning in the movement control system disclosed in the present application. FIG. 2 is an explanatory diagram conceptually showing an example of the relationship between a state pattern and an array of Q values in the movement control system disclosed in the present application. FIG. 2 is an explanatory diagram conceptually showing an example of the relationship between a state pattern and an array of Q values in the movement control system disclosed in the present application. FIG. 2 is an explanatory diagram conceptually showing an example of the relationship between a state pattern and an array of Q values in the movement control system disclosed in the present application. 1 is a conceptual diagram showing an example of a system configuration of a movement control system disclosed in the present application. FIG. 2 is a block diagram showing an example of the hardware configuration of a mobile device used in the movement control system disclosed herein. FIG. 2 is a block diagram showing an example of the hardware configuration of a movement control device used in the movement control system disclosed in the present application. FIG. 1 is a block diagram showing an example of a hardware configuration of a machine learning device disclosed in the present application. It is a flowchart which shows an example of the machine learning process of the machine learning device with which the movement control system of this application disclosure is provided. It is a flowchart which shows an example of movement control processing of a movement control device provided in a movement control system disclosed in the present application. It is a flow chart which shows an example of movement instruction processing of a movement control device provided in a movement control system disclosed in the present application. It is a flowchart which shows an example of the movement control process of the mobile device with which the movement control system of this application disclosure is provided. FIG. 2 is an explanatory diagram showing a graph used in a simulation test of the movement control system disclosed in the present application. FIG. 3 is an explanatory diagram showing the movement routes of each mobile device superimposed on a graph used in a simulation test of the movement control system disclosed in the present application. It is a graph which shows an example of the result of the simulation test of the movement control system of this application disclosure. It is a graph which shows an example of the result of the simulation test of the movement control system of this application disclosure. FIG. 2 is an explanatory diagram conceptually showing an example of a nearby area defined around a mobile device in the movement control system disclosed in the present application.

Embodiments of the present invention will be described in detail below. Note that the following embodiment is an example of embodying the present invention, and is not intended to limit the technical scope of the present invention.

<Application example>
The movement control system disclosed in the present application is applied to control the movement of mobile devices such as AGVs (Automated Guided Vehicles) and AMRs (Automated Mobile Robots). For example, the present invention is applied to on-premises or off-premises logistics in facilities such as large-scale factories such as semiconductor factories, warehouses such as distribution centers, etc., where multiple AGVs are used simultaneously. Specifically, multiple AGVs move along guide paths laid on the ground within the facility, avoiding obstacles such as installed objects and objects, as well as collisions with other AGVs. Applies to the form in which tasks are executed. Note that the installed object refers to a fixed obstacle such as a wall, a pillar, or a building, and the arranged object refers to a repositionable obstacle such as a frame or a conveyed object. Furthermore, objects that are located in a different position than expected, such as overturned objects, other malfunctioning moving devices, etc., are also treated as obstacles. Examples of tasks include movement involving work such as loading and unloading at a designated location.

<Theory>
The basic theory of the movement control system disclosed in this application will be explained. First, an overview of the theory applied to the mobile control system will be explained. The mobility control system disclosed herein allocates a movable range of a mobile device such as an AGV to a graph in which connection relationships are defined as nodes and edges. Then, the movement control system searches for the shortest route from the start node, which is the starting position of the movement of the mobile device, to the goal node, which is the target position, using a graph search algorithm such as the A ^* algorithm.

Furthermore, a neighborhood area (state space) in which unit areas set based on the size of the mobile device are arranged is defined around the mobile device. If there are obstacles such as various installed objects, placement objects, or other moving devices in a unit area included in the defined nearby area, that unit area is determined to be a unit area into which moving devices cannot enter. . Note that for other mobile devices, not only when they exist in that unit area, but also when they move into that unit area, it is determined that the unit area is not allowed to enter. Further, a unit area in which an obstacle or another moving device exists or does not move is determined to be a unit area into which the moving device can enter. On the other hand, in the movement control system, the strategy of the movement device is associated with various state patterns indicating whether or not entry into each unit area included in the neighborhood area is possible. The strategy of the mobile device associated with the state pattern is determined in advance by reinforcement learning such as Q learning, for example, as a learning model such as a Q function (behavior value function).

The mobile device moves along the searched movement route, and when obstacles and other mobile devices exist or move within the nearby area, it is determined whether or not it is possible to enter each unit area included in the nearby area. Act by referring to the policy associated with the state pattern corresponding to the state. Specifically, the moving device continues moving along the moving route, or takes actions to avoid collisions such as decelerating, stopping, changing course, etc. indicated in the strategy.

Next, a detailed explanation of the movement control system will be given using specific examples. FIG. 1 shows an example of a graph applied to a layout in the movement control system disclosed herein. FIG. 1 shows an example in which 16×16 nodes are assigned to a square horizontal area that resembles the layout of a factory. In the figure, white circles numbered "0" to "5" indicate the positions of mobile devices such as AGVs. In the figure, vertical hatching, horizontal hatching, and squares with vertical and horizontal crosshatching indicate nodes that perform standby charging, loading, and unloading, respectively. The area indicated by diagonal cross-hatching in the figure indicates obstacles such as various installed objects and objects, and is an area into which the moving device cannot enter.

In the example of FIG. 1, mobile device "0" is on standby while charging, and mobile devices "1" to "5" are performing a given task. In the example of Figure 1, the task is for a mobile device to move from a starting position to a designated loading node while avoiding collisions with obstacles and other mobile devices, interference, deadlock, and other abnormalities. This shows a series of operations in which cargo is loaded at a loading node, moved to a specified unloading node, and unloaded at the unloading node.

FIG. 2 is an explanatory diagram conceptually showing an example of a nearby area defined around a mobile device in the movement control system disclosed herein. FIG. 2 illustrates a neighborhood area defined on a horizontal plane in which a mobile device moves. The neighborhood area includes a plurality of unit areas indicated by squares. In FIG. 2, the moving device is represented by a white circle, and an arrow extending from the white circle indicates the traveling direction of the moving device. The size of the unit area included in the neighborhood area is set based on the size of the mobile device. FIG. 2 shows an example in which a square having a size that accommodates the horizontal spread of the mobile device is set as a unit area. In the vicinity area, unit areas are arranged side by side for each unit movement distance, which is a reference for movement of the moving device. In FIG. 2, the unit moving distance of the moving device is the length of one side of the unit area, and each time the moving device moves by the unit moving distance, it determines whether the movement is possible based on the surrounding state. The neighborhood area illustrated in FIG. 2 is defined based on movement of two units of unit movement distance.

FIG. 3 is an explanatory diagram illustrating whether or not a unit area included in a nearby area can be entered in the movement control system disclosed herein. FIG. 3 illustrates a neighborhood region defined around mobile device "0". In FIG. 3, the unit areas shown by cross-hatching are obstacles such as installed objects and arranged objects, and indicate unit areas into which the moving device cannot enter. The white circles marked with "3" and "5" are other moving devices "3" and "5", the white circles indicate the current positions, and the arrows extending from the white circles indicate the direction of movement. The unit area where another moving device is located and the unit area into which the other moving device moves are unit areas that should not be entered in order to prevent collisions. That is, for mobile device "0", the unit area where the obstacle is located, the unit area where other mobile devices are located, and the unit area where other mobile devices move are unit areas that cannot be entered; Unit areas other than the area are enterable unit areas. In the example of FIG. 3, when moving device "0" moves forward in the direction of movement, a collision occurs because it enters the unit area in which another moving device "3" moves. Therefore, mobile device "0" needs to take action to avoid collision.

A learning model such as a Q function created in advance using reinforcement learning such as Q learning is used to determine the behavior of a mobile device to avoid a collision. FIG. 4 is an explanatory diagram conceptually showing an example of Q learning used to determine the behavior of a mobile device in the movement control system disclosed herein. Q-learning is one of the typical reinforcement learning methods. Reinforcement learning consists of two components: a dynamic environment and an agent. Reinforcement learning is a method that uses trial and error to find a strategy that maximizes the reward obtained when an agent takes a certain action in a certain environment in response to changes in the environment caused by that action. FIG. 4 conceptually shows the relationship between the agent and the environment, as well as the state s _t , reward r _t , and action a _t . When the agent performs an action a _t , the environment changes, and the changed state s _t and its reward r _t are calculated. The agent takes the next action a _t in response to the changed state s _t . Maximizing the reward r _t is based on the perspective of expressing the value of an action when taking a certain action in a certain state s _t as a Q value using a Q function, and selecting actions so that the Q value is maximized. Therefore, in Q learning, learning is performed so as to maximize the action value function Q _π (s _t , a _t ) defined by the following equation (1).

Q _π (s _t , a _t )=E _π [r _t |s _t =s, a _t =a] ...Formula (1)

In the movement control system disclosed herein, the neighborhood region illustrated in FIG. 2 is defined as a state space used for Q learning. The neighborhood area illustrated in FIG. 2 is defined by arranging unit areas corresponding to two units of the unit movement distance of the mobile device, and is based on the minimum configuration when considering the movement of other mobile devices. As described above, the movement control system disclosed in the present application defines a simple state space as a neighborhood area, which is relative to the position of the moving device and focuses only on the surrounding state, and does not require a large storage capacity. The relatively simple state space is significantly different from the conventional path planning in general Q-learning, in which the entire layout of the movable range of the mobile device is defined as the state space. The movement control system disclosed in the present application is relative and does not require a large storage capacity, and by defining the neighborhood area as a simple state space, the layout of the range in which the mobile device can move, the layout of the range in which the mobile device can move, etc. Even if the environment such as the number of devices changes, there is no need to re-learn the Q function, which should be learned and acquired in advance, resulting in excellent effects. Furthermore, the relative, simple neighborhood region that does not require a large storage capacity has excellent effects, such as being able to shorten the time required for Q learning itself.

Next, a specific example of determining the behavior of a mobile device by the movement control system disclosed in the present application will be described. FIG. 5 is an explanatory diagram showing an example of a nearby area of a moving device and a movement route superimposed on a layout in the movement control system disclosed in the present application. FIG. 6 is an explanatory diagram illustrating an example of the state of the vicinity area around the moving device in the movement control system disclosed herein. Figure 5 shows the target unloading from the range of the vicinity area of mobile device "0" searched using the A ^* algorithm and the current position of mobile device "0" on the graph applied to the layout illustrated in Figure 1. The travel route to the node that performs this is shown superimposed. FIG. 6 shows the state of the neighborhood area defined around the mobile device "0" in FIG. Within the vicinity area defined around the mobile device "0" located at the node that performs loading, there are unit areas that cannot be entered due to the presence of obstacles, but there are unit areas in the traveling direction indicated by the arrows. There are no obstacles in the area. Then, within the vicinity area defined around the mobile device "0", there is no other mobile device existing or moving. Therefore, the Q function is not updated, and the mobile device "0" moves along the travel route.

FIG. 7 is an explanatory diagram showing an example of a nearby area of a moving device and a movement route superimposed on a layout in the movement control system disclosed in the present application. FIG. 8 is an explanatory diagram showing an example of the state of the vicinity area around the moving device in the movement control system disclosed herein. FIG. 7 shows a nearby area and a moving route regarding the mobile device "1," and FIG. 8 shows the state of the nearby area regarding the mobile device "1." Since another mobile device "3" exists within the neighborhood area defined around the mobile device "1", the Q function is updated. However, since the unit area in front of the moving device "1" is neither the unit area where the other moving device "3" exists nor the unit area where the moving device "3" is moving, it is difficult to move under the situations illustrated in FIGS. Device "1" makes a behavioral decision to move along the moving route without changing the moving route.

FIG. 9 is an explanatory diagram showing an example of a nearby area of a moving device and a movement route superimposed on a layout in the movement control system disclosed in the present application. FIG. 10 is an explanatory diagram illustrating an example of the state of the vicinity area around the mobile device in the movement control system disclosed herein. FIG. 9 shows a nearby area and a moving route regarding mobile device "2," and FIG. 10 shows a state of the nearby area regarding mobile device "2." Since another mobile device "4" exists within the vicinity area defined around the mobile device "2", the Q function is updated. In the situations illustrated in FIGS. 9 and 10, in order to avoid collision with the moving moving device "4", the moving device "2" does not move along the moving route, but decelerates, stops, turns left, A decision is made to change to an evasive action such as turning right or going backwards.

FIG. 11 is an explanatory diagram conceptually showing an example of behavior updating by Q learning in the movement control system disclosed herein. In FIG. 11, the upper right diagram shows the surrounding observation results applied to the vicinity area defined around the mobile device "1". In the example of FIG. 11, the moving device "1" is shown to move in the direction shown downward in the figure. The movement control system rotates the state fitted to the nearby area so that the direction of movement along the searched movement route is upward in the figure. The rotation of the neighborhood is shown in FIG. 11 as a transition from the top right diagram to the left diagram. In the example shown in FIG. 11, the traveling direction is changed from downward to upward, so the rotation is performed by 180°. Rotation of the neighborhood region is a process applied to a learning model created in advance using reinforcement learning such as Q learning.

The lower diagram in FIG. 11 conceptually shows the content of the learning model. The learning model assigns state numbers, which serve as identification numbers, to various states in the neighborhood, and stores the state numbers in association with the Q value, which serves as a reward for each action, according to the state pattern that indicates the state of the neighborhood. are doing. In Figure 11, N state patterns (N is a natural number) with state numbers 1, 2, 3, 4, ..., N-3, N-2, N-1, N are displayed vertically. The Q values for each behavior according to the state pattern are shown side by side. For example, if the layout allows movement in only four directions: up, down, left, and right (front, back, left, and right), there are five actions: stop, up (forward), down (backward), left (left turn), and right (right turn). A Q value is calculated and assigned to each action.

The movement control system refers to the trained Q-learning model based on the identification number that identifies the state pattern corresponding to the result of rotating the neighborhood area based on the surrounding observation results, and determines the behavior from the Q-learning model. Get the array of assigned Q values. The diagram on the right side of FIG. 11 conceptually shows the arrangement of Q values obtained from the Q learning model. In the example shown in Figure 11, the upward direction indicating movement along the searched movement route is "1", the left and right directions where obstacles exist and collisions occur are "-1", and no collisions occur but useless Stop and downward direction, which are movements, are "0". The movement control system controls the movement device to take an upward action, which is an action with the highest Q value, according to the acquired arrangement.

The relationship between the state pattern of a nearby region and the behavior of a mobile device will be explained using several examples. FIG. 12 is an explanatory diagram conceptually showing an example of the relationship between the state pattern and the Q value arrangement in the movement control system disclosed herein. In Figure 12, the left diagram shows the surrounding observation results applied to the nearby area, and the right diagram is rotated so that the traveling direction along the searched movement route is upward. It shows an array of Q values obtained from the Q learning model based on the state pattern and the state pattern. FIG. 12 shows an example in which an obstacle, a mobile device "1", and a mobile device "2" exist around a mobile device "0" that is a target of behavior control. In FIG. 12, moving device "0" collides with an obstacle when moving to the left (turning left), colliding with moving device "2" when moving upward (moving forward), and moving to the right (turning right). Collision with mobile device "1". The Q values are "0", "-1", "0", "-1" and "-1" for stop, top, bottom, left and right, respectively. In the example shown in FIG. 12, the maximum Q value is "0", so stop and downward are selected. For example, depending on the random number, the movement of the moving device "0" is determined to be stop or downward movement. (retreat) is decided.

FIG. 13 is an explanatory diagram conceptually showing an example of the relationship between a state pattern and an array of Q values in the movement control system disclosed herein. FIG. 13 shows another example of the state illustrated in FIG. 12. FIG. 13 shows an example in which an obstacle and mobile devices "1", "2", and "3" exist around mobile device "0", which is the target of behavior control. There is. In FIG. 13, moving device "0" collides with an obstacle when moving downward, colliding with moving device "2" when moving upward, and colliding with moving device "1" when moving or stopping to the right. do. The Q values are "-1", "-1", "-1", "0", and "-1" for stop, top, bottom, left, and right, respectively. In the example shown in FIG. 13, since the maximum Q value is "0", the movement of mobile device "0" is determined to be movement to the left.

FIG. 14 is an explanatory diagram conceptually showing an example of the relationship between a state pattern and an array of Q values in the movement control system disclosed herein. FIG. 14 shows still another example of the state illustrated in FIGS. 12 and 13. FIG. 14 shows an example in which an obstacle, a mobile device "1", and a mobile device "2" exist around a mobile device "0" that is a target of behavior control. Note that the mobile device "2" is stopped above (in front of) the mobile device "0". In FIG. 14, moving device "0" collides with an obstacle when moving downward, colliding with a stopped moving device "2" when moving upward, and moving device "2" when moving or stopping to the right. 1" collides. The Q values are "-1", "-1", "-1", "0", and "-1" for stop, top, bottom, left, and right, respectively. In the example shown in FIG. 14, since the maximum Q value is "0", the movement of mobile device "0" is determined to be movement to the left.

As described above, the movement control system disclosed in the present application searches the movement route of the mobile device using a graph search algorithm such as the A ^* algorithm, and determines the behavior of the mobile device after the movement starts according to the observation results of the nearby area. It is determined based on the Q value of the state pattern obtained from the Q learning model.

<Implementation example>
Hereinafter, an implementation example of the movement control system S of the present disclosure described in the drawings will be described with reference to the drawings.

<System configuration>
FIG. 15 is a conceptual diagram showing an example of the system configuration of the movement control system S disclosed herein. The movement control system S includes a plurality of movement devices 1, a movement control device 2 that controls the movement devices 1, and a machine learning device 3 that provides a learning model M to the movement control device 2. The movement control system S is applied to physical distribution within or outside a facility such as a large-scale factory or warehouse. Inside the facility, a guide path P such as a magnetic tape is laid down on the ground, and the moving device 1 moves along the guide path P laid out vertically and horizontally. The mobile device 1 performs bidirectional communication with the mobile control device 2. The machine learning device 3 generates a learning model M such as a Q learning model by reinforcement learning such as Q learning. The machine learning device 3 provides the generated learning model M to the movement control device 2 via communication or a recording medium such as a semiconductor memory.

<Hardware configuration of various devices>
Next, configuration examples of various devices included in the movement control system S will be described. FIG. 16 is a block diagram showing an example of the hardware configuration of the mobile device 1 used in the mobile control system S disclosed herein. The moving device 1 is a device such as an AGV capable of autonomous movement, and performs transportation operations such as loading and unloading of transported objects such as various parts and products in addition to movement. The mobile device 1 includes a control section 10, a storage section 11, a communication section 12, a position acquisition section 13, a route detection section 14, a surrounding situation acquisition section 15, a drive section 16, a drive control section 17, a transport section 18, and a transport control section 19. It has various configurations such as

The control unit 10 is a processor such as a CPU (Central Processing Unit) that executes processing to control the entire device, and includes various circuits such as an information processing circuit, a time measurement circuit, and a register circuit.

The storage unit 11 is configured using nonvolatile memories such as hard disks, SSDs (Solid State Drives), RAIDs (Redundant Arrays of Inexpensive Disks), flash memories, and volatile memories such as various RAMs (Random Access Memory). It is a storage unit. The storage unit 11 stores various programs such as a mobile device program 110 for implementing the mobile device 1 according to the mobile control system S disclosed herein. When the control unit 10 executes various steps included in various programs such as the mobile device program 110 stored in the storage unit 11, the AGV functions as the mobile device 1 of the present disclosure.

The communication unit 12 is a unit configured using hardware and software for performing communication defined by a wireless communication standard such as Wi-Fi (registered trademark) defined by IEEE. The communication unit 12 performs a transmission process of transmitting information such as position information indicating the position of the mobile device 1 to the movement control device 2, and a reception process of receiving information such as action commands transmitted from the movement control device 2. .

The position acquisition unit 13 is a unit that acquires position information indicating the position of its own device. The position acquisition unit 13 is configured using, for example, hardware and software such as GNSS (Global Navigation Satellite System) and various position detection sensors. In addition, in the movement control system S, for example, a detection device such as an image sensor, an optical sensor, a magnetic sensor, etc. is arranged in the facility, and position information indicating the position of the movement device 1 obtained based on the results detected by the detection device is provided. may be transmitted from the detection device to the mobile device 1. When transmitting position information from the detection device to the mobile device 1, the position acquisition unit 13 is configured using a unit that receives position information. Furthermore, position information indicating the position of each mobile device 1 obtained based on the detection result by the detection device is transmitted from the detection device to the movement control device 2, and the movement control device 2 detects the positions of all the mobile devices 1. The mobile device 1 may access the mobile control device 2 to obtain position information. In this case as well, the position acquisition section 13 is configured using a unit that receives position information.

The route detection unit 14 is a unit such as various sensors that detects that the mobile device 1 is moving along a route. For example, when a guide path P using magnetic tape is laid, a sensor such as a magnetic sensor is used as the route detection section 14. Further, when a tape having characteristics such as a specific color and light emission is laid as the guide path P, a sensor such as an image sensor is used.

The surrounding situation acquisition unit 15 is a unit that obtains the surrounding situation using a sensor that detects the surrounding situation and an analysis circuit that analyzes the detection result by the sensor. To detect the surrounding situation, an image sensor captures an image of the surrounding situation and acquires the status of other mobile devices 1 and obstacles, a laser sensor detects the surrounding situation using a laser, and communicates with the other mobile device 1. A short-range wireless sensor or the like is used.

The drive unit 16 is a unit required to move tires, motors that drive the tires, and the like, and the drive control unit 17 is configured using a driver that controls the drive unit 16. The conveyance section 18 is a unit required for conveyance such as a robot arm or a belt conveyor, and the conveyance control section 19 is configured using a driver that controls the conveyance section 18 .

FIG. 17 is a block diagram showing an example of the hardware configuration of the movement control device 2 used in the movement control system S disclosed herein. The movement control device 2 is a device that controls the movement device 1, and is realized using a computer such as a server computer. The movement control device 2 includes various components such as a control section 20, a storage section 21, a first communication section 22, a second communication section 23, and a task acquisition section 24.

The control unit 20 is a processor such as a CPU that controls the entire device, and includes various circuits such as an information processing circuit, a clock circuit, and a register circuit.

The storage section 21 is a storage unit configured using nonvolatile memory and volatile memory. The storage unit 21 stores various programs such as a basic program (OS: Operating System), application programs that operate on the basic program, and various data. As one of the application programs, the storage unit 21 stores a movement control program 210 for realizing the movement control device 2 according to the movement control system S disclosed herein. Further, the storage unit 21 stores a learning model M such as a Q learning model acquired from the machine learning device 3.

The first communication unit 22 is a unit for communicating with the mobile device 1. The first communication unit 22 receives information such as position information indicating the position of the mobile device 1 from the mobile device 1, and transmits information such as action commands to the mobile device 1.

The second communication unit 23 is a unit for communicating with the machine learning device 3, and communicates with the machine learning device 3 through communication means such as a WAN (Wide Area Network), a LAN (Local Area Network), a dedicated communication network, and a communication line. connect. The movement control device 2 receives information such as the learning model M from the machine learning device 3 via the second communication unit 23, but it is also possible to exchange information via a communication medium such as a semiconductor memory. .

The task acquisition unit 24 is a unit that acquires a task to be executed by the mobile device 1. The task to be executed by the mobile device 1 is, for example, an operation of transporting objects such as parts or products from a designated loading node to a designated unloading node. Tasks are acquired by methods such as automatic generation by a scheduling program, input by a person in charge, or reception of commands from other devices.

The computer configured as described above reads various programs such as the movement control program 210 from the storage unit 21 under the control of the control unit 20, and executes various steps included in the various programs, thereby achieving the system disclosed in the present application. It operates as a movement control device 2.

FIG. 18 is a block diagram showing an example of the hardware configuration of the machine learning device 3 disclosed herein. The machine learning device 3 is a device that generates a learning model M such as a Q learning model used for determining the behavior of the mobile device 1, and is realized using a computer such as a general-purpose computer. The machine learning device 3 includes various components such as a control section 30, a storage section 31, a communication section 32, and the like.

The storage unit 31 stores various programs such as a machine learning program 310 for realizing the machine learning device 3 related to the movement control system S disclosed herein. Furthermore, the storage unit 31 stores a learning model M such as a Q learning model generated by machine learning. The learning model M stored in the storage unit 31 is provided to the movement control device 2 via the communication unit 32, for example.

<Software processing of various devices>
Next, software processing of various devices included in the movement control system S will be explained. FIG. 19 is a flowchart illustrating an example of machine learning processing of the machine learning device 3 included in the movement control system S disclosed herein. The machine learning process is a process that generates a learning model M. The control unit 30 included in the machine learning device 3 executes machine learning processing by executing the machine learning program 310.

The control unit 30 of the machine learning device 3 initializes a Q function targeting a nearby region that defines the behavior of the mobile device 1 (S101). The control unit 30 creates a state pattern assumed in the nearby area (S102). In step S102, a state pattern is created that reflects the situation assuming the arrangement of obstacles and the movement of other moving devices 1 as shown in FIG. create.

The control unit 30 observes the current state of the mobile device 1 (S103), determines various patterns of behavior (S104), and observes the next situation that will occur when the determined behavior is taken (S105). A reward serving as a Q value is calculated based on the result (S106), and a Q function is updated based on the calculated reward (S107). In step S104, for example, actions such as forward movement, left turn (move to the left), right turn (move to the right), retreat, and stop are determined for movement along the set movement route. In step S106, the reward is calculated such that, for example, forward motion indicating movement along the set movement route is "1", left turn and right turn is "-1", and backward motion and stop is "0".

The control unit 30 determines whether reinforcement learning for all assumed state patterns has been completed (S108).

In step S108, if it is determined that there is a state pattern for which reinforcement learning has not been completed (S108: NO), the control unit 30 returns to step S102 to perform reinforcement learning on other state patterns, and performs subsequent processing. repeat.

In step S108, if it is determined that the reinforcement learning for all state patterns has been completed (S108: YES), the control unit 30 ends the machine learning process. The Q function updated by reinforcement learning is provided from the machine learning device 3 to the movement control device 2 as a learning model M such as a Q learning model.

As described above, the machine learning device 3 executes machine learning processing.

FIG. 20 is a flowchart illustrating an example of a movement control process of the movement control device 2 included in the movement control system S disclosed herein. The movement control process is a process that controls the behavior of the mobile device 1 according to the surrounding situation. The control unit 20 included in the movement control device 2 executes movement control processing by executing the movement control program 210.

The control unit 20 of the movement control device 2 determines whether the task acquisition unit 24 has acquired a task (S201).

If it is determined in step S201 that no task has been acquired (S201: NO), the control unit 20 returns to step S201 and repeats the subsequent processing.

In step S201, if it is determined that the task has been acquired (S201: YES), the control unit 20 determines that among the plurality of mobile devices 1 under control, the mobile device 1 that has not been given a task and is on standby is It is determined whether it exists (S202). In step S202, the movement control device 2 communicates with each mobile device 1, acquires the status of the mobile device 1 in standby, executing a task, returning to the waiting place, etc., and based on the acquired status, the mobile device 1 in standby Determine the presence or absence.

If it is determined in step S202 that there is no mobile device 1 on standby (S202: NO), the control unit 20 returns to step S202 and repeats the subsequent processing. The repetitive process of determining whether there is a mobile device 1 on standby is a process of waiting until a mobile device 1 on standby occurs. Note that processing when there is no mobile device 1 on standby can be designed as appropriate, such as processing to reply to the acquired task that the task cannot be executed.

If it is determined in step S202 that there is a mobile device 1 on standby (S202: YES), the control unit 20 determines a mobile device 1 to which a task is to be assigned from among the mobile devices 1 on standby, and assigns the task to the mobile device 1 that is on standby. A task is assigned (S203).

The control unit 20 acquires the standby position of the mobile device 1 to which the task has been assigned, and searches for a route from the acquired standby position to the target position of movement (S204). Step S204 is a process of searching for a movement route from the start node corresponding to the movement start position of the mobile device 1 to the goal node corresponding to the target position using a graph search algorithm such as the A ^* algorithm. In step S204, for example, if the acquired task is transportation of goods, first, a route search is performed from the node that performs standby charging to the loading node, a route search is performed from the loading node to the unloading node, Furthermore, a route search is performed from the unloading node to the node that performs standby charging.

The control unit 20 transmits route information indicating the route obtained through the search as a movement command from the first communication unit 22 to the mobile device 1, and performs movement instruction processing to instruct movement along the movement route (S205). . In step S205, the control unit 20 transmits a movement command from the first communication unit 22 to the mobile device 1 to which the task has been assigned. The mobile device 1 that has received the movement command starts moving according to the received movement command.

The control unit 20 determines whether the mobile device 1 has completed the task (S206).

If it is determined in step S206 that the task has not been completed (S206: NO), the control unit 20 acquires the surrounding situation of the mobile device 1 (S207). If the movement control device 2 is able to grasp the positions and situations of all the movement devices 1 and the situations of obstacles, the control unit 20 executes the acquisition process in step S207 by reading the surrounding situation. If the movement control device 2 is unable to grasp the position and situation of another movement device 1, the control unit 20 sends a command to the movement device 1 that is the target of behavior control to request acquisition of the surrounding situation through the first communication. The acquisition process of step S207 is executed by receiving, at the first communication unit 22, the information indicating the surrounding situation transmitted from the first communication unit 22 and acquired by the surrounding situation acquisition unit 15 by the mobile device 1. Note that the movement control device 2 may use these acquisition processes in combination, and by using them in combination, it is possible to improve the accuracy of situation acquisition.

The control unit 20 determines whether entry is possible for all unit areas included in the vicinity area based on the acquired situation around the mobile device 1 (S208). In step S208, the control unit 20 determines that the unit area where the other moving device 1 is located, the unit area where the other moving device 1 moves, and the unit area where the obstacle exists cannot be entered.

Based on the determination result of step S208, the control unit 20 determines whether there is a unit area in the vicinity that cannot be entered (S209). In step S209, since unit areas that cannot be entered due to the presence of obstacles have already been considered at the route search stage, only unit areas that cannot be entered due to other mobile devices 1 are taken into consideration. By determining the presence or absence of a region, it is possible to speed up the processing. Note that the process in step S209 may also be performed by taking into account the occurrence of unexpected obstacles such as overturning of placed objects and falling of conveyed objects, and determining the presence or absence of a unit area that cannot be entered, including obstacles. good. When determining the presence or absence of a unit area that cannot be entered, including obstacles, the surrounding situation of the mobile device 1 can be obtained by using the detection result of the surrounding situation obtained by the surrounding situation acquisition device provided in the mobile device 1. is desirable.

In step S209, if it is determined that there is no unit area in the vicinity that cannot be entered due to another moving device 1 (S209: NO), the process returns to step S205, and the control unit 20 controls the movement along the movement route. A movement instruction process is performed (S205). In step S205, the control unit 20 transmits a movement command to continue movement along the movement route from the first communication unit 22 to the mobile device 1, and repeats the processing from step S206 onwards.

In step S209, if it is determined that the unit areas included in the nearby area include a unit area that cannot be entered (S209: YES), the control unit 20 sets a state indicating whether or not it is possible to enter each unit area included in the nearby area. A policy corresponding to the pattern is acquired from the learning model M (S210). Based on the acquired policy, the control unit 20 updates the behavior of the mobile device 1 (S211), and transmits an action command instructing the updated behavior from the first communication unit 22 to the mobile device 1. An action instruction is given to the device 1 (S212). The processing in steps S210 to S212 is a process for causing the mobile device 1 to take the action with the highest Q value based on the learning model M described using FIG. 11. Note that the actions instructed by the action command include not only actions such as left turn, right turn, and backing up, but also actions such as deceleration and stopping.

The mobile device 1 that has received the action command through the communication unit 12 performs actions such as moving, decelerating, and stopping based on the action command.

The control unit 20 determines whether the mobile device 1 after the action has deviated from the travel route searched in the travel route search process (S213). In step S213, if the action taken by updating the action is a left turn or right turn, it is determined that the action is off the movement route, and if the action is deceleration, stopping, or retreating, it is determined that the action is located on the movement route.

In step S213, if it is determined that the object is located on the movement route and not off the movement route (S213: NO), the control unit 20 returns to step S205 and repeats the subsequent processing.

If it is determined in step S213 that the vehicle has deviated from the travel route (S213: YES), the control unit 20 resets the route (S214). The resetting of the route is performed as a return process in which the vehicle returns to a position from which it deviated from the movement route and resumes movement from there, and a re-search process in which the movement route is searched again from the position from which it deviated.

In step S206, if it is determined that the task has been completed (S206: YES), the control unit 20 ends the movement control process.

As described above, the movement control device 2 executes movement control processing.

The movement control processing of the movement control device 2 described above is a form in which the movement control device 2 centrally controls the behavior of the movement device 1. The movement control system S disclosed in the present application can be deployed in various forms, and can also be implemented as distributed control in which the movement device 1 itself makes behavior decisions. Below, an example of a mode in which each mobile device 1 makes a behavior determination will be described.

FIG. 21 is a flowchart illustrating an example of movement instruction processing of the movement control device 2 included in the movement control system S disclosed herein. The movement instruction process is a process in which the movement control device 2 assigns a task to the mobile device 1 and instructs it to move. The control unit 20 included in the movement control device 2 executes movement instruction processing by executing the movement control program 210.

The control unit 20 of the movement control device 2 determines whether the task acquisition unit 24 has acquired a task (S301). The process in step S301 is repeatedly executed until a task is acquired.

In step S301, if it is determined that the task has been acquired (S301: YES), the control unit 20 determines whether or not there is a waiting mobile device 1 (S302). The process of step S302 is repeatedly executed until the waiting mobile device 1 is confirmed.

In step S302, if it is determined that there is a mobile device 1 on standby (S302: YES), the control unit 20 assigns a task to the mobile device 1 on standby (S303), and searches for a route to the target position ( S304), and instructs movement along the travel route indicating the searched route (S305). The movement instruction in step S305 is performed by the movement control device 2 transmitting a movement command including a movement route from the first communication unit 22 to the mobile device 1. Note that the movement control device 2 may notify the movement device 1 of the target position as a movement command, and the movement device 1 may perform a route search.

As described above, the movement control device 2 executes the movement instruction process.

FIG. 22 is a flowchart illustrating an example of a movement control process of the mobile device 1 included in the movement control system S disclosed herein. The movement control process is a process in which the mobile device 1 controls the movement of its own device according to a movement command. A control unit 10 included in the mobile device 1 executes a movement control process by executing a mobile device program 110.

The control unit 10 of the mobile device 1 receives a movement command through the communication unit 12 (S401), moves along the movement route based on the received movement command (S402), and determines whether or not the task is completed. Determination is made (S403).

If it is determined that the task has not been completed (S403: NO), the control unit 10 acquires the surrounding situation (S404), and determines whether or not each unit area included in the nearby area can be entered (S405). In step S404, if the movement control device 2 knows the status of all the mobile devices 1, the mobile device 1 accesses the movement control device 2 and acquires the surrounding situation. If the movement control device 2 does not know the situation of the mobile device 1, the mobile device 1 uses the surrounding situation acquisition unit 15 to acquire the surrounding situation including the situation of other mobile devices 1.

Based on the determination result in step S405, the control unit 10 determines whether there is a unit area that cannot be entered in the nearby area (S406), and determines that there is no area that cannot be entered due to another mobile device 1. If so (S406: NO), the process returns to step S402 and continues moving along the movement route (S402).

In step S406, if it is determined that the unit area included in the neighborhood area includes a unit area that cannot be entered (S406: YES), the control unit 10 acquires a policy corresponding to the state pattern from the learning model M (S407 ). The learning model M may be stored in the storage unit 11 of the mobile device 1, and the mobile device 1 may access the movement control device 2 and use the learning model M stored in the movement control device 2. You can.

The control unit 10 updates the behavior based on the acquired policy (S408), and executes the updated behavior (S409). After the action, the control unit 10 determines whether or not it has deviated from the movement route (S410), and if it is determined that it is located on the movement route (S410: NO), returns to step S402 and continues moving along the movement route. (S402).

In step S410, if it is determined that the object has deviated from the movement route (S410: YES), the control unit 10 resets the path (S411), returns to step S402, and continues movement (S402).

In step S403, if it is determined that the task has been completed (S403: YES), the control unit 10 ends the movement control process.

As described above, the mobile device 1 executes the movement control process.

<Simulation results>
Next, the results of a simulation test performed on the movement control system disclosed in the present application will be described. FIG. 23 is an explanatory diagram showing a graph used in a simulation test of the movement control system disclosed in the present application. FIG. 24 is an explanatory diagram showing the movement route of each mobile device superimposed on the graph used in the simulation test of the movement control system disclosed in the present application. The explanatory diagram shown in FIG. 23 is a graph in which the movable range of a mobile device such as an AGV is defined as a relationship between nodes and edges. In FIG. 23, black circles indicate nodes, and solid lines connecting nodes indicate edges. In FIG. 24, the white circles numbered "1" to "7" indicate the start nodes of the mobile device, the arrows extending from the white circles indicate the route traveled by the mobile device, and the tip of the arrow indicates the goal node. ing. In FIGS. 23 and 24, mobile device "3" waited for one step at node "76" to avoid a collision with mobile device "7". Furthermore, after reaching node "131" once, mobile device "1" once avoids to node [113] in order to avoid collision with mobile device "2", and then returns to node [131] again. Moved to. As described above, in the movement control system disclosed herein, the mobile device moves to the goal node while avoiding collisions with other mobile devices.

FIG. 25 is a graph showing an example of the results of a simulation test of the movement control system disclosed in the present application. FIG. 25 shows the relationship between the number of moving devices on the horizontal axis and the calculation time required for route planning on the vertical axis. Note that the number of nodes was fixed at 30. As shown in FIG. 25, as the number of mobile devices increases, the calculation time tends to increase linearly with respect to the number of mobile devices.

FIG. 26 is a graph showing an example of the results of a simulation test of the movement control system disclosed in the present application. FIG. 26 shows the relationship between the number of nodes on one side of a layout arranged in a grid on the horizontal axis and the calculation time required for route planning on the vertical axis. Note that the number of moving devices was fixed at five. As shown in FIG. 26, although the calculation time increases as the number of nodes increases, the calculation time tends to be within the square of the number of nodes.

As described above, the movement control system disclosed in the present application determines the behavior of the mobile device based on the state pattern in which the surrounding conditions of the mobile device are reflected in the neighborhood area in which the unit area is arranged around the mobile device. . By determining behavior by comparing with state patterns, it is possible to respond flexibly to changes in the surrounding situation. For example, situations that are different from those at the time of route searching may occur, such as situations in which another mobile device is in a different position than planned due to an abnormality such as a malfunction, or the position of an obstacle has changed due to falling objects, etc. Even if a collision occurs, it is possible to take action to avoid a collision, which has excellent effects. Furthermore, when the size of the nearby area is defined as two units of the unit movement distance, the movement control system disclosed in the present application suppresses an increase in state patterns and an increase in calculation load due to enlarging the nearby area. , it is possible to take into consideration the position of another moving device after it has been moved, and other excellent effects can be achieved.

Furthermore, although the movement control system disclosed in the present application finds the relationship between state patterns and actions using a reinforcement learning method such as Q learning, it is possible to create a learning model using reinforcement learning separately. Therefore, it is possible to suppress an exponential increase in calculation costs such as calculation load and calculation time during actual movement with respect to the number of mobile devices and the number of nodes, and other excellent effects can be achieved. Moreover, the movement control system disclosed in the present application has excellent effects such as being able to reuse the same learning model even if the layout of the movement range of the mobile device is different.

Furthermore, the automatic control system disclosed in the present application can derive the shortest route by using a graph search algorithm such as the A ^* algorithm when searching for a route, so if there is no interference between mobile devices, the shortest route It has excellent effects, such as making it possible to move around.

The present invention is not limited to the embodiments described above, and can be implemented in various other forms. Therefore, such embodiments are merely illustrative in all respects, and should not be interpreted in a limiting manner. The scope of the present invention is indicated by the claims, and is not restricted in any way by the main text of the specification. Furthermore, all modifications and changes that come within the scope of equivalents of the claims are intended to be within the scope of the present invention.

For example, in the above embodiments and simulations, an AGV that moves on a plane is applied as a moving device, but the movement control system disclosed in the present application is not limited to this, and can move in a three-dimensional space including above and below. It is also possible to apply devices such as UAV (unmanned aerial vehicle; so-called "drone") and underwater mobile robots as the mobile device. When applied to a mobile device that performs three-dimensional movement, the nearby area is also defined as a three-dimensional three-dimensional space.

FIG. 27 is an explanatory diagram conceptually showing an example of a nearby area defined around a mobile device in the movement control system disclosed herein. FIG. 27 illustrates a neighborhood area defined in a three-dimensional space in which a mobile device moves. The neighborhood area includes a plurality of unit areas shown as cubes. The size of the unit area included in the neighborhood area is set based on the size of the mobile device. FIG. 27 shows an example in which a cube having a size that accommodates the width and width of the moving device is set as the unit area. The neighborhood area illustrated in FIG. 27 is defined based on movement of two units of unit movement distance. Note that the image simulating a drone shown in the center of the nearby area shows the mobile device itself, and the images simulating a drone shown in the surrounding unit area show other mobile devices.

Further, in the embodiment, the unit area is defined as a square, but the movement control system disclosed in the present application is not limited to this, and may be defined in other shapes such as a regular hexagon.

Furthermore, in the embodiment, the range of values that the Q value can take is expressed as an integer from "-1" to "1", but the movement control system of the present disclosure is not limited to this, and can be expressed in various ways. It is possible to set compensation regulations. For example, it is possible to set various conditions, such as giving the corresponding Q value as "-10" when the moving device collides and further stops action is selected.

Further, in the embodiment described above, a learning model is used in which the relationship between state patterns and actions is determined by reinforcement learning. However, the movement control system disclosed in the present application is not limited to this, and can be deployed in various ways. It is possible to do so. For example, value-based reinforcement learning other than Q-learning, or furthermore, relationships between state patterns and behaviors determined by methods other than value-based reinforcement learning may be stored in association with a table-format database. It is possible to develop it in various forms, such as determining actions using a database of information.

Furthermore, although the embodiment described above shows a mode in which the route search is performed using the A ^* algorithm, the movement control system disclosed in the present application is not limited to this, and can be developed in various ways. For example, it is possible to use graph search algorithms other than the A ^* algorithm, such as depth-first search, beam search, Bellman-Ford method, and heuristic method.

Furthermore, the division of processing among the mobile device, the movement control device, and the reinforcement learning device shown in the embodiment is merely an example, and can be expanded into various forms. For example, the processing of a movement control device and a reinforcement learning device may be implemented in one server computer, and one of a plurality of movement devices may be used as a movement control device that controls the movement of another movement device. It is possible to develop it into various forms, such as implementing the functions of

S Movement control system 1 Movement device 10 Control section 11 Storage section 110 Program for movement device 13 Position acquisition section 15 Surrounding situation acquisition section 2 Movement control device 20 Control section 21 Storage section 210 Movement control program 3 Machine learning device 30 Control section 31 Storage Part 310 Machine learning program M Learning model

Claims (12)

A movement control system comprising a movement device and controlling movement of the movement device, the system comprising:
a route search means for searching a movement route of the mobile device from a movement start position to a target position;
situation acquisition means for acquiring the surrounding situation of the mobile device;
A unit area set based on the size of the moving device is defined as a neighborhood area, which is an area arranged around the moving device for each unit movement distance serving as a reference for movement of the moving device, and the neighborhood area is defined as a neighborhood area. an entry permission determination unit that applies the surrounding situation acquired by the situation acquisition unit to determine whether or not the mobile device can enter with respect to each unit area included in the nearby area;
Based on the determination result of whether or not each unit area can be entered, which is determined by the entry permission/impossibility determination means, based on the policy of the mobile device that is associated with each state pattern indicating whether or not it is possible to enter each unit area included in the nearby area. policy acquisition means for acquiring a policy associated with a corresponding state pattern;
Movement possibility determining means for determining whether or not movement is possible along the travel route searched by the route searching means based on the policy acquired by the strategy acquisition means;
A movement control system comprising: action determining means for determining to take the action indicated by the policy acquired by the policy acquisition means when it is determined that movement along the movement route is not possible. The movement control system according to claim 1,
comprising a mobile control device capable of communicating with the mobile device,
The mobile devices are plural;
The movement control device includes:
the route search means, the situation acquisition means, the approach permission determination means, the policy acquisition means, the movement permission determination means, and the action determination means;
means for transmitting an action command for each of the mobile devices to perform the action determined by the action determining means to the corresponding mobile device;
The entry permission determining means determines that one of the plurality of mobile devices cannot enter a unit area in which another mobile device is located and/or a unit area in which the other mobile device moves. Features a movement control system. The movement control system according to claim 1,
The mobile device includes:
multiple,
One of the plurality of mobile devices includes:
At least the situation acquisition means, the approach permission determination means, the policy acquisition means, the movement permission determination means, and the action determination means,
The movement control system is characterized in that the entry permission determining means determines that entry is not possible to a unit area in which another mobile device is located and/or a unit area in which the other mobile device moves. The movement control system according to any one of claims 1 to 3,
The route searching means includes:
From the start node corresponding to the movement start position of the mobile device for a graph in which connection relationships are defined with unit areas as nodes and movable paths between unit areas as edges, the range in which the mobile device can move is defined. A movement control system that uses a graph search algorithm to search for a movement route to a goal node corresponding to a target position. The movement control system according to any one of claims 1 to 3,
The movement control system, wherein the nearby area is defined based on movement of two units of unit movement distance. The movement control system according to any one of claims 1 to 3,
A movement control system characterized in that the relationship between the state pattern and the policy acquired by the policy acquisition means is a learning model obtained by reinforcement learning. The movement control system according to claim 6,
The learning model is
For each assumed state pattern, the respective action values of moving and stopping the moving device in the moving direction are shown,
The policy acquisition means includes:
A movement control system characterized in that a movement or stopping in a direction with the highest action value is acquired as a policy. A movement control method using a movement device and a movement control device that controls movement of the movement device, the method comprising:
searching for a movement route from the movement start position of the mobile device to the target position;
obtaining the surrounding situation of the mobile device;
A unit area set based on the size of the moving device is defined as a neighborhood area, which is an area arranged around the moving device for each unit movement distance serving as a reference for movement of the moving device, and the neighborhood area is defined as a neighborhood area. applying surrounding conditions to the area, and determining whether or not the moving device can enter each unit area included in the nearby area;
Based on the policy of the moving device, which is associated with each status pattern indicating whether or not entry into each unit area included in the nearby area is allowed, the mobile device is associated with a state pattern corresponding to a determination result as to whether or not each unit area can be entered. obtaining a policy based on the
a step of determining whether or not movement is possible along the movement route based on the acquired policy;
If it is determined that the moving device is movable, the moving device moves along the moving route;
A movement control method, comprising the steps of: when it is determined that movement is not possible, the movement device takes an action indicated by a policy. A movement control device that controls movement of a movement device,
means for searching a movement route of the mobile device from a movement start position to a target position;
means for acquiring the surrounding situation of the mobile device;
A unit area set based on the size of the moving device is defined as a neighborhood area, which is an area arranged around the moving device for each unit movement distance serving as a reference for movement of the moving device, and the neighborhood area is defined as a neighborhood area. means for determining whether or not the moving device can enter into each unit area included in the nearby area by applying surrounding conditions to the area;
Based on the policy of the moving device, which is associated with each status pattern indicating whether or not entry into each unit area included in the nearby area is allowed, the mobile device is associated with a state pattern corresponding to a determination result as to whether or not each unit area can be entered. a means for obtaining a strategy;
means for determining whether or not movement is possible along the movement route based on the acquired policy;
A means for transmitting, to the mobile device, an action command that causes the mobile device to move along the travel route when it is determined that the movement is possible, and to take the determined action when it is determined that the movement is not possible. Control device. A mobile device that moves along a travel route,
a means of acquiring surrounding conditions;
A unit area set based on the size of the own aircraft is defined as a neighboring area for each unit movement distance that serves as a reference for movement of the own aircraft, and the surrounding situation is applied to the neighboring area. means for determining whether or not the own aircraft can enter with respect to each unit area included in the nearby area;
Based on the policy associated with each state pattern indicating whether entry into each unit area included in the neighborhood area is allowed, obtain the policy associated with the state pattern corresponding to the determination result of whether entry is allowed into each unit area. means and
means for determining whether or not movement is possible along the movement route based on the acquired policy;
means for moving along the movement route when it is determined that movement is possible;
A means for taking an action indicated by an acquired policy when it is determined that movement is impossible. A movement control program that causes a computer equipped with a communication means for communicating with a mobile device to control movement of the mobile device, the program comprising:
to the computer,
searching for a movement route from the movement start position of the mobile device to the target position;
obtaining the surrounding situation of the mobile device;
A unit area set based on the size of the moving device is defined as a neighborhood area, which is an area arranged around the moving device for each unit movement distance serving as a reference for movement of the moving device, and the neighborhood area is defined as a neighborhood area. applying surrounding conditions to the area, and determining whether or not the moving device can enter each unit area included in the nearby area;
Based on the policy of the moving device, which is associated with each status pattern indicating whether or not entry into each unit area included in the nearby area is allowed, the mobile device is associated with a state pattern corresponding to a determination result as to whether or not each unit area can be entered. obtaining a policy based on the
a step of determining whether or not movement is possible along the movement route based on the acquired policy;
If it is determined that the mobile device is movable, the mobile device moves along the travel route, and if it is determined that the mobile device is not movable, it transmits an action command to the mobile device to take the determined action. Movement control program. A mobile device program executed by a mobile device moving along a travel route, the program comprising:
mobile device,
a step of acquiring surrounding conditions;
A unit area set based on the size of the own aircraft is defined as a neighboring area for each unit movement distance that serves as a reference for movement of the own aircraft, and the surrounding situation is applied to the neighboring area. determining whether or not the own aircraft can enter, for each unit area included in the nearby area;
Based on the policy associated with each state pattern indicating whether entry into each unit area included in the neighborhood area is allowed, obtain the policy associated with the state pattern corresponding to the determination result of whether entry is allowed into each unit area. step and
a step of determining whether or not movement is possible along the movement route based on the acquired policy;
a step of moving along the movement route when it is determined that movement is possible;
A program for a mobile device, characterized in that, when it is determined that movement is impossible, the program executes the following steps: taking an action indicated by the acquired policy.

Images