DE102020127407B4 - Method and control device for generating at least one driving order for at least one driverless transport vehicle, FTF, of a modular assembly system - Google Patents

Abstract

Method for generating at least one travel order (17) for at least one driverless transport vehicle, FTF (16), of a modular assembly system (10), wherein the at least one FTF (16) is intended to transport workpieces (14, 15) from a warehouse (13) and/or between different assembly stations (11) in accordance with the respective travel order (17), comprising the following steps: - providing production target data (22) which, as a production target, describe at least the type and number of workpieces (14, 15) to be assembled by means of the modular assembly system (10) in a period extending into the future, in successive time steps (t) respectively: - providing status data (21) which each describe a current system status (St) of the modular assembly system (10), - generating the at least one travel order (17) for the at least one FTF (16) taking into account the production target data and the status data (21), wherein the generation of the at least one travel order (17) is carried out by means of a self-learning Algorithm (25) is carried out which maps the respective current status data (21) by means of an assignment rule (26) to action data sets (19), each of which describes one of the travel orders (17) as an action (At) to be carried out, wherein the assignment rule assigns the action for the next time step to the current system state of the assembly system and the action in the form of a travel order to an AGV causes a change in the system state of the assembly system, wherein the assignment rule (26) is adapted by a learning process (28) on the basis of reward data sets (27), each of which describes a progress (Rt) that was achieved in reaching the production target by the respective travel order (17), characterized in that in the learning process (28) the assignment rule (26) is adapted iteratively in successive learning steps (29) and for the respective learning step (29) several reward data sets (27) describing the respective progress (Rt) from several travel orders (17) are combined and the assignment rule (26) is adapted in the learning step (29) on the basis of the plurality of reward data sets (27), to recognize a control strategy which extends over several driving orders to achieve progress, if with this control strategy only slight progress or even negative progress has to be accepted in between and/or to make the learning process robust against the influence of incorrect decisions and/or to achieve an optimization of the finished products per unit of time.

Description

The invention relates to a method for generating at least one travel order for at least one driverless transport vehicle, FTF, as can be provided in a modular assembly system in order to transport workpieces from a warehouse and/or between assembly stations. The invention also relates to a control device for carrying out the method.

In a modular assembly system, workpieces are transported from one assembly location to the next using driverless transport vehicles (FTVs) of a driverless transport system. The flexible controllability of such FTFs allows different assembly sequences to be implemented for different products.

Workpieces are usually transported by driverless transport vehicles, each of which can be designed as a railless robot or a rail vehicle, for example. A workpiece can be an initial component from a warehouse or a product that has already been partially assembled.

To control an automated guided vehicle system with automated guided vehicles, it is important that the workpieces are always transported from the warehouse or from one assembly location to the next assembly location in a timely manner so that each assembly station is always supplied with workpieces in order to be able to continue its operation without interruption. Due to the different assembly sequences mentioned above, the transport does not take place in the same order as when the workpieces are placed at the start of assembly, but rather chaotically, so that a simple assembly line solution is not possible. This "chaos" is also caused by the fact that it regularly happens that several workpiece carriers have the same assembly station as their destination at the same time, which can lead to waiting times. This also requires a control system for the automated guided vehicles.

A workpiece can be held or housed in a workpiece carrier in order to maintain mechanical compatibility with the driverless transport vehicle.

Currently, an assembly system is controlled using relatively simple so-called pull principles. A workpiece carrier with the workpiece housed in it is always brought to a subsequent station if this assembly station has space to accommodate the workpiece carrier. There is no comprehensive coordination between several workpiece carriers. This means that the optimum control is not achieved and the aforementioned chaos is not globally resolved.

In order to contain this chaos, i.e. to optimize the workpiece transport in relation to the output of the overall system, one can work with a schedule that takes travel times, processing times and the occupancy of the assembly stations into account. Such schedules can be optimized using mathematical optimization methods or genetic algorithms, for example, so that the overall schedule is as short as possible in terms of time, i.e. as many finished products as possible are produced in one unit of time. To do this, it must always be possible to look ahead over a certain period of time. However, calculating or designing such a schedule is very complex and inflexible when it comes to adapting an assembly sequence or even combining several different assembly sequences for different products. In addition, a schedule calculation is required for each assembly system individually, so that this approach cannot achieve generalizability in order to enable the control of different assembly systems with a universal method.

Artificial intelligence in the form of self-learning machine learning algorithms can also be used to control assembly systems.

From the website https://robinaut.net/optimierte-produktionsplanung-dankkuenstlicher-intelligenz/ it is known that for production planning in an assembly system, the sequence can be optimized using machine learning by applying so-called reinforcement learning.

From the publication by Illhoe Hwang and Young Jae Jang, “Q(λ) learning-based dynamic route guidance algorithm for overhead hoist transport systems in semiconductor fabs”, https://doi.org/10.1080/00207543.2019.1614692, it is known that a control system for rail-bound transport equipment can be optimized on the basis of machine learning. The artificial neural network used for this purpose can first be pre-trained in a simulation by receiving a control decision of the network as input from the simulator and determining the resulting change in state of the assembly system in a simulation, whereupon the resulting new system state and an evaluation of this state are provided as feedback for the learning algorithm.

From the website https://www.fraunhofer.at/de/referenzen/reinforcement_learning.html it is known that production planning can be optimized using artificial intelligence, whereby so-called deep reinforcement learning is applied and the training takes place in a simulation. The aim here is to be able to identify potential bottlenecks in an assembly process at an early stage.

The EP 3 696 744 A1 describes a control system and a method for controlling technical units, such as robots or autonomous vehicles, in order to use these technical units to carry out tasks in a common environment.

The EN 20 2020 104 238 U1 describes a device for learning a strategy which optimally adapts at least one parameter of an evolutionary algorithm, in particular a CMA-ES or Differential Evolution Algorithm.

The WO 2019/228848 A1 describes an active traffic management system that includes sensors in the road network and output devices for controlling traffic. Decisions of the traffic management system are made on the basis of determined future states, whereby these future states are determined using a model based on a neural network.

The EN 10 2019 107 462 A1 describes a system, a method and a cloud-based server for the automated checking of a functional status of components of a transport network in an intralogistics storage and/or picking system.

The invention is based on the object of controlling driverless transport vehicles (FTVs) in an assembly system.

The object is achieved by the subject matter of the independent patent claims. Advantageous embodiments of the invention result from the dependent patent claims, the following description and the figures.

According to the invention, the control is based on a self-learning algorithm, i.e. machine learning, by means of which a so-called agent is trained to carry out actions in time steps in order to influence its environment, in this case the assembly system. The agent applies a so-called policy to assign an action for the next magazine to the current system state of the assembly system. The policy is therefore referred to here as a mapping rule or assignment rule. In the present case, the respective action is carried out in the form of a travel order to an AGV. The action results in a new system state of the assembly system, which is reported back to the agent together with an assessment of the new system state (so-called reward). From this, the assignment rule can be adapted or optimized in a manner known per se using a self-learning algorithm. The assembly system can carry out a manufacturing process or production process with a production goal. The production goal can provide for the manufacture of products of a predetermined type or several different types, as well as an associated number of products in each case. The control system should achieve an optimization in terms of throughput (finished products per unit of time).

• - Bereitstellen von Produktionszieldaten, welche als Produktionsziel wenigstens Art und Anzahl von mittels des modularen Montagesystems in einem in die Zukunft reichenden Zeitraum zu montierenden Werkstücken oder fertig zu montierenden Produkten beschreiben,

in aufeinanderfolgenden Zeitschritten t jeweils:

• - Bereitstellen von Zustandsdaten, welche jeweils den aktuellen Systemzustand (St) des modularen Montagesystems beschreiben (z.B. Fahrzeugpositionen und Zustand der Montagestationen),
• - Erzeugung des wenigstens einen Fahrauftrags für das wenigstens eine FTF unter Berücksichtigung der Zieldaten und der Zustandsdaten, wobei die Erzeugung des wenigstens einen Fahrauftrags mittels eines selbstlernenden Algorithmus erfolgt, welcher die jeweils aktuellen Zustandsdaten mittels einer Zuordnungsregel (Policy) auf Aktionsdatensätze abbildet, von denen jeder einen der Fahraufträge als durchzuführende Aktion (At) angibt, und durch einen Lernprozess die Zuordnungsregel anhand von Belohnungsdatensätzen angepasst wird, die jeweils einen fahrauftragsbezogenen Fortschritt (Reward Rt) beim Erreichen des Produktionsziels beschreiben.

The invention provides a method for generating at least one travel order (action) for at least one driverless transport vehicle, FTF, of a modular assembly system, wherein the at least one FTF is intended to transport or move workpieces from a warehouse and/or between different assembly stations in accordance with the respective travel order. The method provides the following steps:

• - Providing production target data which, as a production target, describes at least the type and number of workpieces or fully assembled products to be assembled using the modular assembly system in a period extending into the future,

in successive time steps t:

• - Provision of status data which describe the current system status (St) of the modular assembly system (e.g. vehicle positions and status of the assembly stations),
• - Generation of the at least one driving order for the at least one AGV taking into account the target data and the status data, wherein the generation of the at least one driving order takes place by means of a self-learning algorithm which maps the respective current status data by means of an assignment rule (policy) to action data sets, each of which specifies one of the driving orders as the action to be carried out (At), and through a learning process the assignment rule is adapted on the basis of reward data sets, each of which describes a driving order-related progress (reward Rt) in reaching the production target.

The term “modular” refers to an assembly system in which the individual assemblies stations can be freely combined. An example of such an assembly station can be, for example, a production machine for drilling or soldering a workpiece or for assembling several workpieces. The workpieces are transported between these individual assembly stations in order to be ready for the subsequent production step at another assembly station. The status data describes the overall current system status. The system status can indicate the respective position of the at least one AGV and/or a supply status of each assembly station with waiting workpieces and/or the number of completed workpieces to be transported away per assembly station. In the successive time steps, the self-learning algorithm can issue an action in the form of a travel order in response to the current system status, which causes a change in the system status, namely a change in a vehicle position of an AGV and the resulting progress in production or manufacturing. This means that there is also a change in the respective status of at least one assembly station because a workpiece has been provided on the input side and/or an assembled workpiece has been transported away on the output side.

The self-learning algorithm can implement the assignment rule (policy) through a so-called agent, e.g. through an artificial neural network. In the case of an artificial neural network, the assignment rule results from the values of the weights in the connections between the artificial neurons.

The progress can be described in the reward data set as a number of completed products and/or as the time elapsed during the driving task, whereby the latter information can be specified as undesirable effort, i.e. as a negative value for the progress. In order to maximize the throughput, i.e. to achieve as much progress as possible in as little time as possible, the assignment rule must be optimized or adapted in the learning process. It is known that after each driving task, the system state is used together with the reward data set resulting from the driving task, i.e. the progress or reward at the current time step t, for a learning step. However, this makes the learning process vulnerable in that a single driving task and the resulting progress can cause a change in the entire assignment rule, which makes convergence difficult due to the complexity of the control task. In addition, there is the disadvantage that a control strategy that extends over several driving tasks in order to achieve progress cannot be recognized if this control strategy only results in minor progress or even disadvantages (negative progress) in between.

According to the invention, the assignment rule is therefore also adapted iteratively in successive learning steps in the learning process, but in this case several reward data sets that indicate the progress (Rt) from several driving orders are summarized for each learning step, and the assignment rule is thus adapted for each learning step on the basis of several rewards or reward data sets. For example, reward data sets from 5 to 200 actions or driving orders can be summarized. This has the advantage that a change in the assignment rule based on the self-learning algorithm always includes a large number of driving orders and the resulting reward data sets. This makes the learning process robust against the influence of individual incorrect decisions or setbacks (negative progress).

The invention also includes embodiments which provide additional advantages.

In order to be able to apply the method in a generalized manner, i.e. to different assembly systems and without prior knowledge of a possible optimal production process, the assignment rule should be able to be started with standard values and from there the assignment rule can be adapted to the assembly system. To do this, learning can first take place in an initial learning process on a simulation of the assembly system. One embodiment includes operating a mathematical model of the modular assembly system, based on which simulation data is generated and used for an initial learning process of the self-learning algorithm by operating the assembly system as a simulation. Preferably, it can be provided that the assembly system is simulated several times in parallel, for example in two to ten simulation environments. A different driving order can thus be generated for each simulated assembly system and several reward data sets with rewards for the next learning step arise from the simulated assembly systems. One embodiment accordingly includes determining reward data sets for the respective learning step from identical assembly systems operated in parallel, in each of which at least one driving order is carried out per learning step.

In order to be able to optimize a predictive control strategy that includes several driving orders generated one after the other in an assembly system, a learning step can also be based on several consecutive time steps. One embodiment includes the fact that for the respective learning step in the same assembly system, several driving orders are generated one after the other and a tuple is formed in each case, which specifies the current state data, the resulting action data set of the driving order and the resulting reward data set, and the series of tuples is passed on to the learning process as a trajectory. Each tuple therefore describes the initial system state, the action generated thereafter (driving order) and the resulting progress (reward). The tuples generated one after the other in this way therefore result in one reward data set per tuple, so that several reward data sets are also available for a learning step.

A learning step can start from an error value from which, for example, a gradient for a change in the assignment rule, in particular a change in the weighting values in an artificial neural network, can be calculated using so-called back propagation. In order to determine such an error value from several reward data sets, the mean value of the reward data sets or at least some of them can advantageously be used as a basis. One embodiment accordingly includes calculating a mean value of at least some of the reward data sets for the respective learning step that result from the driving orders on which the learning step is based. If the trajectories described are used as a basis for a learning step, the mean value for the trajectories collected in parallel can therefore be calculated in parallel simulation environments. Thus, despite the use of several reward data sets, a self-learning algorithm from the prior art can be used.

One embodiment includes the self-learning algorithm being implemented by means of at least one reinforcement learning, in particular by means of a proximal policy optimization, PPO. By means of reinforcement learning, an agent that contains or implements the assignment rule can be efficiently trained based on reward data sets. The PPO has the additional advantage that, despite the large number of possible system states and the large number of possible driving orders, convergence can still be achieved when training or learning the assignment rule.

One embodiment includes limiting the self-learning algorithm in terms of its learning rate using a Kullback-Leibler boundary condition. Limiting the learning rate prevents the change in the assignment rule from being greater than a predetermined maximum value in a single learning step. This prevents an improvement in the assignment rule from being undone in a single learning step if an incorrect decision was made.

Step-by-step learning in individual learning steps usually means that the data obtained (state data and reward data sets) are only used once for a learning step and then discarded. If the learning rate is limited in the manner described, a large number of learning steps may be necessary for complete training. To avoid this, one embodiment provides that the assignment rule is adapted several times in succession for each learning step by repeatedly using the current reward data sets. This has the advantage that the same data (state data and reward data sets) with an actively reduced learning rate can still lead to a greater change in the assignment rule than a single adjustment of the assignment rule, without there being a risk that the assignment rule will degenerate in the manner described due to a learning rate that is too high.

One embodiment includes that the respective driving order is determined by selecting from a predetermined set of a finite number of possible driving orders. An example of such a driving order can be: "Drive from A to B". A limited number of possible actions from which one can choose advantageously results in a limitation of the state space, for example by specifying the number of possible driving orders to be less than 1000, in particular less than 200. This has the advantage that the assignment rule can converge reliably.

A particular problem arises when there are several identical machines or assembly stations in an assembly system, so that a workpiece can be transported to one of several possible assembly stations. This has the advantage that the allocation rule is not optimized to control the assembly station with the shortest waiting time, but rather the cross-system optimization goal of maximizing throughput is aimed for. One embodiment includes that several assembly stations are provided in the assembly system for the same production step and an AGV is controlled by means of the allocation rule to the assembly station that currently results in the shortest waiting time and/or the highest throughput in the production process.

The optimization goal of "throughput maximization" must be mapped numerically into a calculation formula, by means of which the reward data sets can be calculated in order to influence the change in the allocation rule in each learning step. The invention has found that a bonus-malus system enables throughput optimization as a learning goal for the learning process. One embodiment includes the fact that, in order to generate the reward data sets, a reward value is determined which provides a predetermined malus value for a respective past time step and/or a predetermined bonus value for the completion of a workpiece and/or a predetermined malus value for a predetermined action that requires revision. Since the passage of time or the length of time conflicts with the optimization goal, the duration of a time step can be included as a malus value, i.e. a negative value, for example. For example, a malus value of -13 seconds, for example, can be used as a basis for each past time step, depending on the assembly system. If a workpiece is fully assembled, a finished product of a certain type is created, and this can be given a bonus value, for example +50. An action that requires revision could be, for example, if an AGV is maneuvered into a dead end without anything being done there; instead, the maneuver must be reversed. Such an action that requires revision can be given a penalty value of, for example, -100. The resulting value for each driving order can then be provided as a reward data set. The more time passes, the worse the current assignment rule is rated. The advantageous learning process is that the assignment rule loses as little time as possible and workpieces are completed without generating actions that require revision.

In order to be able to adapt to different assembly systems, the respective learning process for an assembly system should be able to take its equipment into account. Depending on the characteristics of the assembly system, for example, the number of so-called hidden layers in an artificial neural network and/or the number of artificial neurons in individual layers of the network can be determined for a self-learning algorithm. One embodiment includes setting at least one metaparameter of the self-learning algorithm depending on at least one characteristic of the assembly system, in particular a number of assembly stations.

The invention also provides a control device for at least one driverless transport vehicle, FTF, of a modular assembly system, wherein the control device has a processor circuit that is set up to carry out an embodiment of the method according to the invention. The control device can have a data processing device or a processor device that is set up to carry out an embodiment of the method according to the invention. For this purpose, the processor device can have at least one microprocessor and/or at least one microcontroller and/or at least one FPGA (Field Programmable Gate Array) and/or at least one DSP (Digital Signal Processor). Furthermore, the processor device can have program code that is set up to carry out the embodiment of the method according to the invention when executed by the processor device. The program code can be stored in a data memory of the processor device.

The invention also includes combinations of the features of the described embodiments. The invention therefore also includes implementations that each have a combination of the features of several of the described embodiments, provided that the embodiments have not been described as mutually exclusive.

• 1 eine schematische Darstellung eines Montagesystems mit einer Ausführungsform der erfindungsgemäßen Steuervorrichtung;
• 2 eine Skizze zur Veranschaulichung einer Ausführungsform des erfindungsgemäßen Verfahrens;
• 3 eine Skizze zur Veranschaulichung eines selbstlernenden Algorithmus auf Basis eines künstlichen neuronalen Netzes.

Exemplary embodiments of the invention are described below.

• 1 a schematic representation of an assembly system with an embodiment of the control device according to the invention;
• 2 a sketch to illustrate an embodiment of the method according to the invention;
• 3 a sketch to illustrate a self-learning algorithm based on an artificial neural network.

The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the components of the embodiments described each represent individual features of the invention that are to be considered independently of one another and which also develop the invention independently of one another. Therefore, the disclosure should also include combinations of the features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by other features of the invention already described.

In the figures, identical reference symbols designate functionally identical elements.

1 shows a production system or assembly system 10 in which assembly stations 11 can be operated as individual modules for carrying out a production process or manufacturing process of products 12. The products 12 can be of different types. Components 14 can be stored as starting material in a warehouse 13, which can be fed to several assembly stations 11 as starting workpieces so that partially assembled workpieces 15 can be manufactured from them, which can then be taken to another assembly station 11 where they can be further processed, for example by adding another component 14 and/or making a change.

To transport or move the components 14 and/or the partially assembled workpieces 15, driverless transport vehicles (FTFs) 16 can be provided in the assembly system 10, which can travel between the warehouse 13 and/or the different assembly stations 11. These can be railless or rail-bound FTFs. A respective journey to transport a workpiece carrier, in which a component 14 and/or a partially assembled workpiece 15 can be located, is commanded by a travel order 17, which can be generated by a control device 18. For this purpose, the control device 18 can generate action data records 19, each of which specifies a travel order 17 for a respective FTF 16. In a manner known per se, a current system state 20 of the assembly system 10 can be described by state data 21, which can be recorded in the assembly system 10 in a manner known per se.

Controlling the AGV 16 by the control device 18 enables a production target to be achieved, which can be specified by production target data 22 and can describe a number and a type of products 12 that are to be produced by the assembly system 10. The control device 18 can be optimized with respect to a throughput 23 of products 12. For this purpose, the control device 18 can have a processor device 24 by which a self-learning algorithm 25 can be operated, which maps the respective current status data 21 to a respective action data set 19 by means of an assignment specification or assignment rule 26 in order to control the AGV 16 in successive time steps. The assignment rule 26 can be implemented as an artificial neural network. In order to optimize the throughput 23 as an optimization goal, a learning process can be carried out in the self-learning algorithm 25, i.e. a training phase can be carried out beforehand. For this training phase in the learning process, in addition to the state data 21 and the action data set 19, the respective progress of the production process, as it results for each driving order 17 individually, can be described by a respective reward data set 27.

2 illustrates the learning process 28 in detail. Shown is a learning step 29 which the self-learning algorithm 25 can carry out for the assignment rule 26 for a time step from time t to t+1 (where “+1” can stand for a predetermined period of time, e.g. 13 s). The state data 21 for the system state St and the resulting reward data set 27 of the reward R _t+1 based on the last action At or driving task can be provided from the assembly system 10. In an initial learning phase, this can be done as part of a simulation, so that in addition to the assembly system, which can be simulated in the simulation by a corresponding model 30, at least one further instance or simulation environment can be operated on the basis of the model 30. This results in several system states St and corresponding reward data sets 27 with a respective value for a reward R _t+1 per time step. For the learning step 29, in addition to the described data of the current time step t per operated or simulated assembly system 10, a trajectory 31 can also be provided in which the data from several time steps t, t-1, t-2, t-3,... can be summarized (e.g. the last 4 to 10 time steps).

The learning step 29 can then be carried out, which can be carried out on the basis of reinforcement learning. In particular, the described PPO can be applied. The adapted or modified assignment rule 26 can then continue to control the AGV 16, which again results in new data for the next trajectory 31.

3 illustrates the learning step 29. It is shown how the assignment rule 26 can be formed as a neural network 33 in a manner known per se with a layer structure or layer structure made up of layers 34 of artificial neurons 35 (with input layer 37 and output layer 38). For the sake of clarity, only some of the artificial neurons 35 are provided with a reference symbol. The neurons 35 can be linked to one another by weightings 36 in a manner known per se. The number of layers 34 and/or the number of neurons 35 per layer 34 can be determined by metaparameters 39, which in turn can be selected depending on the equipment of the assembly system 10. By The reinforcement learning 32 allows the weightings 36 to be adjusted by the described evaluation of the progress in such a way that the assignment rule 26 moves towards the optimization goal of increased throughput 23 (see 1 ) is changed.

The state data 21 of the system state St contained in the respective tuple (St, At, Rt) have led to an action data set 19 for an action At in the associated time step t, from which a reward data set 27 of the reward Rt followed (and correspondingly for t-1, t-2, t-3,...). Now, an average value 27` of the individual progress values can be calculated from the reward data sets 27, which can be used as error or feedback in the reinforcement learning 32 in order to carry out a back propagation or error propagation through the neural network 33 of the assignment rule 26 in a manner known per se. This can also be done for the reward data sets 27 from several trajectories 31.

This results in a method as a generalizable control solution in which the control device 18 implements an agent with an assignment rule 26 by means of reinforcement learning. This agent learns, without knowing beforehand what good control decisions look like in a given assembly system or production system, by controlling the production system through trial and error and experiences based on this (evaluations / reward Rt). This learning process is in 2 illustrated.

The reinforcement learning principle is applied here in such a way that the task of learning consists in finding a suitable allocation rule 26 according to the trial-and-error principle, which optimizes the throughput 23. The learning computer program is referred to as the "agent" here. This agent is the decision maker who interacts with its environment, here the at least one assembly system 10, and makes decisions for this environment. The environment is everything outside the agent and therefore everything over which the agent does not have absolute control. The environment is represented by a series of variables that are related to the problem to be solved. This set of variables and all possible values that they can assume are referred to as the state space. A state St in a time step is an instantiation of the state space, a set of values that the variables assume at the current point in time. In each state, the agent has a set of actions At from which it can choose. Through these actions, the agent influences the environment. In response to the agent's action, the state of the assembly system changes to S _t+1 . The environment also provides a reward signal R _t+1 to the agent, which represents the quality of the action and thus the agent's decision. The interactions between the agents and the environment last for several cycles. Each cycle is called a time step t. At each time step, the agent observes the environment as described, takes an action and receives a reward. In this way, the agent gains experience that gives it the opportunity to learn and improve performance.

Generalizability: This principle can be applied to implement a Cornel algorithm that has the ability to learn the control of modular assembly systems. This reduces the essential implementation effort to a single time, whereby only general parameters (hyperparameter 36) have to be readjusted for newly controlled or modified modular assembly systems and a lot of implementation effort is saved compared to other control solutions.

Dealing with complexity: The complexity of the modular assembly systems is dealt with by a deep neural network. This stores experiences gained through trial and error and learns on the basis of these experiences.

1. 1. Modell: Das Modell des modularen Montagesystems kann beispielsweise in Tecnomatix Plant Simulation® oder einer anderen Simulationsumgebung erstellt werden. Es dient als Umgebung für den Agenten und sendet die entsprechenden Aktionsanfragen, die der Agent beantworten muss und aus denen er lernt.
2. 2. State: Der Zustand/State St, der bei einer Aktionsanfrage von der Umgebung zur Verfügung gestellt wird, ist eine Liste von Elementen, beispielsweise dargestellt als: $${\text{S}}_{\text{t}}=\left[\text{0 10 0311 1320}\right]$$
   

To implement the idea, a reinforcement learning agent needs to be implemented that can learn, based on a model of the modular assembly system (environment), the associated state representation and the corresponding reward system, how optimal decisions are to be made in the given production system/assembly system depending on the system state. A predefined number of actions can be considered as decisions (example: vehicle 1, drive to station B). Overall, for example, the production span (duration of processing all production orders) is minimized.

1. 1. Model: The model of the modular assembly system can be created, for example, in Tecnomatix Plant Simulation® or another simulation environment. It serves as the environment for the agent and sends the corresponding action requests that the agent must answer and from which it learns.
2. 2. State: The state/State St provided by the environment in response to an action request is a list of elements, for example represented as: $${\text{S}}_{\text{t}}=\left[\text{0 10 0311 1320}\right]$$
   

1. 3. Aktion A_t: Als Aktion stehen alle in der Umgebung möglichen Aktionen der FTF zur Verfügung. Beispielsweise die Aktion: „FTF 1, fahre zu Station B“. Dadurch werden die Produktionssteuerungsentscheidungen getroffen.
2. 4. Reward R_t: Auf Basis der gewählten Aktionen bekommt der Agent einen Reward von der Umgebung zurück, den er zum Lernen nutzt. Er sammelt positive wie negative Rewards für Erfahrungen. Eine Belohnung bekommt er beispielsweise, wenn er die ganze Produktion abschließt (zum Beispiel Tagesprogramm eines Produktionstages). Außerdem wird als Reward die Zeit der Durchführung der gewählten Aktion als Maluswert zurückgegeben. So können Durchlaufzeiten minimiert werden.

The State St can be coded in such a way that the current production status is fully represented. For example, the first bit could mean that 0 orders have been processed so far. The following two bits could represent the position of an AGV by storing a 1 if the AGV is in this position. The following two four-bit packets, for example, represent two different orders in the system and contain information about the product type (kind), the processing history and the current position of the order in the system.

1. 3. Action A _t : All actions possible for the AGV in the environment are available as an action. For example, the action: “AGV 1, drive to station B”. This is how the production control decisions are made.
2. 4. Reward R _t : Based on the selected actions, the agent receives a reward from the environment, which it uses to learn. It collects positive and negative rewards for experiences. For example, it receives a reward when it completes the entire production (for example, the daily program for a production day). In addition, the time taken to carry out the selected action is returned as a penalty value. In this way, throughput times can be minimized.

Experience: Proximal Policy Optimization (PPO) is the preferred deep reinforcement learning method. In PPO, the collected experiences are stored in a neural network (NN), which in 3 The effects of the processing of a respective travel order on the system state of the modular assembly system and/or the assembly progress are then automatically evaluated and, together with the evaluation, made available to an iterative learning process of the self-learning algorithm, whose optimization criteria are continuously adapted through this learning process.

The state representation (state data) is fed into the input layer 37. The output or softmax layer 38 represents the potential actions in the production system/assembly system. The connections between the neurons or nodes are the weights of the neural network, which are learned and updated through experience. In this way, for a defined input/state, the best action for this state is gradually learned.

Transfer to reality: If the control system has been trained using simulation, it can be transferred to reality. There it can continue to learn according to the same principle and thus become even more similar to reality. The prerequisite for achieving this transfer is, in particular, to make the simulation model used above as realistic as possible.

Overall, this results in a method for learning the control using deep reinforcement learning.

In the simulation model, a real production system/assembly system is preferably first simulated (depending on the application). A reinforcement learning agent is configured, which is designed using the agent's hyperparameters. The resulting control agent, i.e. its assignment rule, is then transferred to reality or used there.

The advantages are a holistic optimization of the assembly system, a generalizable solution, which provides flexibility in transferring to other assembly systems, as well as time savings through application to the various assembly systems, which results in cost savings (both in so-called greenfield and brownfield applications).

Overall, the examples show how generalizable control of modular assembly systems can be provided using deep reinforcement learning (DRL).

Claims (13)

Method for generating at least one travel order (17) for at least one driverless transport vehicle, FTF (16), of a modular assembly system (10), wherein the at least one FTF (16) is intended to transport workpieces (14, 15) from a warehouse (13) and/or between different assembly stations (11) in accordance with the respective travel order (17), comprising the following steps: - providing production target data (22) which, as a production target, describe at least the type and number of workpieces (14, 15) to be assembled by means of the modular assembly system (10) in a period extending into the future, in successive time steps (t) respectively: - providing status data (21) which each describe a current system status (St) of the modular assembly system (10), - generating the at least one travel order (17) for the at least one FTF (16) taking into account the production target data and the status data (21), wherein the generation of the at least one travel order (17) is carried out by means of a self-learning Algorithm (25) which assigns the current status data (21) to action data sets by means of an assignment rule (26) (19), each of which describes one of the travel orders (17) as an action (At) to be carried out, wherein the assignment rule assigns the action for the next time step to the current system state of the assembly system and the action in the form of a travel order to an AGV causes a change in the system state of the assembly system, wherein a learning process (28) is used to adapt the assignment rule (26) on the basis of reward data sets (27), each of which describes a progress (Rt) that was achieved in reaching the production target by the respective travel order (17), characterized in that in the learning process (28) the assignment rule (26) is adapted iteratively in successive learning steps (29) and for the respective learning step (29) a plurality of reward data sets (27) describing the respective progress (Rt) from a plurality of travel orders (17) are combined and the assignment rule (26) is adapted in the learning step (29) on the basis of the plurality of reward data sets (27), in order to recognize a Control strategy that extends over several driving orders to achieve progress, if this control strategy requires only minor progress or even negative progress to be accepted in between and/or to make the learning process robust against the influence of incorrect decisions and/or to achieve an optimization of the finished products per unit of time. Procedure according to Claim 1 , characterized in that a mathematical model (30) of the modular assembly system (10) is operated, on the basis of which simulation data are generated and used for an initial learning process (28) of the self-learning algorithm (25) by operating the assembly system (10) as a simulation. Method according to one of the preceding claims, characterized in that for the respective learning step (29) reward data sets (27) are determined from parallel operated, identical assembly systems (10), in which at least one driving order (17) is carried out per learning step (29). Method according to one of the preceding claims, characterized in that for the respective learning step (29) in the same assembly system (10) several driving orders are generated one after the other and a tuple is formed in each case, which indicates the current state data (21), the action data set (19) of the driving order (17) formed therefrom and the resulting reward data set (27), and the series of tuples is transferred to the learning process (28) as a trajectory (31). Method according to one of the preceding claims, characterized in that for the respective learning step (29) a mean value (27`) of at least some of the reward data sets (27) is calculated which result for the driving orders on which the learning step (29) is based. Method according to one of the preceding claims, characterized in that the self-learning algorithm (25) is realized by means of at least one reinforcement learning, in particular by means of a proximal policy optimization, PPO. Method according to one of the preceding claims, characterized in that the self-learning algorithm (25) is limited in terms of its learning rate by means of a Kullback-Leibler boundary condition. Method according to one of the preceding claims, characterized in that for each learning step (29) the assignment rule (26) is adapted several times in succession by repeatedly using the current reward data sets (27). Method according to one of the preceding claims, characterized in that the respective driving order (17) is determined by selecting from a predetermined set of a finite number of possible driving orders (17). Method according to one of the preceding claims, characterized in that several assembly stations (11) are provided in the assembly system (10) for a same production step and by means of the allocation rule (26) an AGV (16) is controlled to the assembly station (11) with the currently shortest waiting time and/or, independently of the waiting time, to the assembly station (11) for which a throughput maximization results. Method according to one of the preceding claims, characterized in that for generating the reward data sets (27) a reward value is determined which provides a predetermined penalty value for a respective past time step (t) and/or a predetermined bonus value for the completion of a workpiece (14, 15) and/or a predetermined penalty value for a predetermined action requiring revision. Method according to one of the preceding claims, characterized in that at least one metaparameter (39) of the self-learning Algorithm (25) is set depending on at least one characteristic value of the assembly system (10), in particular a number of assembly stations (11). Control device (18) for at least one driverless transport vehicle, FTF (16), of a modular assembly system (10), wherein the control device (18) has a processor circuit which is configured to carry out a method according to one of the preceding claims.

Images