DE102018214828B4 - Method, device and computer program for predicting a location of a plurality of objects - Google Patents

Abstract

Method (30) for predicting a location, in particular a movement path, of a plurality of objects (P1, P2), wherein a map (20) which comprises at least one action area of the objects (P1, P2) and is divided into a plurality of cells (21) is provided (31), wherein each cell (21) is assigned a first size (22) which characterizes, in particular a probability (P(Oc)), whether at least one of the objects (P1, P2) is located within the respective cell, wherein a plurality of fictitious objects are placed (32) on the map (20) and the fictitious objects are each assigned a weight depending on the first size (22) which is assigned (33) to the cell within which the respective fictitious object is placed, wherein a machine learning system determines a plurality of different discrete speeds depending on the map (20), wherein a new position (r') of the fictitious objects is determined depending on a selected discrete speed the majority of the different discrete speeds is determined and the fictitious objects are placed (34) on the map (20) based on the new position (r'), wherein the first size (22) of the cells is determined (35) depending on the weights of those fictitious objects placed within the respective cell.

Description

Technical area

The invention relates to a method for predicting a location of a plurality of objects by means of a machine learning system. The invention also relates to a device and a computer program, each of which is configured to carry out the method.

State of the art

The unpublished EN 10 2017 217 412 A1 discloses methods for operating a robot control system which includes a machine learning system. The machine learning system determines a movement path of at least one object in the robot's action space depending on a map which represents an action space of the robot.

The unpublished EN 10 2018 201 570 A1 discloses a multiple target object tracking method. First, a speed distribution is determined using a speed transition distribution. Then, transition probability information is calculated that indicates a probability of objects reaching the possible neighboring grid cells. Then, an occupancy probability is determined for each grid cell based on the transition probability information.

EN 10 2013 223 803 A1 relates to a method for segmenting an occupancy grid for an environment model of a driver assistance system for a vehicle.

EN 10 2013 203 239 A1 discloses a method for predicting the position of an object in the environment of a vehicle.

Disclosure of the invention

In a first aspect, a method, in particular a computer-implemented method, is presented for predicting a location, in particular a movement path, of a plurality of objects. First, a map is provided which comprises an action space of the objects and is divided into a plurality of cells. Each cell is assigned a first size which characterizes, in particular a probability, whether at least one of the objects is located within the respective cell. A plurality of fictitious objects are placed on the map and each fictitious object is assigned a weight depending on the first size assigned to the cell within which the respective fictitious object is placed. A machine learning system then determines a plurality of different discrete speeds, in particular for each cell, depending on the map provided, in particular a geometry and/or outline and/or shape of the action space. A new position of the fictitious objects is then determined depending on a selected discrete speed of the plurality of different discrete speeds and the fictitious objects are placed on the map based on the new position. The first size of the cells is determined depending on the weights of the fictitious objects placed within the respective cell.

The method has the advantage that it can be carried out with any and unknown number of objects and their locations can be predicted simultaneously, since the respective objects do not have to be assigned to a data point, such as a cell or a fictitious object. The method also has the advantage that knowledge learned by the machine learning system about the interaction of the objects with the environment can be used for prediction. For example, a movement of the objects can be derived from the shape of the action space using the machine learning system, which is then used to determine the new positions of the fictitious objects. Predicting the location means that the location is predicted at a subsequent point in time.

It is proposed that, in particular, randomly and repeatedly, the fictitious objects are selected depending on the assigned weights and the selected fictitious objects are placed within the respective cell within which the selected fictitious objects are placed.

The advantage of this is that by selecting the fictitious objects depending on their weights, the fictitious objects are distributed on the map in the way that the objects are expected to be arranged. This allows a resource-efficient estimation of the current arrangement of the objects. Furthermore, this procedure has the advantage that several modalities of a non-deterministic dynamic can be mapped. Therefore, it is not necessary to use the fictitious objects to decide which path to take, e.g. at an intersection, but the fictitious objects are distributed along the possible paths according to probabilities and random initialization. Preferably, the selection is realized by drawing and replacing according to an urn model, where the probabilities that the fictitious object is pulled depends on the weights of the fictitious objects.

It is further proposed that the machine learning system determines a plurality of probabilities for the different discrete speeds, in particular for each discrete speed of the fictitious objects of a previous point in time.

Furthermore, it is proposed that the first sizes of the cells are determined depending on a product of the weights of the fictitious objects placed within the respective cell.

The advantage here is that this approach can be used to determine the first variables after placing the fictitious objects in their new positions in a computationally and time-efficient manner, whereby the non-deterministic behavior of real objects is also modeled.

Furthermore, it is proposed that a measurement, in particular a detection of the objects, is carried out, wherein, depending on the measurement, a plurality of the first sizes of the cells are determined, in particular adapted.

The advantage is that the measurement incorporates information from the environment, which means that the initial values can be adjusted depending on the measurement. The measurement can have a corrective effect on the process and thus increase the accuracy of the process. Since the process can be randomly initialized, the real arrangement of the objects can be determined more reliably based on the measurements.

It is further proposed that the machine learning system is a convolutional neural network. The machine learning system determines a plurality of the different discrete speeds, in particular a speed distribution, for each cell.

A speed distribution means that the different discrete speeds are each assigned a probability that indicates how likely it is that an object within this cell has the respective discrete speed. The advantage here is that the selected speeds for the fictitious objects can be precisely determined and the robustness of the method can be increased.

It is further proposed that a control variable for controlling a robot is determined depending on the first sizes of the cells. Alternatively, the control variable can be used to control an actuator of a technical system. The technical system can be, for example, an at least partially autonomous machine, an at least partially autonomous vehicle or a flying object such as a drone.

In a further aspect, a computer program is proposed. The computer program is designed to carry out one of the previously mentioned methods. The computer program comprises instructions that cause a computer to carry out one of these mentioned methods with all its steps when the computer program is running on the computer. Furthermore, a machine-readable memory module is proposed on which the computer program is stored. Furthermore, a device is proposed that is designed to carry out one of the methods.

Embodiments of the above-mentioned aspects are shown in the accompanying drawings and explained in more detail in the following description. Shown are:

Short description of the drawings

• 1 a schematic representation of a robot;
• 2A a schematic representation of a map of the room in which the robot and two objects are located;
• 2 B a schematic representation of the map of the room, with the map divided into cells;
• 3 a schematic representation of an embodiment of the method for predicting a location of objects;

1 shows a schematic representation of an at least partially autonomous robot, which in a first embodiment is provided by an at least partially autonomous vehicle (10). In a further embodiment, the at least partially autonomous robot can be a service, assembly or stationary production robot, alternatively an autonomous flying object, such as a drone.

The at least partially autonomous vehicle (10) can comprise a detection unit (11). The detection unit (11) can be, for example, a camera that detects the surroundings of the vehicle (10) or a radar that detects distances to objects in the surroundings of the vehicle (10). In a further embodiment, the detection unit (11) can be a depth sensor. The detection unit (11) can be connected to a prediction module (12). The prediction module (12) determines an output variable depending on a provided map and preferably depending on a provided input variable, e.g. provided by the detection unit (11), compare 3 The output variable characterizes, for example, where objects in the surroundings of the vehicle (10) will be located at a subsequent point in time. The output variable can then be forwarded to a control unit (13).

The control unit (13) controls an actuator depending on the output variable of the prediction module (12), preferably controlling the actuator in such a way that the vehicle (10) carries out a collision-free maneuver. In the first embodiment, the actuator can be an engine or a braking system of the vehicle (10). Alternatively, a trajectory of the vehicle (10) or the objects can be determined depending on the output variable of the prediction module (12).

Furthermore, the vehicle (10), in particular the semi-autonomous robot, comprises a computing unit (14) and a machine-readable storage element (15). A computer program can be stored on the storage element (15), which includes instructions which, when the instructions are executed on the computing unit (14), lead to the computing unit (14) executing the method for predicting the locations of objects, such as in 3 It is also conceivable that a download product or an artificially generated signal, each of which can comprise the computer program, causes the computing unit (14) to carry out the method after being received at a receiver of the vehicle (10).

In an alternative embodiment, the prediction module (12), which is connected to the detection unit (11) as in the first embodiment, can be used for building control. User behavior is detected by means of a sensor, for example a camera or a motion detector, and the control unit controls, for example, an automated door depending on the output variable of the prediction module (12). The prediction module (12) can then be set up to predict where the user will go based on the detected user behavior in order to specifically control the door, for example. Alternatively, the prediction module (12) can be used in a production hall to monitor that no person gets too close to a production machine and to switch it off if necessary.

In a further embodiment, the prediction module (12) can be used for an assistance system, for example a parking assistance system, which carries out a parking maneuver depending on the prediction module (12) that predicts the movement of the objects in the environment. It is conceivable that instead of the detection unit (11) in this embodiment, stationary sensors are arranged in the environment of the assistance system.

The prediction module (12) can comprise a deep neural network, in particular a convolutional neural network. For example, the deep neural network can be the trained deep neural network from one of the documents mentioned above.

2A shows a schematic representation of a map (20), which in this embodiment represents a floor plan of a room. Two objects (P1, P2) are shown on the map, which can move, for example, in the direction of the door shown at the top right. Furthermore, a robot (R) is shown, which guides the at least partially autonomous vehicle (10) to 1 The prediction module (12) can, for example, predict the locations of the objects (P1, P2) at a subsequent point in time depending on the map (20).

2 B shows a schematic representation of the card (20) which is divided into a plurality of cells (21). As in 2 B As shown, the cells (21) are each assigned a probability of occurrence (22). The probability of occurrence (22) characterizes how likely it is that at least one of the two objects (P1, P2) is located within the respective cell. Since the object (P2) is located in the spatial proximity of the constriction in the middle of the room, that cell is assigned a probability of occurrence with the value 0.9. For example, due to an ineptly chosen division of the cells, the neighboring cell below may have a probability of occurrence with the value 0.67.

It should be noted that the choice of cell size, in particular the cell shape (e.g. round or triangular) can be arbitrary. It should also be noted that the map (20) can alternatively or additionally show geographical features, such as road courses or building layouts.

3 shows a schematic representation of the method (30) for predicting the locations of the objects (P1, P2), in particular on the map (20).

The method (30) begins with step 31. In step 31, a card (20), which for example an environment of the robot. The map (20) is then broken down into cells (21). Each cell is initially assigned an estimated or random probability of occurrence (22). The probability of occurrence (22) can characterize whether one of the objects is located within the respective cell or how likely it is that one of the objects is located within the respective cell, compare 2 B . Furthermore, in step 31, a predeterminable number of fictitious objects can be initialized. Preferably, the predeterminable number of fictitious objects is approximately 1,000 to 1,000,000. Furthermore, each fictitious object can initially be assigned a position on the map and a speed. The position and the speed can be assigned randomly, alternatively the speed can be initialized with 1 m/s.

After step 31 has been completed, step 32 follows. In step 32, the fictitious objects are placed on the map, particularly depending on their assigned position.

In the following step 33, the fictitious objects are each given a weight. The respective weight is determined depending on the probability of occurrence (22) that was assigned to the respective cell in which the respective fictitious object is located. For example, the weight of one of the fictitious objects can be the value of the probability of occurrence of the respective cell in which the fictitious object is located.

After step 33 has been completed, step 34 is carried out. In this step, a predefined number of fictitious objects are drawn by means of drawing and replacement, depending on the weight of the fictitious objects. The number of fictitious objects drawn is appropriately equal to the number of initialized fictitious objects. For example, if 1000 fictitious objects are placed on the map and each of these fictitious objects has a weight, 1000 fictitious objects are randomly drawn and replaced from these 1000 fictitious objects, which can result in individual fictitious objects being drawn multiple times, particularly if they have a high weight. The drawn fictitious objects are then placed on the map. They are placed based on the position assigned to the fictitious objects. The drawn fictitious objects preferably take on the speed of the respective fictitious objects.

Step 35 can then be performed. Here, the weights of the fictitious objects are adjusted depending on the number of fictitious objects that are located within the same cell. For example, each weight of the fictitious objects that are all located within the same cell can be divided by the number of fictitious objects located within that cell.

In the following step 36, the speed (v) assigned to the fictitious objects can be normalized. For example, the assigned speed can be a vector that is normalized depending on an amount of the vector. $$\tilde{v}=\frac{1}{\Vert v\Vert }v$$

This is followed by step 37. In step 37, the map (20) is provided as an input variable to a trained machine learning system. The trained machine learning system then determines a speed distribution for each cell depending on the map, in particular the outlines or geometry of the action spaces of the objects and/or the robot (10). The speed distribution can, for example, be a matrix whose dimensions each correspond to a direction in which the objects move on the map, and a speed can be assigned to each column and/or row. The entries in the matrix can indicate the probability that the fictitious object has the respective direction with the respective speed. Then, depending on the determined speed distribution and the respectively assigned speed of the fictitious objects, each fictitious object is assigned an updated speed from the determined speed distribution, preferably drawn randomly (see equation 4 below).

It should be noted that the machine learning system can have been trained for this purpose as in the documents mentioned above. It should also be noted that the machine learning system also determines the speed distribution depending on the speed of the objects, especially fictitious ones.

$$u_i=exp\left(-\frac{\mathrm{\Delta }\tilde{v}}{\sigma }\right)$$

$$P\left(v_{i,NN}\right)=\frac{u_i}{{\displaystyle {\sum }_i{u}_i}}$$

wobei der Index i die ermittelten Geschwindigkeiten des maschinellen Lernsystems indiziert und σ vorzugsweise einen Wert von 0.3 aufweist.The assignment of the updated speed can be performed, for example, with the following equations, where the updated speed is randomly drawn according to equation 4: $$\mathrm{\Delta }\tilde{v}=\tilde{v}-v_{i,NN}$$

$$u_i=exp\left(-\frac{\mathrm{\Delta }\tilde{v}}{\sigma }\right)$$

$$P\left(v_{i,NN}\right)=\frac{u_i}{{\displaystyle {\sum }_i{u}_i}}$$

where the index i indicates the determined speeds of the machine learning system and σ preferably has a value of 0.3.

In step 38, a change in speed, in particular an acceleration, is then determined depending on the determined speed distribution. See, for example, EN 102018201570.8 . In this document, the acceleration is determined depending on a difference between at least two speed values from the speed distribution at a given updated speed. Alternatively, an acceleration distribution can be derived from the speed distribution by determining speeds from the respective direction of a probability distribution from the differences, which characterizes the probability with which the respective acceleration occurs. The change in speed can be drawn randomly from this acceleration distribution, for example.

Alternatively, the machine learning system may further be trained to determine an acceleration distribution depending on the map (20), from which an acceleration is determined for a given updated speed.

wie in Schritt 36 gezeigt, skaliert.After step 38 has been completed, step 39 can follow. Here, the change in speed depends on the normalization of the speed with $\frac{1}{\Vert \text{v}\Vert },$

scaled as shown in step 36.

und der Änderung der Geschwindigkeit a eine neue Geschwindigkeit v' der einzelnen fiktiven Objekten ermittelt. Dies kann beispielsweise mit der Gleichung 5 durchgeführt werden: $$v\prime =\widetilde{v_{akt}}+a$$

In step 40, depending on the updated speed $\left(\widetilde{v_{akt}}\right)$

and the change in speed a, a new speed v' of the individual fictitious objects is determined. This can be done, for example, with equation 5: $$v\prime =\widetilde{v_{akt}}+a$$

wobei r die aktuelle Position der fiktiven Objekte ist.Subsequently, in step 40, a new position (r') of the fictitious objects is determined depending on the new speed (v') of the fictitious objects. This can be done, for example, using equation 6: $$r\prime =r+v\prime$$

where r is the current position of the fictitious objects.

wobei P(O_cp) das Gewicht des p-ten fiktiven Objekt in der Zelle c ist. Es sei angemerkt, dass diese Gewichte in Schritt 35 den fiktiven Objekten zugewiesen wurden. P(O_c) ist die Auftrittswahrscheinlichkeit (22) der Objekte (P1,P2) der c-ten Zelle.Then, in step 41, the occurrence probabilities of the cells are updated depending on the weights of the fictitious objects that are located within the respective cells. For example, with the equation: $$P\left(O_c\right)=1-{\displaystyle \prod _p\left(1-P\left(O_{cp}\right)\right)}$$

where P(O _cp ) is the weight of the p-th fictitious object in cell c. Note that these weights were assigned to the fictitious objects in step 35. P(O _c ) is the occurrence probability (22) of the objects (P1,P2) of the c-th cell.

mit der den bestimmten Auftrittswahrscheinlichkeiten aus der Umfeldmessung: $$P\left(Z_c=z_c|O_c=o_c\right)=\{\begin{array}{cc}1-w_Z,& if{z}_c=o_c\\ w_z,& else\end{array}$$

wobei w_Z ein Rauschen der Messung charakterisiert und o_c entspricht einer besetzten Zelle mit einem der Objekte (P1, P2).Optionally, an environment measurement can be carried out in a subsequent step, in which, for example, the positions of the objects are recorded. Depending on this environment measurement, the occurrence probabilities of the cells can be adjusted. This can be done, for example, with the following equation: $$P\left(O_c\right)=P\left(Z_c|O_c\right)P\left(O_c\right)$$

with the determined occurrence probabilities from the environmental measurement: $$P\left(Z_c=z_c|O_c=O_c\right)=\{\begin{array}{cc}1-w_Z,& ie{z}_c=O_c\\ w_z,& else\end{array}$$

where w _Z characterizes a noise of the measurement and o _c corresponds to an occupied cell with one of the objects (P1, P2).

This ends the method (30). Advantageously, the method (30) is started again cyclically with step 33. When dragging the fictitious objects in step 34, their determined positions and, above all, their determined speeds can be retained. It should be noted that the sequence of steps can be carried out in any conceivable order.

Depending on the probability of occurrence of the cells, the position of the objects (P1, P2) at a subsequent point in time can be determined. For example, if the probability of occurrence of individual cells is relatively high compared to their neighboring cells, it can be deduced that an object will be located within these cells at the subsequent point in time.

Claims (17)

Method (30) for predicting a location, in particular a movement path, of a plurality of objects (P1, P2), wherein a map (20) which has at least one action space of the objects (P1, P2) and is divided into a plurality of cells (21), wherein each cell (21) is assigned a first size (22) which characterizes, in particular a probability (P(O _c )), whether at least one of the objects (P1, P2) is located within the respective cell, wherein a plurality of fictitious objects are placed (32) on the map (20) and the fictitious objects are each assigned a weight depending on the first size (22) which is assigned to the cell within which the respective fictitious object is placed (33), wherein a machine learning system determines a plurality of different discrete speeds depending on the map (20), wherein a new position (r') of the fictitious objects is determined depending on a selected discrete speed of the plurality of different discrete speeds and the fictitious objects are placed (34) on the map (20) based on the new position (r'), wherein the first size (22) of the Cells are determined depending on the weights of the fictitious objects placed within each cell (35). Method according to one of the preceding claims, wherein the location of the objects (P1, P2) is predicted as a function of the updated first sizes (22) of the cells. Method according to one of the preceding claims, wherein, in particular randomly and repeatedly, the fictitious objects are selected depending on the assigned weights and the selected fictitious objects are placed within the respective cell within which the selected fictitious objects were placed. Method according to one of the preceding claims, wherein each fictitious object is assigned a position (r) and a speed (v), wherein a new speed (v') is determined for each fictitious object depending on the assigned speed of the respective fictitious object and the selected discrete speeds, wherein the new position (r') of the fictitious objects is determined depending on the new speed (v'). Method according to one of the preceding claims, wherein the first sizes of the cells are determined depending on a product of the weights (P(O _cp )) of the fictitious objects placed within the respective cell. Method according to one of the preceding claims, wherein a measurement, in particular a detection of the objects, is carried out, wherein depending on the measurement a plurality of the first sizes (22) of the cells (21) are determined, in particular adapted. Method according to one of the preceding claims, wherein the speed of the fictitious objects is a vector and the speeds depend on an amount $\left(\frac{1}{\Vert \text{v}\Vert }\right)$

of the respective vector. Method according to one of the preceding claims, wherein a change in the speed, in particular an acceleration (a), is determined for each fictitious object depending on the plurality of different discrete speeds and the selected discrete speeds of the respective fictitious object, wherein the new position (r') of the fictitious objects is further determined depending on the change in the speed. Procedure according to Claim 7 and 8th , where the determined change in speed depends on the amount $\left(\frac{1}{\Vert \text{v}\Vert }\right)$

of the velocity vector of the respective fictitious object. Method according to one of the preceding claims, wherein the weights of those selected fictitious objects placed within the same cell are reduced depending on a number of these objects. Method according to one of the preceding claims, wherein the machine learning system is a convolutional neural network, wherein the machine learning system determines a plurality of the different discrete speeds for each cell and assigns a probability of occurrence to each discrete speed depending on the previous discrete speed. Method according to one of the Claims 4 until 11 , where differences (Δṽ) between the assigned speed and the determined discrete speeds are determined, where the selected discrete speed of the fictitious objects is randomly selected depending on the difference between one of the determined discrete speeds. Method according to one of the Claims 4 until 12 , whereby a speed of the objects is recorded, whereby the associated speeds of the fictitious objects is determined depending on the detected speed of the objects. Method according to one of the preceding claims, wherein a control variable for controlling a robot is determined as a function of the first sizes of the cells. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of the preceding claims. Machine-readable storage element (15) on which the computer program according to Claim 15 is deposited. Device (13) which is arranged to carry out the method according to one of the Claims 1 until 14 to execute.

Images