WO2018020044A1

WO2018020044A1 - Method and system for associating data pertaining to the detection and tracking of moving objects for an automotive vehicle

Info

Publication number: WO2018020044A1
Application number: PCT/EP2017/069252
Authority: WO
Inventors: Hala LAMDOUAR
Original assignee: Valeo Schalter Und Sensoren Gmbh
Priority date: 2016-07-29
Filing date: 2017-07-28
Publication date: 2018-02-01
Also published as: FR3054672B1; FR3054672A1

Abstract

The invention relates to a method for associating data pertaining to the detection and tracking of moving objects with a view to merging them, said data arising from a first object detector and from a second object detector fitted to an automotive vehicle, in the form of a first list and a second list of detected objects, the method being characterized in that it comprises the construction (110) of an initial bipartite graph between a first set and a second set, in which the vertices of the first set of the bipartite graph correspond to the objects of the first list and the vertices of the second set of the bipartite graph corresponding to the objects of the second list, said construction comprising a step of creating (111) links between vertices of the first set and vertices of the second set and of assigning (112) a weight to each created link; and the determination (120) of a minimum-weight perfect matching by combinatorial optimization of said initial bipartite graph in order to obtain a simple final bipartite graph in which one vertex of the first or of the second set is linked at most to one vertex of the second or of the first set.

Description

METHOD AND SYSTEM FOR ASSOCIATING DETECTION DATA AND SUI VI OF MOBILE OBJECTS FOR VEHICLE AUTOMOBILE

The present invention relates generally to the field of motor vehicles, and more specifically to a method and a system for associating data for detecting and tracking moving objects with a view to their merging.

It is known to equip certain motor vehicles with driving assistance systems using different sensors for detecting different objects located in the vehicle environment in order to allow drivers and / or driving assistance systems to operate. adapt driving to the situation.

The objects to be detected may be, depending on the desired applications, obstacles on the road, such as pedestrians or other vehicles, or any information relating to the route taken, such as road marking lines, the recognition of road signs signaling or traffic lights. In the following, we are particularly interested in the detection of moving objects (pedestrians or vehicles of any type).

Any detection system used for driving assistance conventionally comprises at least one sensor capable of detecting objects in the vehicle environment, typically at the front, rear or on one side of the vehicle in an area of the vehicle. observation, as well as a processing module associated with this sensor. The processing module is capable of delivering at least one piece of information relating to each detected object, typically the position (Cartesian coordinates or distance associated with an angle) of this object with respect to the vehicle. Some processing modules also allow, from image processing captured by a camera, a classification of the detected object, for example to identify the presence of a pedestrian, or vehicles likely to present a danger. In the following, we call "object detector" the assembly formed by a sensor of a given technology and its associated processing module.

Different sensor technologies (camera, radar, lidar, laser sensor, ultrasonic sensor) can be used as needed. The aforementioned object detectors nevertheless have the drawback of being imprecise in certain types of measurements. Thus, a detection system using a camera does not allow very precise measurements in distance, unlike the radar. On the contrary, the radar is less precise in angle than a camera. In addition, the active sensors (radar, lidar, laser) are accurate in position but not in classification. In particular for the lidar and the laser, classification inaccuracy is due to the fact that they try to reconstruct an object from the geometric distribution of the reflected beams.

To ensure a reliable perception of the vehicle environment, it is known to use several detectors of different technology objects, and to merge the data from these different detectors.

This so-called "high level" multi-sensor fusion, described for example in the article entitled "A multi-sensor fusion System for moving object detection and tracking in urban driving environments" (Cho et al., 2014 IEEE International Conference on robotics & Automation (ICRA) Hong Kong Convention and Exhibition Center - May 31-June 7, 2014), basically comprises the following three steps:

an association step of determining whether two objects detected by two different detectors correspond to the same object or not;

a step of fusion / reconstruction of the object by combining the objects associated with the preceding step.

a step of tracking (or tracking) the different objects.

To illustrate the problem of the so-called high-level fusion, FIG. 1 represents a road situation configuration with a motor vehicle 1 equipped with several detectors of objects of different technology, here a first detector R, for example using a sensor. radar placed at the front of the motor vehicle 1, and a second detector L, using for example a lidar sensor placed on the windshield of the motor vehicle 1. Figure 1 schematically shows the moving objects respectively detected by the detectors L and R. The detected objects, as they come from the different detectors, are in the form of bounding boxes, here rectangles. The boxes in solid lines correspond to the objects detected by the detector R whereas the boxes in dotted lines correspond to the objects detected by the detector L. Thus, in the example represented:

- the object detector R detected the presence of four objects

1 2 3 4

mobiles referenced O _R , O _R , O _r and O _R.

- the L object detector detected the presence of two moving objects

1 2

referenced O ^ and O ^.

The indices R and L used in the notations identify the detector responsible for the detection of objects.

Each bounding box corresponds to an object as detected by the detector in question, including, in particular, estimated information concerning its relative position with respect to the motor vehicle 1, its dimensions, its relative speed and possibly, depending on the type of detector used. , other attributes like its class (type of objects).

FIG. 1 also shows the uncertainty ellipses around the position of each detected object, or covariance ellipses, the position being modeled by a 2D Gaussian distribution whose parameters (standard deviations) are given by the suppliers. sensors.

Thus, by way of example, the ellipse E _R corresponds to the uncertainty in position of the object O _r detected by the detector R.

The purpose of the above-mentioned association step is to determine what are,

1 2 3 4

among the objects from the list of objects {O _R , O _R , O _R , O _r } delivered by

1 2

the detector R and the objects coming from the list of objects {O ^, O ^} delivered by the detector L, the objects which correspond in fact to one and the same mobile obstacle. In other words, the association step consists in coupling or matching the objects coming from the two lists, according to similarity criteria, so that the data of the objects thus matched can be merged to reconstruct an object closer to the reality. Current algorithms rely on the position of the bounding boxes and overlapping areas of uncertainty ellipses to decide whether two boxes may or may not be associated. Typically, the positions of the objects are modeled by a two-dimensional Gaussian distribution, centered on a particular point, for example the center of the bounding boxes or, as shown in FIG. 1, the middle of the segment of the box closest to the vehicle. 1. Two objects detected by two different detectors are then matched if the ellipses of associated uncertainties overlap the most. Each paired object is removed from the object list, and the algorithm is repeated iteratively on the remaining object lists.

The disadvantage of this type of algorithm is that it does not solve conflicts when an object of a list can be paired with two objects of the other list. For example, in the case of Figure 1, one could consider

3 1 2 that the object O _r could be matched either to the object O ^ or to the object O ^, taking into account the similar distances separating these objects, and

1 3 3 2 similar recoveries of the ellipses E ^ and E _R on the one hand, and E _R and E ^ on the other hand.

The present invention aims to overcome the disadvantages of high level fusion algorithms hitherto used.

To do this, the subject of the invention is a method for associating data for detecting and tracking moving objects with a view to their fusion, said data coming from a first object detector and a second detector. of objects equipping a motor vehicle, in the form of a first list and a second list of detected objects, the method being characterized in that it comprises the following steps:

constructing an initial bipartite graph between a first set and a second set, in which the vertices of the first set of the bipartite graph correspond to the objects of the first list and the vertices of the second set of the bipartite graph correspond to the objects of the second list , said construction comprising a step of creating links between vertices of the first set and vertices of the second set and assignment of a weight to each link created;

- Determining a perfect coupling of minimum weight by combinatorial optimization of said initial bipartite graph to obtain a simple final bipartite graph in which a vertex of the first, respectively the second set, is connected at most to a vertex of the second, respectively first set.

According to other possible features:

each detected object being associated on the one hand with at least one attribute representative of a relative speed between the motor vehicle and the detected object, and on the other hand with an ellipse of detection uncertainty, a link is created in said initial bipartite graph between a first vertex corresponding to a first object of the first list and a second vertex corresponding to a second object of the second preference list according to a comparison between the corresponding attributes representative of the relative velocities and ellipses corresponding uncertainties;

said link may in particular be created if the ellipses of corresponding uncertainties overlap and if a difference between the corresponding attributes representative of the relative speeds is less than a predetermined threshold value;

said threshold value is advantageously predetermined as a function of the relative speed associated with the first object;

the step of creating a link between said first vertex and said second object can also take into account at least one other attribute of the corresponding detected objects, such as a classification of the objects or a direction of movement;

for each link created in the initial bipartite graph between a first vertex and a second vertex, the associated weight preferably corresponds to a difference between the relative velocities associated with the first object and the second object.

The subject of the invention is also a system for associating data for detecting and tracking moving objects with a view to their fusion, said data coming from a first object detector and an object detector. second object detector equipping a motor vehicle, in the form of a first list and a second list of detected objects, the system being characterized in that it comprises means capable of:

constructing an initial bipartite graph between a first set and a second set, in which the vertices of the first set of the bipartite graph correspond to the objects of the first list and the vertices of the second set of the bipartite graph correspond to the objects of the second list, said construct comprising creating links between vertices of the first set and vertices of the second set and assigning a weight to each created link;

- Determine a perfect coupling of minimum weight by combinatorial optimization of said initial bipartite graph to obtain a simple final bipartite graph in which a vertex of the first, respectively the second set, is connected at most to a vertex of the second, respectively first set.

The two detectors may be of different technologies, each of said detectors uses a sensor selected from the group comprising a vision sensor, a radar, a lidar. The invention and the various advantages that it provides will be better understood from the following description, made with reference to the appended figures, in which:

- Figure 1 already described above, schematically shows a road situation and the obstacle detection results obtained with a motor vehicle equipped with two detectors of different technologies;

FIG. 2 illustrates steps that can be performed in a data association method according to the invention;

FIGS. 3a and 3b schematically illustrate an initial bipartite graph and a final simple graph obtained according to the principles of the invention for the traffic situation of FIG. 1;

FIG. 4 illustrates, in the form of a simplified block diagram, an example of a data association system according to the invention. The principle on which the invention is based is that of modeling the problem of the association of objects from at least two detectors by graph theory.

Figure 2 schematically illustrates possible steps for such modeling:

Each detector R or L provides, during a prior step 100, its own list of detected objects that corresponds to a finite set of objects. More precisely, the detector L delivers a list of n objects, which can be represented mathematically by the set {O ^} for which i is an integer varying from 1 to n. Similarly, the detector R delivers a list of objects, which can be represented mathematically by the set

{for which j is an integer varying from 1 to m. The two lists do not necessarily include the same number of objects, and it will be assumed in the following that the integer n is less than or equal to the integer m.

Each detected object is associated on the one hand with at least one attribute representative of a relative speed between the motor vehicle and the detected object, and on the other hand with an ellipse of detection uncertainty.

Thus, detector L also provides a set of n ellipses of uncertainty that can be represented mathematically by {E ^}, and a set of n relative velocities whose mathematical representation is

{V _j } Similarly, the set of uncertainty ellipses associated with the objects detected by the detector R, and {V. } the set of relative speeds corresponding to these same objects.

To illustrate this principle in the context of the road configuration of

In Figure 1, the detector L delivers the set of two objects {O ^, O ^} and the

1 2 3 4 detector R delivers the set of four objects {O _R , O _R , O _R , O _R }.

A first step 110 of the association method according to the invention consists in constructing an initial bipartite graph between the two sets of detected objects in which the vertices of the first set of the bipartite graph correspond to the objects detected by the detector L, and the vertices of the second set of the bipartite graph correspond to the objects detected by the detector R. The construction of this initial graph also requires the creating links between the vertices of the first set and those of the second set, and the assignment of a weight or cost to each link created.

The link creation step 111 consists of searching, for each object or vertex of the first set, the possible candidates, in terms of resemblance, in the vertices of the second set.

In a preferred implementation of the invention, an object of the second set is candidate to be matched to an object of the first set if and only if:

- there is intersection of its uncertainty ellipse with the uncertainty ellipse of the object of the first set; and

the relative speeds associated with the objects are close.

This can be mathematically translated by the following expressions:

V I [1,1:

0 and

in which :

I is an integer representing the total number of possible candidates to be matched to an object; and

V ^ corresponds to a predetermined threshold value. Advantageously, this threshold value is a function of the relative speed of the object for which the candidates are sought.

In particular, it can be defined that: Link creation takes into account at least the relative speeds. However, in other implementations, the creation of a link between said first vertex and said second object may also take into account at least one other attribute of the corresponding detected objects, such as a classification of the objects or a direction of movement.

In order to complete the initial graph, weights or costs are also assigned to each link created (step 112). The associated weight preferably corresponds to a difference between the relative velocities associated with the objects concerned by the connection, which can be expressed mathematically by the expression

where C (i _L , j _R ) is the weight assigned between a link connecting the object O ^ and the object O _r . Another metric can be used, such as the Euclidean distance between speeds.

FIG. 3a shows the example of the initial bipartite graph obtained for the object detection configuration shown in FIG. 1. It contains the two vertices corresponding to the two objects detected by the detector L, the four vertices corresponding to the detected objects. by the detector R and four links shown in dashed lines, which were created in this initial graph 4 according to the previous calculations, as well as the weights or costs associated with these links. In this FIG. 3a, it can be seen in particular that the objects 0 _R and

3 1

0 _R are two possible candidates to be matched to the object O _l , with associated weights respectively noted C (1 _T , 1 _R ) and C (1, ¾).

In the next step 120, we will seek to minimize these costs so as to eliminate links and retain only those that will be representative of the final association of objects. In other words, step 120 consists of to determine a perfect coupling of minimum weight by combinatorial optimization of the initial bipartite graph to obtain a simple final bipartite graph in which a vertex of the first, respectively the second set, is connected at most to a vertex of the second, respectively first set. Eliminate as much connection as possible with the least cost.

This can be mathematically translated by the following expressions:

n m

m ⁱⁿ ΣΣc (i _L , j _R ) ^χ _¾ ^with

i = 1 j = 1

χ _¾ e {0, l} V e [l, n] * [l, m] m

Σx = 1 V ie [l, n]

j = 1

not

and Σx .. <1 V j _e [l, m]

i = 1 which means in practice that one seeks to obtain a simple final graph in which:

each object detected by the detector L must be associated with a single object detected by the detector R;

it is nevertheless accepted that an object detected by the detector R is not matched.

In the preceding equations, the integer variable _x reflects the fact that a bond exists between two vertices. It is equal to 1 if are

connected and 0 otherwise. Figure 3b shows the example of the single final bipartite graph obtained for the object detection configuration shown in Figure 1. The process as just described is particularly advantageous in the context of multi-sensor fusion using sensors of different technologies. Nevertheless, it can also be applied in cases where the sensors are of the same nature.

FIG. 4 summarizes, in the form of a simplified block diagram, various possible components of a system 6 for associating multi-sensor detection data fitted to a motor vehicle, according to the invention. In the example taken, the system 6 receives the objects detected on the one hand by a first detector of objects R (objects O ^), and on the other hand by a second detector of objects L (objects Oj _^ ) _. The detectors may be, as represented in FIG. 4, components external to the system 6, used for example for other driving assistance functions. Alternatively, the object detectors are an integral part of the system 6.

References 7, 8 and 9 in FIG. 4 illustrate the data processing modules associated with each step of a high-level data fusion process. Thus, the system 6 comprises means 7 responsible for the association of objects detected by the different detectors, means 8 responsible for merging the objects that have been associated, and means 9 capable of tracking the objects.

In accordance with the principles of the invention, the association of two objects delivered by each of the two detectors is based on the prior construction of a bipartite graph, by means referenced 70, then on the combinatorial optimization, by means referenced 71, until a simple bipartite graph is obtained, as explained above.

Claims

1. A method for associating data for detecting and tracking moving objects with a view to their fusion, said data coming from a first object detector (R) and a second object detector (L) equipping a motor vehicle (1), in the form of a first list and a second list of detected objects, the method being characterized in that it comprises the following steps:

constructing (110) an initial bipartite graph (4) between a first set and a second set, in which the vertices of the first set of the bipartite graph correspond to the objects of the first list and the vertices of the second set of the bipartite graph correspond to the objects of the second list, said construct comprising a step (111) of creating links between vertices of the first set and vertices of the second set and assigning (112) a weight to each link created;

determination (120) of a perfect coupling of minimum weight by combinatorial optimization of said initial bipartite graph (4) to obtain a simple final bipartite graph (5) in which a vertex of the first or second set is connected at most to a summit of the second, respectively first set.

2. Method according to claim 1, characterized in that each detected object is associated on the one hand, with at least one attribute representative of a relative speed between the motor vehicle (1) and the detected object, and other to an ellipse of detection uncertainty, a link is created (111) in said initial bipartite graph between a first vertex corresponding to a first object of the first list and a second vertex corresponding to a second object of the second list based on a comparison between the corresponding attributes representative of relative velocities and ellipses of corresponding uncertainties.

3. Method according to claim 2, characterized in that said link is created if the ellipses of corresponding uncertainties overlap and if a difference between the corresponding attributes representative of the relative speeds is less than a predetermined threshold value.

4. Method according to claim 3, characterized in that said threshold value is predetermined according to the relative speed associated with the first object.

5. Method according to any one of claims 2 to 4, characterized in that the step (111) for creating a link between said first vertex and said second object also takes into account at least one other attribute of the detected objects. corresponding, such as a classification of objects or a direction of movement.

6. Method according to any one of claims 2 to 5, characterized in that, for each link created in the initial bipartite graph between a first vertex and a second vertex, the associated weight corresponds to a difference between the relative velocities associated with the first object and the second object.

7. System (6) for associating data of detection and tracking of moving objects with a view to their fusion, said data coming from a first detector (R) of objects and a second detector (L) of objects equipping a motor vehicle (1), in the form of a first list and a second list of detected objects, the system being characterized in that it comprises means (7, 70, 71) adapted to :

constructing an initial bipartite graph (4) between a first set and a second set, in which the vertices of the first set of the bipartite graph correspond to the objects of the first list and the vertices of the second set of the bipartite graph correspond to the objects of the second set list, said construct comprising creating links between vertices of the first set and vertices of the second set and assigning a weight to each created link; determining a perfect coupling of minimum weight by combinatorial optimization of said initial bipartite graph to obtain a simple final bipartite graph (5) in which a vertex of the first or second set is at most connected to a vertex of the second or first set respectively .

System according to claim 7, characterized in that said at least two detectors are of different technologies.

System according to any one of claims 7 or 8, characterized in that each of said detectors uses a sensor selected from the group comprising a vision sensor, a radar, a lidar.