DE102013102153A1 - Method for combining sensor signals of LiDAR-sensors, involves defining transformation value for one of two LiDAR sensors, which identifies navigation angle and position of sensor, where target scanning points of objects are provided - Google Patents

Abstract

The method involves defining a transformation value for one of the two LiDAR sensors (16,20,24,30), which identifies a navigation angle and a position of the sensor. The target scanning points of the objects are provided, and are detected by the sensors. The target scanning points provide a separate target scanning map for each sensor. The target scanning map is projected from one sensor to the other sensor through a current transformation value. The multiplicity of the weighting values is determined through the current transformation value.

Description

Cross-reference to related applications

This application claims the priority of priority from US Provisional Patent Application No. 61 / 611,465 entitled "Method for Registration of Range Images of Multiple LiDARs" filed on Mar. 15, 2012.

BACKGROUND OF THE INVENTION

1. Field of the invention

This invention relates generally to a system and method for registration of range images of multiple LiDAR sensors, and more particularly to a system and method for registering range images of multiple LiDAR sensors on a vehicle, the method comprising registering two frames of data includes multiple LiDAR sensors at the same time.

2. Discussion of the Related Art

Many modern vehicles include object detection sensors that are used to warn of or avoid a collision and other active safety applications. The object detection sensors may use any number of detection techniques, such as short range radar, image processing cameras, laser or LiDAR, ultrasound, etc. The object detection sensors detect vehicles and other objects located in the path of the host vehicle, and the application software uses the object detection information. to provide alerts or take appropriate action. In many vehicles, the object detection sensors are integrated directly into the front bumpers or other front trim panels of the vehicle.

In order for the application software to run optimally, the object detection sensors must be correctly aligned with the vehicle. For example, if a sensor determines an object that is in the path of the host vehicle due to misalignment that the object is slightly shifted left to the path of the host vehicle, this can have significant consequences for the application software. Even though a plurality of forward-looking object detection sensors are located on a vehicle, it is important that all of them are properly aligned to minimize or eliminate conflicting sensor evaluations.

LiDAR sensors are a type of sensor that is sometimes used on vehicles to detect objects that are around the vehicle. These sensors provide the distance to these objects. LiDAR sensors are desirable because they are able to provide the orientation of a tracked object, which other types of sensors, such as vision systems and radar sensors, generally can not. In one type of LiDAR sensor, the reflections from an object are represented as a sample point as part of a dot cluster area map, with a separate sample point provided every 0.5 ° across the field of view of the sensor. As a result, multiple sample points that are displayed representing the distance of the target vehicle from the host vehicle may be present when a target vehicle is detected in front of the host vehicle.

A vehicle may include multiple LiDAR sensors to provide a 360 ° field of vision around the vehicle. These multiple LiDAR sensors include side-facing sensors, rear-facing sensors, and front-facing sensors. Each of these sensors tracks objects in its field of view independently of the other sensors. Using the reproduction of sampling points from the plurality of sensors, a distance map is generated to track the objects in proximity to the host vehicle. For a vehicle with multiple LiDAR sensors, multiple point cluster maps are issued, and for sensors where the field of view overlaps, the sensors can track the same object. It is necessary to combine the sampling point maps of the sensors so that the same object tracked by the sensors is processed as a single target.

US Patent Application 12 / 942,456 entitled "Systems and Methods for Tracking Objects" filed on Nov. 9, 2010 by the assignee of the present application and incorporated herein by reference, discloses a technique for monitoring the range and orientation of a target object of US Patent Application Serial Numbers a host vehicle using LiDAR sensors. This patent application is limited to a single LiDAR sensor and does not discuss the connection of multiple LiDAR sensors.

SUMMARY OF THE INVENTION

In accordance with the teachings of the present invention, a system and method for registering range images of objects detected by multiple LiDAR sensors on a vehicle is disclosed. The method includes aligning data frames of at least two LiDAR sensors having overlapping fields of view in a sensor signal combining operation to track objects detected by the sensors. The method defines a transformation value for at least one of the LiDAR sensors that identifies an orientation angle and position of the sensor and provides target scan points from the objects detected by the sensors, the target scan points providing a separate target point map for each sensor. The method projects the target point map from the at least one sensor to another of the LiDAR sensors using a current transform value to overlap the target scan points from the sensors, and determines a plurality of weight values using the current transform value, each weight value representing a position change of one of the sampling points for the at least one sensor is identified to a location of a sampling point for another one of the sensors. The method computes a new transformation value using the weighting values, compares the new transformation value with the current transformation value to determine a difference thereof, and corrects the plurality of weighting values based on the difference between the new transformation value and the current transformation value until the new transformation value the current transformation value, the sensors being aligned with each other.

Further features of the present invention will become apparent from the following description and the appended claims, taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE FIGURES

1 Fig. 11 is an illustration of a host vehicle following a target vehicle, showing the fields of view of four LiDAR sensors on the host vehicle;

2 Figure 4 is a general block diagram of a merging system for merging sampling points of multiple LiDAR sensors on a vehicle;

3 (A) and (B) show sample points from a LiDAR sensor;

4 Fig. 10 is a flow chart showing a method of registering sample points of multiple LiDAR sensors;

5 is a contour probability density function from the back of an object;

6 is an iterative boundary optimization graph;

7 Fig. 10 is a block diagram showing two interwoven methods for estimating object movement and updating an object model when a new LiDAR sample point map is available;

8th is a dynamic Bayesian network for an object that is tracked by a tracking algorithm;

9 FIG. 10 is a flowchart showing a method for a multi-object tracking algorithm in a single time step; FIG.

10 FIG. 12 is a bipartite graph illustrating predicted object model points and segmented scan map points for a step of matching sample clusters to predicted object models in the diagram of FIG 9 shows;

11 is an induced bipartite graph derived from the bipartite graph of the 10 was generated;

12 Fig. 10 is an illustration of sampling points of a LiDAR sensor, an image system, and a radar sensor;

13 Figure 11 is an illustration of a host vehicle following a target vehicle and showing the fields of view of a LiDAR sensor, a radar sensor and an imaging system on the host vehicle;

14 Figure 4 is a general flowchart for a tracking algorithm for a LiDAR sensor using cueing from a radar sensor or imaging system;

15 FIG. 10 is a flowchart showing a method for a multi-object tracking algorithm using cueing from a radar sensor or an imaging system; FIG.

16 Fig. 10 is a bipartite graph showing match of target data with object models from all LiDAR sample points, radar sensor readings and image system images;

17 is an induced bipartite graph derived from the bipartite graph of the 16 was generated;

18 FIG. 4 is a bipartite graph showing merging of two projected object models by providing matches using an image system; FIG.

19 is an induced bipartite graph that matches the object models 18 shows;

20 is a bipartite graph showing the splitting of the projected object models by matching using an image system;

21 is an induced bipartite graph that matches the object models 20 shows;

22 is a bipartite graph showing the projected object models by matching using a radar sensor;

23 is an induced bipartite graph that matches the models of the 22 shows; and

24 is a dynamic Bayesian network for a tracking algorithm and for modeling an update with cueing information from a radar sensor or imaging system.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The following discussion of embodiments of the invention directed to a system and method for registering range images of multiple LiDAR sensors on a vehicle is merely exemplary in nature and is in no way intended to limit the invention or its applications or uses. For example, the present invention will be described in terms of registration of range images for LiDAR sensors on a vehicle. However, those skilled in the art will appreciate that the registration method of the invention will also have application outside of vehicle applications.

1 shows an overview of a vehicle system 10 with a carrier vehicle 12 behind a target vehicle 14 lies and pursues this. The carrier vehicle 12 includes four LiDAR sensors, a front-facing sensor 16 with a field of view 18 , a rear-facing sensor 20 with a field of view 22 , a left-facing sensor 24 with a field of view 26 and a rightward sensor 28 with a field of view 30 , The sensors 16 . 24 and 28 are on the front of the vehicle 12 attached and have, as shown, overlapping fields of view. If an object, such as the target vehicle 14 , in the field of view of a particular sensor of the sensors 16 . 20 . 24 and 30 the respective sensor reproduces a plurality of sample points identifying the object. The points 32 on the target vehicle 14 represent the sampling points of the target vehicle 14 from each of the sensors 16 . 24 and 28 be reproduced. The points 32 be in the vehicle coordinate system (x, y) on the host vehicle 12 transferred using a coordinate transformation technique. Thereafter, the object detection in the vehicle coordinate system using the points 32 executed. The points 32 can be used to rear the shape of the target vehicle 14 as will be discussed below.

Every sensor 16 . 20 . 24 and 28 will provide a sample point cloud for each separate object detected by the sensor. The present invention proposes a merging algorithm that includes the outputs of each of the sensors 16 . 20 . 24 and 28 so combined that when the sensors 16 . 20 . 24 and 28 tracking the same object, this object is processed as a single target, the algorithm outputting the position, orientation and velocity of each tracked object. Although this discussion includes four LiDAR sensors, the proposed merging algorithm may be applicable to any number and location of multiple LiDAR sensors having overlapping fields of view.

The position of the target vehicle 14 is represented in this diagram by an anchor point 34 , the center of the scan card. The following values are used to construct an object model M of the target vehicle 14 to represent in a time step t. In particular, the object model M defines the relative longitudinal velocity V _X , the relative lateral velocity V _Y , the lateral displacement Y and the vehicle direction ξ, or the direction of the ground velocity vector of the target. The value M is a list of Gaussian components generated by the parameter mean value m _J and the variance σ ² are shown. The mean value is characterized by several hyperparameters v _j , η _j , k _j and flags as seen and aged.

2 is a schematic block diagram for a merge system 36 for merging sample point cloud values from multiple LiDAR sensors, such as the sensors 16 . 20 . 24 and 28 , The box 38 Sets the sample point cloud representation from the left-facing LiDAR sensor 24 , the box 40 the sample point cloud representation from the rightward LiDAR sensor 28 , the box 42 the sample point cloud representation from the forward-looking LiDAR sensor 16 and the box 44 the sample point cloud representation from the backward LiDAR sensor 20 The distance maps from the LiDAR sensors 16 . 20 . 24 and 28 are registered and in the box 46 A 360 ° distance map (point cloud) is constructed. Once the point clouds from the sensors 16 . 20 . 24 and 28 are registered and a 360 ° point cloud is placed in the vehicle coordinates, the algorithm places the point clouds from the multiple targets in the box 48 together, as will be discussed below. After the targets are merged into the vehicle coordinate frame, the algorithm gives the position, orientation, and velocity of the targets in the box 50 out.

Before the merging method of combining the sample points of multiple LiDAR sensors will be discussed in detail, a discussion of a sample point registration algorithm disclosed in the box 46 is performed to estimate the movement of the object if the object model M and the current sampling rate S associated with the object are available.

In many vehicles, the object detection sensors are integrated directly into the front body of the vehicle. This type of installation is simple, effective, and aesthetically pleasing, but has the disadvantage that there is no practical way to physically align the sensors. Accordingly, conventionally, there is no other way to correct a misalignment that may result from a sensor being misaligned with respect to the real vehicle direction due to damage to the front structure or aging or weathering distortion, than by replacing the entire front structure with the sensors.

As will be discussed below, the frame registration in the box matches 46 is executed, the distance sampling points from the sensors 20 . 24 and 28 To detect a possible drift in the position and orientation of the sensors 20 . 24 and 28 adapt. The sensors 20 . 24 and 28 are calibrated in the beginning when the vehicle 12 new is. As noted, various factors cause these orientations to change over time, and accordingly, a method must be implemented to recalibrate the sensor orientation so that objects located in the overlapping region of the fields of view 26 and 30 be detected, to be detected more accurately. The present invention proposes an expectation maximization (EM) match algorithm to find a transformation T between multiple LiDAR sensors that define an orientation angle and the x and y position of the sensors. For example, the algorithm becomes the transformation T from the left-handed LiDAR sensor 24 on the right-facing LiDAR sensor 28 match and when the transformations T from consecutive calculations agree with each other, the sensors become 24 and 28 aligned with each other.

The EM algorithm begins by selecting an initial transformation value T ₀ , zero, a previously estimated value, an orientation between the sensors 24 and 28 which is provided by the manufacturer, etc. may be. The algorithm then projects a left sensor removal map to the rightward LiDAR sensor frame using transformation T 28 , 3 is an illustration of a sample point map 120 , where circles 124 Sample point representations from the left LiDAR sensor 24 and ovals 126 Sample point representations from the right-handed LiDAR sensor 28 represent. 3 (A) shows all sample point representations and 3 (B) shows an enlarged area in the circle 122 for some of the sample point representations. 3 (B) shows how the sample point representations 124 of the left-facing LiDAR sensor on the sample point representations 126 of the right-facing LiDAR sensor with the arrows 128 be imaged. Using the currently available transformation T for the projection map arrows 128 become the sample point representations 124 of the left-facing LiDAR sensor relative to the sample point representations 126 of the rightward LiDAR sensor moves to make it overlap.

wobei die Kernel-Funktion K ist:

The presently used transformation T may be for the present orientation of the left-handed LiDAR sensor 24 on the right-facing LiDAR sensor 28 not be accurate enough so that the transformation T for the current position of the sensors 24 and 28 must be updated. The algorithm uses the current transformation T to update a weight a _jk between the left sensor sample point s _j and the right sensor sample point m _{k as} follows:

where the kernel function is K:

A new transformation T 'is then determined using the newly calculated weight a _{jk as} follows: T '= argmin _T - 1 / 2Σ _{j, k} a _jk logK (|| s _j - T ∘ m _k ||). (3)

The new transformation T 'is then compared with the previously calculated transformation T and based on its difference, the weight a _{jk is} recalculated using the new transformation T' until the newly calculated transformation T 'matches the previously calculated transformation T, where the sensors 24 and 28 aligned with each other.

In certain cases, the transformation T to align the sample data points will be sufficiently large, and it may be advantageous to improve the solution of T 'in equation (3). For this example, the transformation T is defined as x '= T * x, where x' = Rx + t, and, where R is a rotation matrix and t is a translation vector. In this analysis: S = [S ₁ , S ₂ , ..., S _L ] ^T , (4) M = [m ₁ , m ₂ , ..., m _N ] ^T , (5) A = [ _ajk ], (6) 1 = [1, 1, ..., 1] ^T , (7) μ _s = 1 / LS ^T A ^T 1, (8) μ _m = 1 / NM ^T A1, (9) Ŝ = S - 1μ T / S (10) M ^ = M - 1μ T / m (11)

The solution of the new transformation T 'in equation (3) can now be represented as follows: T '= [R', t '], (12) in which: R '= UCV, (13) t '= μ _s - R'μ _m, (14) and where U and V are the factors for a single value decomposition of: UPS = sνd (Ŝ ^T A ^T M ^), (15) are and C is:

The above-described EM algorithm for determining the transformation T can only be locally optimal and sensitive to the initial transformation value. The algorithm can be improved so that a particle swarm optimization (PSO) is used to find the initial transformation T _0th In this optimization, let E be the set of selectable transforms T from the left LiDAR sensor 24 on the right LiDAR sensor 28 , The algorithm randomly generates N particles (t _i | t, ε E) according to a uniform distribution in the selectable transformation E and each particle t _i is associated with a normalized weight w _i = 1 / N. For each particle t _i , the EM algorithm is set to the best transformation T _i , where t _{i is given} as the initial value. The weight w _i is the percentage of the correspondence between the two samples for the transformation T _i . The algorithm then returns the transformation T _k with the best match percentage as the nominal value for the transformation from the left LiDAR sensor 24 on the right LiDAR sensor 28 from where w _k = max (w _i ).

In most cases, the change in the transformation T from one sample time to the next sample time will typically be small, and hence it is not necessary to use a computationally intensive method of computing the new large transformation transformation T 'T as described above. to use. In particular, the new estimate from the above PSO algorithm of the transform T _{n can} be recursively refined using the following low-computational EM match algorithm. First, the transformation T _{n is} applied to each sample point of the right LiDAR sensor 28 applied as follows: S _j ← T _n ∘ S _j . (17)

The corrected transformation ΔT is determined as: ΔT: x '= ΔT ∘ x, (18) and defined as: x '= x - Ɛy t + _x, (19) y '= y + t _y + Ɛx, (20) wherein the transformation ΔT is modeled as (t _x , t _y , Ɛ), and wherein the previous transformation ΔT _o (t _xo , t _yo , Ɛ _o ).

wobei: S_j = (x ' / j, y ' / j), (23) m_k = (x_k, y_k), (24)

und wobei λ ein Richtungsfaktor ist, der dazu eingestellt werden kann, inwieweit die vorausgegangene Schätzung verwendet wird.Equation (3) is then replaced by: T '= argmin _T - 1 / 2Σ _{j, k} α _jk logK (|| S _j - T ∘ m _k ||) + λ || ΔT - ΔT _o || ² , (21) and the solution is:

in which: S _j = (x '/ j, y' / j), (23) _mk = ( _xk , _yk ), (24)

and where λ is a direction factor that can be adjusted to what extent the previous estimate is used.

4 is a flowchart 130 This is the operation to align the LiDAR sensors, such as the LiDAR sensors 24 and 28 as discussed above. In the box 132 For example, the algorithm selects the initial transformation T _n , such as the manufacturer's origin setting. In the box 134 The algorithm collects scanning distance map data from the LiDAR sensors 24 and 28 and determines if a sufficient number of sample points from an object in the field of view of the LiDAR sensors 24 and 28 are present, that the calculations adequate in the decision diamond 136 be executed. If there are not enough sample points, the algorithm returns to the box 134 back to collect more distance map data. If there are enough sample points in the decision diamond 136 The algorithm then uses the EM matching algorithm to find the corrected transformation ΔT of the data in the box 138 to find. The algorithm then determines if the corrected transform ΔT is greater than a predetermined threshold in the decision diamond 140 and, if so, a large transform estimate is used, using the PSO algorithm, to construct the new nominal transform T _n in the box 142 to find. If the match percentage is not greater than the threshold in the decision diamond 140 then the algorithm returns to the box 134 back to collect the next sample points.

To return to the merging algorithm, the discussion below first introduces a proposed sample registration algorithm that determines the movement of the target vehicle 14 if the object model M and the current scan map S that belong to the target vehicle 14 corresponds, are given. The discussion above about the EM algorithm for determining the transformation T, which aligns the frames between the LiDAR sensors, reflects spatial matching, in particular matching between two frames of different LiDAR sensors at the same time. The discussion of sample point registration also uses a sample point registration algorithm to find the transformation T matching the two frames in time between the current sample card S and the object model M derived from the previous sample cards.

definiert. 5 zeigt einen Schnappschuss für eine beispielhafte PDF für die Rückseite des Zielfahrzeugs 14. Die PDF wird lieber direkt durch eine Liste von Partikeln (Dots) dargestellt, welche als M = (m₁, m₂, ..., m_nM)^T bezeichnet werden, als dass sie unter Verwendung bestimmter Parameter dargestellt werden.A probabilistic object model M is first defined, and then a proposed iterative algorithm is provided to find a fixed transform so that the probability is maximized because the next succeeding frame's scan maps are predetermined. To characterize the geometric shape of an object, a contour probability density function (PDF) is used in a sample space

Are defined. 5 shows a snapshot for an exemplary PDF for the back of the target vehicle 14 , The PDF is rather represented directly by a list of particles (dots), which are referred to as M = (m ₁ , m ₂ , ..., m _nM ) ^T , as being represented using particular parameters.

wobei

eine Gauß-Kernek-Funktion ist und σ > 0 ein Glättungsparameter, der als Bandbreite oder als Größe eines Parzen-Fensters bezeichnet wird. This non-parameterization of the way the PDF is displayed can be represented as:

in which

is a Gauss-Kernek function and σ> 0 is a smoothing parameter called the bandwidth or the size of a Parzen window.

Let y be the parameters of the transformation T _Y. The operators are called transformation operator T _y (·) with the parameters Y and the corresponding inverse transformation T -1 / y (·) Are defined. Without loss of generality, the object model M may be considered as originating 34 be considered centered. In the following frame, an object at location Y is characterized by the PDF:

wobei Abtastpunkte s_k als unabhängig und gleichförmig verteilte Bereiche des Objektmodells M am Ort y angenommen werden.Let S = (s ₁ , s ₂ , ..., s _ns ) ^{T be} the current scan map consisting of a list of sample points. The probability function can be expressed by:

wherein sample points s _k are assumed to be independent and uniformly distributed regions of the object model M at location y.

The task here is to find the maximum of L (S; y, M) with respect to the transformation parameters y, which is equivalent to the minimum of J (y; M) ≡ -log L (S; y, M ) to find: y * = argmin _y J (y; M), (31) in which:

This local minimum indicates the presence of the object in the following frame, which defines a representation similar to p (x; M) at the origin 34 having.

für alle k.An auxiliary matrix A ≡ {a _kj } with a _kj ≥ 0 for j = 1, ..., n _M and k = 1, ..., n _s is introduced, where:

for all k.

As a result, the equation (32) becomes:

in einen konstanten Term aufgenommen, welcher vernachlässigt wird. Gleichung (36) ergibt sich dann durch Anwenden der Ungleichung von Jensen auf die Gleichung (34).In the equation (32), the normalization factor becomes

taken into a constant term, which is neglected. Equation (36) is then obtained by applying the inequality of Jensen to equation (34).

für die Gleichung (36) durch Bilden der Ableitung des Ausdrucks der Gleichung (36) und des Lagrange-Multiplikators der Randbedingung aus der Gleichung (33) im Hinblick auf a_kj und durch Nullsetzen erlangt werden. Die optimale Hilfsvariable kann ausgedrückt werden durch:

für j = 1, ..., n_M and k = 1, ..., n_s.Assuming the negative logarithm function is convex, the best upper bound can be

for the equation (36), by taking the derivative of the expression of the equation (36) and the Lagrange multiplier of the constraint from the equation (33) with respect to a _kj and zeroing. The optimal auxiliary variable can be expressed by:

for j = 1, ..., n _M and k = 1, ..., n _s .

in Hinblick auf die Hilfsmatrix A, der Parameter y der Transformation T und der Bandbreite der Kernel-Funktion σ terminiert wird.As a result, equation (31) can be iteratively solved by "boundary optimization", which is between optimizing

with respect to the auxiliary matrix A, the parameter y of the transformation T and the bandwidth of the kernel function σ is terminated.

die Graphenlinie 56

und die Graphenlinie 58

6 is a graph in which y is plotted on the horizontal axis showing the iterative "boundary optimization relationship", where the line 52 is the solid curve J (y; M) and the broken curves are consecutive upper limits of y ₀ , y ₁ and y ₂ , namely the graph line 54

the graph line 56

and the graph line 58

The iterative procedure of Algorithm 1 below is repeated until convergence occurs; H. the difference of the estimated y in two iterations is less than a predetermined small number. Empirical results show that 2-3 iterations are sufficient for the proposed convergence algorithm.

Algorithm 1: Sample Point Registration

• Input: object model M, current scan map S and start transformation parameter y ₀ .

• 1) Let σ = σ ₀ , where σ _{0 is} a positive number.
• 2) Compute A: Given the previous estimate of the transformation parameter y _n , a _{kj is} updated using equation (37).
• 3) Minimize
  
  Compute y * to minimize
  
  ie: y * = argmin _y - Σ _{k, j} a _kj logK _σ (T -1 / y (s _k ) - m _j ). (38)
• 4) Calculate σ: differentiating equation (36) according to σ, setting the derivative to zero gives:
  
• 5) Let y _{n + 1} = y *. If
  
  then go to step 2. Otherwise, output y * is used as the estimated parameter for the transformation.

The following discussion describes a specific case of dot-set registrations that can be used in robotics with LiDAR scanners. A fixed transformation x '= T _y (x) and x' = Rx + t can be defined, wherein the parameter vector y consists of a rotation matrix R and the translation vector t. Equation (38) can be simplified to: y * = argmin _y Σ _{k, j} a _kj || s _k - Rm _j - t || ² , (40) such that det (R) = 1 and R ^T R = I.

wobei 1 = [1, 1, ..., 1]^T ist By forming the partial derivative of Equation (40) respectively after t and R, Equation (40) can be solved. To illustrate the solution, the following quantities are defined:

where 1 = [1, 1, ..., 1] ^T

The solution of equation (40) is: R = UCV ^T , (45) t = μ _s - Rμ _m, (46) wherein U and V as the factors of singular value decomposition are defined, that is, ^T = USV sνd (S ^T A ^T M ^) and C = diag (1, det (UV ^T)).

The discussion below suggests a Bayesian algorithm that recursively estimates the motion and updates the object model M. Let S ₀ , ..., S _t , and S _{t + 1 be} the scan maps taken by a dynamic object at time steps 0, ..., t, and t + 1, respectively.

7 shows that tracking as i) can be considered a problem for estimating the motion of an object and ii) a problem for updating the object model M when a new frame scan map is received. In particular, the shows 7 a procedure 60 to appreciate an object location in the box 64 where the object location is used to represent an object model in the box 62 to update and estimate the object location if a new scan card is available.

The discussion below will include references to bipartite graphs and Bayesian graphs. These two types of graphs contain nodes that represent different things. The nodes in a Bayes graph represent variables that must be estimated, such as the transform T and the object model M, which are random and can only be represented by a PDF. These nodes are arranged in a sequence over different time frames and each individual sequence models only one object. In contrast, the nodes in the bipartite graphs are segmentations of scancards. Each scan card can contain many objects in the same time frame. Consequently, it is the task to find the segments that correspond to an object. Forming the associations of the segments (s ₁ , s ₂ , ..., s _n ) along the time axis may result in obtaining a plurality of sequences, each corresponding to an object. In this way, the bipartite graphs can be used to apply the Bayesian method to track each individual object.

8th illustrates a dynamic Bayesian network 70 representing two steps of the proposed tracking algorithm. In the network 70 put the knots 72 the transformation parameters y _t and y _{t + 1} , ie the object position and the location of the targets, the nodes 74 represent the object models M _t and M _{t + 1} and the nodes 76 represent the scan cards S _t and S _{t + 1.} In the 8th The parameters y _t and y _{t + 1 of} the transformation at the time steps t and t + 1 are respectively estimated and M _t and M _{t + 1} are the object models at the time steps t and t + 1. To the Bayes network 70 p (y) becomes a Dirac delta distribution parameter around its center y and the object model M is considered to be the associated PDF of the Gaussian components parameterized by their mean {m _j } and a fixed variance σ ^2, respectively. Each mean m _j is represented by a Gauss PDF with hyperparameters, namely the mean and variance

The discussion below details the update rules for the parameter y and the hyperparameters {v _j , η _j | j = 1, ..., n _M }. Using Bayesian and Chain rules respectively, the following PDF is written with given sets of scancards up to time t + 1 as: p (y, M | S ^{(0: t + 1)} ) α p (S | y, M) p (y, M | S ^{(0: t)} ) = p (S | y, M) p (y | S ^{(0: t)} ) p (M | S ^{0: t} ), (47) where y and M are shortened to t + 1, S ^{(0: t + 1) represent} scan maps until time step t + 1, S the scan map at time t + 1, p (S | y, M) an equivalent representation of L (S; y, M) in equation (30), and the last equation follows by assuming a conditional independence given by S ^{(0: t)} .

In Equation (47), p (y | S ^{(0: t)} ) is the previous PDF of the parameter y at time step t-1 for given prior sampling maps S ^{(0: t)} , which can be calculated as: p (y | S ^{(0: t)} ) = ∫ p (y | y _t ) p (y _t | S ^{(0: t)} ) dy _t . (48)

ist. Falls p(y_t|S^(0:t)) eine Dirac-Delta-Verteilung zentriert um y ist, kann die vorhergehende PDF zum Zeitschritt t = 1 geschrieben werden als:

wobei ỹ ein vorhergesagter Wert des folgenden Fortpflanzungsmodells des Objekts ỹ = f(yt) ist.In equation (48), p (y _t | S ^{(0: t)} ) denotes the following PDF for transformation parameters at time t and p (y | y _t ) the conditional probability of the following propagation model of object motion: y = f (y _t ) + w, (49) where w is a zero-mean Gaussian random variable with variance matrix

is. If p (y _t | S ^{(0: t)} ) centers a Dirac delta distribution y is, the previous PDF can be written at time step t = 1 as:

where ỹ is a predicted value of the following propagation model of the object ỹ = f ( y t) is.

wobei:

und wobei (η_j, ν_j) Hyperparameter der vorhergehenden PDF für die j-te Komponente von M_t-1 sind und c(η_j) ein Normierungsfaktor ist.The expression p (M _t-1 | S ^{(0: t)} ) may be modeled as the conjugate distribution family of L (S; y, M) in equation (30) to produce a traceable subsequent PDF. A product of Gaussian densities with known variance σ ² is taken as:

in which:

and where (η _j , ν _j ) are hyperparameters of the previous PDF for the jth component of M _t-1 and c (η _j ) is a normalization factor.

Like from the 8th indicates that the preceding data for the object model M at time step t:

We now consider the problem of estimating the preceding p (y, M | S ^{(0: t + 1)} at time t + 1. Since y and M are conditionally independent, ie p (y, M | S ^{(0: t + 1)} = p (y | S ^{(0: t + 1)} ) p (M | S ^{(0: t + 1)} ), (54) the following may be estimated separately with respect to y and M with the following two steps.

und ||x|| 2 / Q = x^TQ^–1x sind. Demzufolge kann die Gleichung (38) durch die Gleichung (55) zum Integrieren von einer vorangegangenen Information von vorhergehenden Zeitschritten ersetzt werden und durch folgendes Anwenden des Algorithmus 1 kann y _t+1. abgeleitet werden.Assuming that p (y | S ^{(0: t + 1)} ) adds the Dirac delta PDF y _{t + 1} is obtained by maximizing equation (47) with respect to y, by substituting equation (50) into equation (47), applying a negative logarithm to equation (47) and neglecting terms, that are irrelevant for y: y _{t + 1} = argmin _y - Σ _{k, j} a _kj logK _σ (T -1 / y (s _k ) - m _j ) + || y - ỹ || 2 / Q, (55) in which

and || x || 2 / Q = x ^T Q ^-1 x are. Accordingly, the equation (38) can be replaced with the equation (55) for integrating a previous information of previous time steps and can be followed by applying the algorithm 1 as follows y _{t + 1} . be derived.

By neglecting terms relevant to the object model M, the negative logarithm of equation (47) can be written as:

It is noted that equation (56) is the best upper bound of J (M) for all possible values of y.

wobei: ρ_j = Σ_ka_kj (58) s _j = Σ_ka_kjs_k/p_j. (59) Like in the 6 can be seen, the boundary optimization can be used to iteratively find the optimal object model M by finding the minimum of the above-mentioned upper limit function. By setting the derivatives of equation (56) with the respective object model M to zero, the estimate for the object model M can be obtained as:

in which: ρ _j = Σ _k a _kj (58) s _j = Σ _k a _kj s _k / p _j . (59)

The update rules for the new hyperparameters (η '/ j, ν' / j) for the subsequent distribution of the object model M to S at time t + 1 can be written as:

As a result, the subsequent PDF of the target is at time t:

It is noted that m in the equation (57) is the mode and ν '/ i in equation (61), the mean value of the j-th component (particle) of the object model is M. Due to the Gaussian assumption these are identical.

The recursive tracking and model update method is summarized in Algorithm 2 below. Step 1 provides the object model M for the current time frame, with step 2 estimating the motion of the object model M. Step 3 updates the object model M based on the current scan map. Step 4 adds new particles to the object model M, where step 5 removes the "outliers" from the object model M.

Algorithm 2: Track and update the object model

• Input: current scan map S _{t + 1} , object model M _t-1 = {ν _j , η _j } and transformation y _t .
• Output: estimated y _{t + 1} , and updated M.

• 1) Calculated hyperparameters
  
  for all J of p (M _t | S ^{(0: t)} ) in equation (51). Set all particles as unobserved.
• 2) Be
  
  for all J. Replace equation (38) with equation (55) and then let algorithm 1 expire y _{t + 1} and to get A _{t + 1} .
• 3) If ρ _j - Σ _k a _{kj is} greater than a threshold, compute the hyperparameters using equations (60) and (61), consider the particles as considered, and increase the view value K _j by 1. If K _j > 5 , the jth particle is marked as ripe.
• 4) Calculate
  
  for all K. Falls
  
  is less than a threshold, add a new particle s _k with the following values K _k = 1, ν _k = s _k , η _k = 1 and mark it as considered.
• 5) Remove particles that are not marked as viewed and not ripe.

soll ein ungerichteter Graph mit der Textmenge R sein. Eine Kante

verbindet p₁ und p₂, falls ||p₁ – p₂|| kleiner ist als ein vorbestimmter Distanzschwellenwert. Das Bezeichnen der verbundenen Komponenten wird dann dazu verwendet, um die Abtastkarte R in eine Liste von Clustern {S (t+1) / n} zu segmentieren. Das Segmentieren der Abtastpunkte in Cluster beinhaltet das Separieren der Cluster von Abtastpunkten in Wiedergabepunktewolken, so dass die Cluster ein separates Objekt identifizieren, das verfolgt wird. 9 is a flowchart 80 showing the proposed merge algorithm executed at each time step t. The box 78 represents the object files generated at each time step and represents the position, velocity and orientation of the objects being detected and tracked, and the object model M for each object being tracked. If a new frame of distance data from the sensors 16 . 20 . 24 and 28 at the carrier vehicle 12 in the box 82 The algorithm first constructs the 360 ° point cloud in the box 84 in the manner discussed above. Once the point cloud is constructed, the algorithm then segments the sample points in the cloud into cluster, the specific objects in the box 86 can identify. To perform the segmentation operation, the scan card in the current frame should designate (t + 1) and

should be an undirected graph with the text set R. An edge

connects p ₁ and p ₂ if || p ₁ - p ₂ || is less than a predetermined distance threshold. The designation of the connected components is then used to put the scan card R into a list of clusters {S (t + 1) / n} to segment. Segmenting the Sample points in clusters involves separating the clusters of sample points into rendering point clouds so that the clusters identify a separate object that is being tracked.

für alle j. Durch

werden die vorhergesagten Objektpunkte aus dem Zeitschritt t bezeichnet.Once the sample points are segmented into clusters, the algorithm then matches the clusters with predicted object models M ~ in the box 88 , In particular, the algorithm projects tracked object models M in the previous time step in the box 90 using the object files 78 , To predict the projected object models, should {Mn / t} a list of object models M to the time step t. For every object model M n / t the mean of a Gaussian component is referred to as ν n / j. If y n / t is the estimate of the transformation to the time step t, is the predicted transformation to the time step t + 1 ỹ n / t + 1 = f ( y n / t). As a result, the predicted object model is M ~ for the nth object

for all j. By

the predicted object points from the time step t are designated.

mit vorhergesagten Objektmodellen

im Kasten 88 gematcht. Ein bipartiter Graph

wird aus der gesetzten Vertexmenge

über die Kantenmenge E_B konstruiert. Eine Kante existiert zwischen den Punkten

und

nur dann, wenn ||p – q|| < D, wobei D ein Distanzschwellenwert ist. Das Gewicht der Kante (p, q) ist definiert als w(p, q) = ||p – q||. Die Nachbarschaft von

ist definiert als

Using the projected tracked object models, the segmented clusters become

with predicted object models

in the box 88 matched. A bipartite graph

becomes from the set vertex amount

designed over the edge set E _B. An edge exists between the points

and

only if || p - q || <D, where D is a distance threshold. The weight of the edge (p, q) is defined as w (p, q) = || p - q ||. The neighborhood of

is defined as

kann definiert werden aus dem bipartiten Graph B, wobei die Kantenmenge E_B die mögliche Assoziierung zwischen den Objekten in

und den segmentierten Clustern

darstellt. Eine Kante zwischen

und

existiert dann und nur dann, wenn eine Kante (p, q) in B so existiert, dass

und

gilt.An induced bipartite graph

can be defined from the bipartite graph B, where the edge set E _{B is} the possible association between the objects in

and the segmented clusters

represents. An edge between

and

exists if and only if an edge (p, q) exists in B such that

and

applies.

Das Gewicht und die Kardinalität der Kante werden jeweils definiert als:

Let E (p ', q') be the subset of the set of edges in B, ie

The weight and cardinality of the edge are defined as:

Once the induced bipartite graph B 'is constructed and the weight and cardinality of the edges are calculated, the strong edge and the advanced weak connection are selected using Algorithm 3 below.

10 is a bipartite graph 100 that is, B at a sample point degree for five predicted object models M ~ and five clusters S at the current time step t, as discussed above. In the bipartite graph 100 represent dots 102 the predicted object model points and triangles 104 put the segmented scancard points. The lines 106 between the dots 102 and the triangles 104 are the edges of the graph 100 ,

The algorithm then collapses the data point clusters and splits them in the box 92 on to combine the detected objects that are the same, and splits objects that were originally detected as a single object but have been shown as multiple objects. To perform the merging and splitting of the clusters, Algorithm 3 below performs a sample association.

11 illustrates an induced bipartite graph 110 that is, B 'derived from the graph B by consolidating points into groups. In the induced bipartite graph 110 put the knots 114 the object model points (M), the nodes 112 the clusters of segmented sample points (M) and the lines 116 the edges defining the coincidence between the object model points (M) and the segmented sampling points (S), is. As will be discussed below, M is ~ ₁ at time t to S ₁ at time t + matched 1, M ~ ₂ and M ~ ₃ on S ₂ at time t + 1 combined and M ~ ₄ split. M ~ ₅ is currently disappearing and is removed from the scan card at time t + 1 and S5 arrives at time t + 1. In step 3 of the algorithm 3, the edges (M ~ ₁ , S ₁ ), (M ~ ₂ , S ₂ ), (M ~ ₄ , S ₃ ), and (M ~ ₄ , S ₄ ) are highlighted. In step 4, M ~ _{3 is} matched with S ₂ and the edge (M ~ ₃ , S ₂ ) is highlighted. As a result, in step 5, the broken line (M ~ ₅ , S ₂ ) is trimmed as the soft connection. 11 illustrates that M ~ _{1 is} directly matched with S ₁ . However, the two other cases require clustering and splitting of the clusters.

The predicted object model M ~ ₄ is divided into two mixed clusters S ₃ and S ₄ . split. The sample points of the current clusters S ₃ and S ₄ are clustered again using the prior knowledge that there was an object in the previous frame. Usually, a stronger evidence of dissimilarity is needed for splitting an existing trackable object into two clusters.

The two predicted object models M ~ ₂ and M ~ ₃ are associated to a cluster S ₂ in the current frame.

The sample points in the cluster S ₂ are clustered again because a greater proximity among the sample points is needed for collapsing the two trackable objects into one object.

Algorithm 3: Sample Association Algorithm

• Input: the cluster
  
  and the predicted object models
  
• Output: the shortened induced bipartite graph B '.

• 1) Construct the bipartite graph B with sampling points as vertices.
• 2) Construct the induced graph B 'with clusters and contours such as the vertices and calculate the weight and cardinality of the edge using equations (63) and (64), respectively.
• 3) For everyone
  
  find
  
  so the cardinality is greater than a threshold (ie c (p '/ *, q')> C ) and
  
  
  is and lift the edge (p '/ *, q') out.
• 4) For everyone
  
  which is not covered by the highlighted edges, find the amount
  
  such that each element q '' is covered by one and only one highlighted edge. Find p '' such that (p '', q '')> C and
  
  applies, and lift out the edge.
• 5) Shorten all edges that are not highlighted in B '.

As mentioned above, new objects are created and vanishing objects are in the box 94 away. In particular, two special cases must be dealt with in order to create new objects and to remove existing objects in the object file, with no edge for the Abtastcluster S ₅ exists. One Trace initialization step is triggered and a new object is added to the object file for scanning the cluster S ₅ , and the predicted object model M ~ ₅ disappears because there is no edge. As a result, it is removed from the object file.

Once the new objects are created and the disappearing objects in the box 94 the algorithm then sets a trace and model update in the box 96 ready. For each matched pair of object models M _t and sample clusters S _{t + 1} , the algorithm 2 is used, for example, by the sample registration method of algorithm 1 to track and update the object model.

The object file 78 is then in the box 98 updated and for the next time step for the object file 78 stored. The new transform y _{t + 1} and the updated object model M _t for each object are stored on the object file and the algorithm waits for the arrival of a new scan map to the next time step.

wobei (x, y) und (x', y') zwei Punkte in vorangegangenen und gegenwärtigen Rahmen jeweils sind,

der Rotationswinkel und (t, t) die Translationen sind.If the rotation between two consecutive sample frames is small, the parameters of a 2D transformation can be approximated by:

where (x, y) and (x ', y') are two points in previous and present frames, respectively,

the angle of rotation and (t, t) are the translations.

jeweils gebildet werden und auf Null gesetzt werden. Möge

den vorhergesagten y mit der Gleichung (49) bezeichnen. Wenn ein konstantes Modell angenommen wird, werden t ~_x = ν_xδt, t ~_y = y_xδt und

mit bereitgestellt, wobei δt die Dauer zwischen den Zeitschritten t und t – 1 ist.Let m _j = (x _j , y _j ) and S _k = (x '/ k, y' / k). By substituting equations (65) and (66) into equation (55), the partial derivatives after t _x , t _y and

each be formed and set to zero. May

denote the predicted y by the equation (49). If a constant model is assumed, t ~ _x = ν _x δt, t ~ _y = y _x δt and

with δt being the duration between the time steps t and t-1.

wobei λ_x, λ_y und

die von der Varianzmatrix Q in der Gleichung (50) abgeleiteten Gewichte sind.The following update rules can be derived:

where λ _x , λ _y and

are the weights derived from the variance matrix Q in equation (50).

bei y _t+1 zu berechnen.By replacing equation (38) with equations (67) through (69), algorithm 1 is iteratively applied to t _x , t _y and

at y _{t + 1} to calculate.

When ν and ω _H denote the HV ground speed and yaw rate, respectively. The ground speed of the vehicle 14 is calculated by: ν _gx = ν _x + ν _H - x _c ω _H , (70) ν _gy = ν _y + y _c ω _H , (71) and the orientation of the vehicle 14 by:

The method and apparatus for detecting and tracking multiple objects as described above may provide a 360 ° field of view solution for detecting objects by the host vehicle 12 provide. However, the following issues for a LiDAR sensor need to be addressed.

The first problem is that part of the observation, either by occlusion or laser measurement loss, for example, due to lower reflectance or specular reflection. For example, the detection range of a black vehicle is much smaller than that of a white vehicle.

The next problem is the insufficient measurement due to a low resolution, causing feature protrusion and hence is insufficient to detect far-field objects.

Another problem is the limited vertical field of view, such as that which omits the detection of objects on a non-planar road surface.

Another problem is the lack of context information that can distinguish an object from the background.

The present invention proposes an improved merging algorithm that addresses the limitations discussed above for the LiDAR sensors. In particular, the present invention utilizes the output signals from one or both of a radar sensor or an imaging system on the vehicle to operate as a cueing signal to cause the LiDAR sensor output merging system to identify a target. Radar sensors typically have a long range but a narrow field of view. Image systems typically have a short range, but are contacts to the detected targets. As mentioned above, the improved merging system discussed above is intended to provide the parameters x, y, v _x , v _y, and ξ for each detected object appreciate.

12 is a picture 150 a goal that is within a street 152 as detected by a LiDAR sensor, a radar sensor and an imaging system on a host vehicle. In particular, the target is far enough away or excluded that the LiDAR sample points are few and of the points 156 being represented. To improve the detection of the target, defines an image system 154 same destination from an image system output and a radar data point 158 representing the same target detected by a radar sensor.

13 is a representation of a vehicle system 160 similar to the system 10 That up for a vehicle 162 which was a target vehicle 164 follow and pursue this. The same variables for position, velocity, direction angle, etc. are used as in the 1 shown. In this example, the host vehicle includes 162 a LiDAR sensor 166 with a field of view 168 , a radar sensor 170 with a field of view 172 and an image system 174 with a field of view 176 , The illustrations of the LiDAR sensor 166 are called sample points 178 shown, the representation of the radar sensor 170 is called a triangle 180 and the representation of the image system 174 is called a box 182 shown. As known in the art, image systems and radar sensors output four outputs from a target, namely the distance to the target, the range change to the target, ie the range rate, the heading angle of the target, and the change in the azimuth angle of the target, ie the azimuth angle rate.

) von dem Radarsensor 170 und/oder dem Bildsystem 174 im Kasten 192 an einen Cueing-Kontextblock 194 gesendet wird. Wie oben diskutiert, werden die LiDAR-Abtastclusterkarten im Block 196 an ein LiDAR-Verfolgungsalgorithmusblock 198 bereitgestellt. Der Verfolgungsalgorithmus empfängt die Cueing-Kontextzieldaten von dem Block 194 und die Zieldaten werden mit dem abgespeicherten Objektmodell gematcht, wobei der Wert n_o die Zahl der detektierten Ziele ist. Jedes Ziel o = (x '' / i, y '' / i, ν '' / i, w '' / i) hat die Parameter einer longitudinalen Verschiebung (x''), einer lateralen Verschiebung (y''), einer Radialgeschwindigkeit (ν'') und einer Lateralgeschwindigkeit (w''). Die LiDAR-Abtastrate

mit jedem Abtastpunkt s_k beinhaltet einen longitudinalen Offset x und einen lateralen Offset y_i, wobei N die Anzahl der Abtastpunkte ist. Der Verfolgungsalgorithmus überwacht den Eingang von dem Cueing-Kontext-Kasten 194 und die LiDAR-Abtastkarte und erzeugt die Ausgangsparameter x, y, v_x, v_y, und ξ für jedes detektierte Objekt. Ein Anwendungsblock 200 verwendet die geschätzten Parameter der Objekte und implementiert aktive Fahrzeugsicherheitsanwendungen, beispielsweise eine adaptive Fahrzeugkontrolle, ein Kollisionsvermeidungssystem etc. 14 is a block diagram with a general overview of a proposed merging system 190 , wherein the target O = (o ₁ , o ₂ ,.

) from the radar sensor 170 and / or the image system 174 in the box 192 to a cueing context block 194 is sent. As discussed above, the LiDAR sample cluster maps are in block 196 to a LiDAR tracking algorithm block 198 provided. The tracking algorithm receives the cueing context target data from the block 194 and the target data is matched with the stored object model, where the value n _{o is} the number of targets detected. Every goal o = (x '' / i, y '' / i, v '' / i, w '' / i) has the parameters of a longitudinal displacement (x ''), a lateral displacement (y ''), a radial velocity (ν '') and a lateral velocity (w ''). The LiDAR sampling rate

with each sample point s _k includes a longitudinal offset x and a lateral offset y _i , where N is the number of sample points. The tracking algorithm monitors the Input from the cueing context box 194 and the LiDAR scan map and generates the output parameters x, y, v _x , v _y , and ξ for each detected object. An application block 200 uses the estimated parameters of the objects and implements active vehicle safety applications, such as an adaptive vehicle control, a collision avoidance system, etc.

15 is a flowchart 204 similar to the flowchart 80 discussed above, with the same operations provided with the same reference numerals. The flowchart 204 includes the box 206 , which is the target data from the radar sensor 170 or the image system 174 reads. Thereafter, the algorithm registers the target data to the LiDAR coordinate frames in the box 208 , To achieve this, y _{o should be} the parameters (translation and rotation) of the rigid transformation T that is a target of the radar sensor 170 or the image system 174 maps to the LiDAR coordinate frame so that the mapped targets are in the LiDAR frame:

The registration parameters y _o can be automatically determined by matched pairs between the tracked LiDAR objects and targets from the radar sensor 170 or the image system 174 to be appreciated. The US 7,991,550 entitled "Method and Apparatus for On-Vehicle Calibration and Orientation of Object Tracking Systems" issued Aug. 7, 2011 and assigned to the assignee of this patent application and incorporated herein by reference, discloses a technique suitable for this purpose.

und Abtastpunkten von allen Objektmodellen

sind definiert. Dabei bezeichnet

alle Zielpunkte, entweder Dreiecke oder Rauten, von dem Radarsensor 170 oder dem Bildsystem 174.Thereafter, the algorithm matches the target data from the radar sensor 170 or the image system 174 on the object models M in the box 210 , This procedure is in the 16 as a picture 230 with a carrier vehicle 232 shown, where points 234 Object model points are triangles 236 Radar targets and diamonds 238 virtual samples are taken from the image data from the image system 174 are derived. A radar target o _r is modeled by a point represented by the triangles 236 is shown. An image target O _ν is modeled as a rectangle, represented by a list of recorded points on the edge of the rectangle with "virtual" scan lines drawn from the host vehicle 232 go out, are shown. The value O _ν = {o _l | l = 1, ...., 6} is called the diamonds 238 shown. The set of object models

and sample points of all object models

are defined. This designates

all target points, either triangles or diamonds, from the radar sensor 170 or the image system 174 ,

The method discussed above is used to construct a truncated bipartite graph 240 from the 17 to construct. The graph 240 shows the object movement 242 in the knot 246 and the object models 244 in the knot 248 , where o ₁ ↔ M ~ ₁ , o ₂ ↔ M ~ ₂ , o _{3 is} a candidate for creating a new object, and the predicted object model M ~ ₃ is marked as disappearing, which can be removed from the object file.

Using the projected tracking object models M ~ out of the box 90 and the matched cluster with projected object models M ~ out of the box 88 and the matched target data with the object models from the box 210 the algorithm makes the merging and splitting of object models M in the box 212 ready. Two cases are considered, namely image targets and radar targets. 18 shows a bipartite graph 250 in the case where two predicted object models M ~ ₁ and M ~ ₂ are matched to the image target O ₁ . This cueing information from the image system 74 illustrates that the two object models M ~ ₁ and M ~ ₂ must be combined into a new object model M ~ _1-2 .

19 is an induced bipartite graph 252 that summarizes the object models M ~ ₁ and M ~ ₂ in the node 256 from the image target O ₁ in the node 254 into the single object model M ~ _1-2 .

20 is a bipartite graph 260 showing another case wherein an object model M ~ ₁ matches two image targets O ₁ and O ₂ . The cueing information from the image system 174 illustrates that the object model M ~ _{1 has} to be split into two separate object models M ~ _1a and M ~ _1b .

21 is an induced bipartite graph 262 that splits the object model M ~ ₁ in the node 264 shows which of the two image targets O ₁ and O ₂ that are in the node 266 and 268 are detected in the object models M ~ _1a and M ~ _1b passing through the nodes 270 and 272 are each shown.

22 is a bipartite graph 270 showing a case where the point set R for two object models M ~ ₁ and M ~ ₂ are matched with three radar targets o ₁ , o ₂ and o ₃ .

23 is an induced bipartite graph 276 showing the maps o ₁ → M ~ ₁ , o ₂ → M ~ ₂ and o ₃ → M ~ ₂ . In particular, the graph shows 276 that the radar target in the node 278 on the object model M ~ ₁ in the node 280 and the radar targets o ₂ and o ₃ in the nodes 282 and 284 which are considered to be in a single object model M ~ ₂ in the node 286 are shown summarized.

Then the algorithm creates new objects and removes vanishing objects in the box 214 , Similar to the methods discussed above for the flowchart 80 Two special cases need to be handled to create new objects and remove vanishing objects from the object file. For each unmatched sample cluster S of a LiDAR segment and each unmatched target from the radar sensor 170 or the image system 174 For example, a trace initialization method is used to add a new object to the object file. For an object in the object file, the object is marked as vanishing if there is no match either to a segmented cluster S from the LiDAR sensor 166 or to a target of radar sensor 170 or the image system 174 gives. If the object is consistently marked as disappearing in different consecutive time steps, it is then removed from the object file.

wird die Messung von dem Zeitschritt O bis zu dem Zeitschritt t bezeichnet.After that, the algorithm sets a trace and model update in the box 216 ready. To achieve this, o _o , ..., o _t , and o _{t + 1 become} the mapped measurements in the Li-DAR coordinate frame of a dynamic object by the radar sensor 170 or the image system 174 in each of the time steps O, t and t + 1. By

the measurement is called from the time step O to the time step t.

24 illustrates a revised dynamic Bayesian network 290 from the 8th which represents two time steps for the proposed tracking algorithm with cueing information. The values y _t and y _{t + 1} are parameters that must be estimated for the transformation in the time steps t and t + 1, respectively. The models M _t and M _{t + 1} are the object models in the time steps t and t + 1. In the network 290 , are the knots 292 the target data o _t and o _{t + 1} at the time steps t and t + 1, the nodes 294 are the transformation parameters y at the time steps t and t + 1, the nodes 296 are the object models M at the time steps t and t + 1 and the nodes 298 are the scancards at time steps t and t + 1.

Similar to Equation (47), Bayesian and Chain rules are each used to approximate the following PDF given LiDAR scancards and target data from the radar sensor 170 or the image system 174 derive up to the time step t + 1: p (y, M | S ^{(0: t + 1)} , o ^{(0: t + 1)} ) α p (S | y, M) p (o | y, M) p (y, M | S ^{(0 : t)} , o ^{(0: t)} ) = p (S | y, M) p (o | y) p (y, | S ^{(0: t)} , o ^{(0: t)} ) p (M | S ^{(0: t)} , o ^{(0: t)} ), (74) where y and M y are _{t + 1} and M is abbreviated S ^{(0: t + 1)} , scanning boards up to the time step t + 1, S is the scan map at t + 1, o = o _{t + 1} , the measured target is summarized in the time step t + 1 and the last equation follows assuming conditional independence given S ^{(0: t)} and o ^{(0: t)} .

In Equation (74), p (y, M | S ^{(0: t)} , o ^{(0: t)} ) is the previous PDF of y at time step t + 1 for given preceding scan maps S ^{(0: t)} and target data o ^{( 0: t)} , which can be calculated as: p (y | S ^{(0: t)} , o ^{(0: t)} ) = ∫ p (y | y _t ) p (y _t | S ^{(0: t)} , o ^{(0: t)} ) dy _t , ( 75) where p (y _t | S ^{(0: t)} , o ^{(0: t)} )) denotes the subsequent PDF for the transformation parameter at t, and p (y | y _t ) and represents an expression for the propagation model equation (49).

wobei ỹ der vorhergesagte Wert für das folgende Fortpflanzungsmodell des Objekts ỹ = f(y _t) ist.When p (y _t | S ^{(0: t)} ), o ^{(0: t)} )) centers as a Dirac delta distribution y _t is assumed, the previous PDF can be written in time step t + 1 as:

where ỹ is the predicted value for the following propagation model of the object ỹ = f ( y _t ) is.

und wobei

definiert ist in Gleichung (51). Wie aus der 24 zu sehen ist, kann das vorhergehende Netzwerk für das Objektmodell im Zeitschritt t geschrieben werden als:

In the following, the following object model M is assumed as:

and where

is defined in equation (51). Like from the 24 can be seen, the previous network for the object model can be written in time step t as:

Now consider the problem for estimating the following in time step t + 1, which can be decomposed as: p (y, M | S ^{(0: t + 1)} , o ^{(0: t + 1)} ) = p (y | S ^{(0: t + 1)} , o ^{(0: t + 1)} ) p (M | S ^{(0: t + 1)} , o ^{(0: t + 1)} ) (79) and independently calculated as the following two steps.

It is assumed that p (y | S ^{(0: t + 1)} , o ^{(0: t + 1)} ) the Dirac-Delta PDF is centered at y _{t + 1,,} which can be estimated by maximizing equation (74) with respect to y.

The measurement model of the target data p (o | y) is modeled as: o = h (y) + ν, (80) where v is a zero mean Gaussian random variable with the covariance matrix E. As a result, p (o | y) is a Gaussian PDF, ie:

ist.Substituting equations (76) and (81) into equation (74) and applying a negative logarithm to equation (74) and neglecting irrelevant terms at y, the following is obtained: y _{t + 1} = argmin _y - Σ _kj a _kj logK _σ (T -1 / y (s _k ) - m _j ) + || o - h (y) || 2 / E + || y - ỹ || 2 / Q, (82) in which

is.

Thus, equation (38) in algorithm 1 can be given by equation (55) for integrating previous information from previous time steps and for cueing information from the radar sensor 170 or the image system 174 and then applying Algorithm 1 to y _{t + 1} derive.

When calculating p (M | S ^{(o: t + 1)} , 0 ^{(0: t + 1)} ), it is noted that Equation (78) and Equation (53) are equivalent and therefore the same update rules for the above described hyperparameters are applied.

The object file 78 will then be in the box 98 updated in the manner discussed above.

As will be well understood by those skilled in the art, various or some of the steps and methods discussed herein to describe the invention may be performed by a computer, processor, or other electronic processing unit that manipulates and / or manipulates data using electrical phenomena transformed. These computers and electrical devices may include various volatile and / or nonvolatile memories including a fixed computer readable medium having an executable program thereon containing various codes or executable instructions executed by the computer or processor, the memory and / or the computer readable medium may include all forms and types of memory and other computer readable media.

The foregoing discussion shows and describes purely exemplary embodiments of the present invention. One skilled in the art will readily recognize from the discussion of the attached figures and claims that numerous changes, modifications and variations can be made therein without departing from the spirit and scope of the invention as defined by the following claims.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 7991550 [0118]

Claims (10)

A method for combining sensor signals from at least two LiDAR sensors having overlapping fields of view to track objects detected by these sensors, the method comprising: - defining a transformation value for at least one of the LiDAR sensors that identifies an orientation angle and a position of the sensor; Providing target scan points from the objects detected by the sensors, the target scan points providing a separate target scan map for each sensor; Projecting the target scan map from the at least one sensor to the other of the LiDAR sensors using a current transform value to overlap the target scan points from the sensors; Determining a plurality of the weighting values using the current transform value, each weighting value identifying a position change from one of the sampling points for the at least one sensor to a location of a sampling point for the other of the sensors; Calculating a new transformation value using the weighting values; Comparing the new transformation value with the current transformation value to determine a difference between the two; and - revising the plurality of weight values based on the difference between the new transformation value and the current transformation value until the new transformation value matches the current transformation value. The method of claim 1, wherein the LiDAR sensors are on a vehicle. The method of claim 1, further comprising selecting an initial transformation value for the at least one of the sensors initially used as the current transformation value. The method of claim 3, wherein selecting an initial transformation value includes selecting an initial transformation value from the group consisting of a manufacturer default, zero, and an estimated value. The method of claim 1, wherein determining a plurality of weighting values comprises using the equation

where a _{jk is} the weighting value for a given sampling point, S _{j are} the sampling points of the at least one of the LiDAR sensors, m _{k are} the sampling points of the other of the LiDAR sensors, T is the transform value, and K is a kernel function which is defined as

where σ is a variance. The method of claim 1, further comprising determining that the current transformation value is greater than a predetermined threshold, and if the current transformation value is greater than the predetermined threshold, then defining the transformation value based on a rotation matrix and a translation vector. The method of claim 6, wherein defining a transformation value includes using a particle swarm optimization method comprising generating a plurality of particles over a predetermined number of selectable transformation values, associating a normalized weight to each particle, setting a best transformation value for each particle, and Providing a nominal transformation value having a best match percentage, which includes at least one of the LiDAR sensors on the other of the two LiDAR sensors. The method of claim 1, further comprising determining that a change in the transformation value from a sample time to a next sample time is below a predetermined threshold, and if so reducing the computational complexity for determining the current transformation value. The method of claim 8, wherein determining whether the current transformation value is below a predetermined threshold includes modeling the current transformation value as transformations between the sample points. The method of claim 1, wherein defining a transformation value comprises using an expectation maximization algorithm.

Images