CN104751119A - Rapid detecting and tracking method for pedestrians based on information fusion - Google Patents

Abstract

The invention discloses a rapid detecting and tracking method for pedestrians based on information fusion. The method is characterized by comprising the following steps: step S1: fixing a surrounding environment with a fixed frequency by using a laser scanner, so as to obtain laser data; step S2: filtrating ineffective data in the laser data, so as to obtain candidate target data; step S3: calibrating a coordinate parameter between the laser scanner and a camera, so as to obtain a coordinate coordination parameter between two coordinate systems; Step 4: confirming a candidate target based on the candidate target data; step S5: building a real-time tracking model, and tracking the candidate target confirmed by the step S4 through the real-time tracking model. According to the rapid detecting and tracking method provided by the invention, the advantages of a laser sensor and a vision sensor can be integrated to effectively improve the accuracy and timeliness of detecting and tracking the pedestrians by single sensor.

Description

The quick detecting and tracking method of pedestrian based on information fusion

Technical field

The invention belongs to multisensor, technical field of multisource information fusion, relate to Digital Image Processing, pattern-recognition, the aspects of contents such as data correlation, the quick detecting and tracking method of especially a kind of pedestrian based on information fusion, for the important component part in the intelligent vehicles technology---vehicle auxiliary security control loop, to factor the most uncontrollable in driving environment---pedestrian carries out perception, detect, follow the tracks of, analyze, early warning, thus support vehicles driving safety.

Background technology

The intelligent vehicles technology is mainly divided into independent navigation and safety guarantee two aspects by function.The application of autonomous navigation technology depends on the foundation of whole intelligent transportation system (ITS) and perfect, be difficult in a short time reach practical, and safeguard technology can be applied independently in DAS (Driver Assistant System), by making detecting and tracking to the running environment of surrounding, thus the threat that judgement may cause driver, therefore can provide technical support for the traffic hazard solved because of the generation of driver's subjective factor.

Intelligent vehicle safety safeguards technique is divided into safety monitoring and early warning and active safety guarantee, safety monitoring and early warning mainly refer to come monitor vehicle driver condition, vehicle hidden danger, particular surroundings etc. by sensor and warning system, thus help driver to realize safe driving, and be wherein devoted to, by sensor, noncontact detection is carried out to surrounding enviroment for the human body target detection and tracking of vehicle front and improve driving safety.

Merge with laser and visual pattern and carry out preceding object thing detecting and tracking, not only can overcome and be subject to weather conditions and illumination condition variable effect when separately application image visual transducer detects and the shortcoming that cannot obtain the depth information of detecting and tracking object, also can overcome laser ranging cannot disturbance in judgement thing classification, cannot visually process, the shortcoming of redundant warning.

Summary of the invention

The present invention, by merging the depth data obtained from laser scanner and the pictorial information obtained from ccd video camera, realizes the accurate and quick detecting and tracking to pedestrian target, gives to analyze and early warning, thus reaches the object of driver assistance safe driving.The present invention utilizes two kinds of different types of sensors simultaneously, respective advantage is utilized to complete the quick detecting and tracking of road pavement pedestrian, realize having complementary advantages, and farthest can provide correct steered reference fast for driver by the complex condition such as, haze weather poor at night, illumination condition, realize auxiliary security and drive.

For realizing object of the present invention, the quick detecting and tracking method of the pedestrian based on information fusion provided by the invention, makes full use of laser, quick detection and tracking that the advantage of image carries out pedestrian under traffic driving environment in real time, and it comprises the following steps:

Step S1: utilize laser scanner with fixing frequency sweeping surrounding environment, obtain laser data;

Step S2: the invalid data in laser data described in filtering, obtains candidate target data;

Step S3: demarcate for the coordinate parameters between laser scanner and video camera, obtains the coordinate transformation parameter between two coordinate systems;

Step S4: based on described candidate target data, candidate target is wherein confirmed;

Step S5: set up Real-time Tracking Model, and follow the tracks of according to the candidate target that described Real-time Tracking Model confirms for described step S4.

The invention has the beneficial effects as follows: the present invention makes full use of the advantage of combining under traffic scene in pedestrian's detecting and tracking of laser sensor and vision sensor, it instead of concrete Theories and methods involved in Conventional visual, meanwhile, reliable solution has been had for complicated traffic, complicated occlusion issue.Compared with traditional single-sensor detecting and tracking method, the present invention all has qualitative leap in real-time and accuracy etc.

Accompanying drawing explanation

Fig. 1 is the quick detecting and tracking method flow diagram of pedestrian that the present invention is based on information fusion;

Fig. 2 a is the laser data set of non-filtering invalid data, and Fig. 2 b is the candidate target data acquisition obtained after filtering invalid data;

Fig. 3 is that parameter scheme of installation corrected by laser scanner and video camera according to an embodiment of the invention;

Fig. 4 is that candidate region generates result schematic diagram according to an embodiment of the invention;

Fig. 5 a is HOG feature schematic diagram according to an embodiment of the invention, and Fig. 5 b is human detection schematic diagram according to an embodiment of the invention;

Fig. 6 divides observation space schematic diagram according to an embodiment of the invention;

Fig. 7 is the detecting and tracking result schematic diagram utilizing the inventive method.

Embodiment

For making the object, technical solutions and advantages of the present invention clearly understand, below in conjunction with specific embodiment, and with reference to accompanying drawing, the present invention is described in more detail.

Fig. 1 is the process flow diagram of the quick detecting and tracking method of pedestrian that the present invention is based on information fusion, as shown in Figure 1, said method comprising the steps of:

Step S1: utilize laser scanner with fixing frequency sweeping surrounding environment, obtain laser data;

In an embodiment of the present invention, the laser scanner selected is two dimensional laser scanning instrument SICKLMS291, and its sweep limit is 100 °, and angular resolution is 0.25 °, and distance range is within 80 meters.

The data that described two dimensional laser scanning instrument returns are discrete data points of one group of limited length in its two-dimensional scan plane, these data can reflect geometric position and the shape of surrounding objects, wherein, each data can also represent the distance on correspondence direction and recently between target.Concrete discrete data is counted relevant with the angular resolution of laser scanner, and in an embodiment of the present invention, return 400 discrete datas, these discrete datas are provided by polar form, that is: l at every turn _z=(d _z, φ _z) ^t, z=1...400, can be expressed as in cartesian coordinate system:

u _z＝(x _z,y _z) ^T,z＝1...400，

Wherein, x _z=d _zcos φ _z, y _z=d _zsin φ.

Step S2: the invalid data in laser data described in filtering, obtains candidate target data;

Because the object of laser scanner to unlike material, different colours exists measuring error, therefore invalid data may be there is in the laser data returned, and in invalid data, show as null value and the value exceeding maximum measure distance scope greatly, also have because of solar irradiation, vehicle jolts and reflecting object material causes noise spot in addition, by filtering, this part invalid data is removed.

Described step S2 is further comprising the steps:

Order set L={l _zrepresent one group of laser data, wherein, z=1 ..., Z, Z represent that laser data is counted.

First, will gather L and core template [-1,1] carries out convolution, retention point spacing data point within the specific limits, obtains thick denoising result set: C={c _n, n=1 ... .N, wherein, N represents that in thick denoising result set, laser data is counted;

Then, for thick denoising result set carry out cluster operation, obtain one group of candidate target data acquisition S.

Wherein, described cluster operation is specially:

Cluster is initial, using each data point as a class, calculates the distance between new class and single sample number strong point, if adjacent two class c _n-1and c _nbetween spacing belong in a certain predetermined threshold range, then think that they belong to same class, otherwise just think and belong to inhomogeneity, and using current single sample number strong point as the starting point of newly-increased class.

In the process of cluster, a large amount of invalid laser data points is excluded, and finally obtains the one group candidate target data acquisition S={s of representation class like barrier _m, wherein, m=1 ... .M, represents the quantity of candidate target.These candidate targets on lateral length with detected target class seemingly, its feature includes the degree of depth, length and positional information.

Fig. 2 a is the laser data set of non-filtering invalid data, and Fig. 2 b is the candidate target data acquisition obtained after filtering invalid data.

Step S3: the coordinate parameters between laser scanner and video camera is demarcated, obtain the coordinate transformation parameter (φ, Δ) between two coordinate systems, wherein φ is rotation matrix, and Δ is translation vector;

On the basis of this step scaling method of plane of motion template in laser scanner and image calibration technology, add the laser coordinate system (two-dimensional scan face) at the place of laser scanner own, the method that its core is to utilize scaling board to be caught by laser scanner and video camera is simultaneously to try to achieve rotation between laser coordinate system and camera coordinate system and translation matrix.

The installation site of laser scanner and video camera when Fig. 3 shows parameter calibration, in figure, X _fy _fz _frepresent the three-dimensional coordinate plane of laser scanner, X _cy _cz _crepresent the three-dimensional coordinate of video camera.From camera coordinate system p to the transformation relation of laser coordinate system pf as shown in (1) formula, wherein, φ is the rotation matrix that camera coordinate system transforms to laser coordinate system, represents the relative direction between laser scanner and video camera; Δ is the translation vector transforming to laser coordinate system from camera coordinate system, represents relative position between the two:

p ^f＝φp+Δ， (1)

Under camera coordinate system, scaling board parameter is turned to a tri-vector N, the direction of this vector is parallel to the normal direction of scaling board, || N|| equals the distance of video camera to scaling board.Under camera coordinate system, get any point p on scaling board, because p is positioned at parameterized on the scaling board of N, then have:

N.p＝||N|| ²， (2)

Wherein, N can be obtained by the known outer parameter [R, t] (R is the orthogonal rotation matrix of external parameters of cameras, and t is the translation matrix of external parameters of cameras) of video camera:

N=-r ₃(r ₃ ^t.t) (3) wherein, r ₃represent the 3rd column vector of the orthogonal rotation matrix R of external parameters of cameras.

By converting the position of scaling board, a different set of vectorial N and corresponding laser spots can be obtained at p ^funder position coordinates, namely obtain one group of constraint condition:

N.φ ^-1(p ^f-Δ)＝||N|| ²(4)

Solve this equation can try to achieve rotation between camera coordinate system and laser coordinate system and translation relation √ and this completes the combined calibrating of laser scanner and video camera.As shown in Figure 4, wherein, Fig. 4 a is the imaging of object under camera coordinate system to projection result, and wherein numeral 1 to 6 represents six objects; Fig. 4 b is the imaging of object under laser coordinate system, and Fig. 4 c is the position coordinates of object under three-dimensional coordinate.

Step S4: based on described candidate target data, candidate target is wherein confirmed;

After acquisition candidate target data, need to utilize the region of the method for vision-based detection to candidate target data place to detect, to carry out the detection of confirmatory to candidate target.Need to train different sorters (as human body, vehicle etc.) for different target in vision-based detection part.

Described step S4 is further comprising the steps:

Step S41, by training sample image (can be such as the human body training sample of 64 × 128 pixels) according to pre-sizing, such as 8 × 8, block of pixels be divided into several elementary cells (cell);

Step S42, an often adjacent m unit (such as with 4 unit that field word structure is adjacent) is divided into a region unit block, by (-90 °, 90 °) gradient direction on average divide, specify that every 20 degree is a basic orientation bin, can obtain 9 basic orientation bins within the scope of the direction of 180 °;

Step S43, for each elementary cell cell, carries out projecting to set up respective gradient orientation histogram by wherein all pixels on all basic orientation bins;

Step S44, couples together the gradient orientation histogram of contain in each block 4 elementary cell cell, obtains the vector of 36 dimensions;

Step S45, then be concatenated after the 36 dimensional vector normalization of all region unit block, obtain the HOG proper vector of each training sample;

Fig. 5 a extracts the HOG feature schematic diagram obtained, and that reflects the human body contour outline under two different attitudes.

Step S46, extracts after obtaining the HOG proper vector of training sample image, obtains a svm classifier model for target detection based on extracting the HOG proper vector training obtained;

Step S47, for certain candidate region in training sample image described in camera coordinate system, judges whether there is target in this region by the mode scanned by pixel region multi-scale image.

By pixel region multi-scale image method: as shown in Figure 5 b, in the region of a 192*96 pixel size, scanned by the basic templates of 128*64 pixel size, scan through mobile basic templates line by line at every turn, and whether there is target under judging this region by SVM classifier.

Step S5: set up Real-time Tracking Model, and follow the tracks of according to the candidate target that described Real-time Tracking Model confirms for described step S4.

The model that the present invention proposes is started with from maximum a posteriori probability problem.Suppose, in scene, the observation vector sum total of each target is: T=(T ₁, T ₂..., T _n), and wherein the observation vector of each target is expressed as: T _i={ p _i, o _i, v _i, f _i, wherein, p _irepresent the position coordinates of target; o _irepresent the direction of motion of target; v _irepresent the movement velocity of target; f _irepresent the external performance (as color histogram, direction gradient etc.) of target.Pursuit path for same target is estimated to use a series of path segment set to represent, such as S={s _k, wherein, s _krepresent path segment.

For given observation vector T, the target of data correlation maximizes posterior probability P (S|T) exactly, that is:

$S^{*}=\underset{S}{\arg \max }P\left(S\right|T)=\underset{S}{\arg \max }P(T/S)P\left(S\right).---\left(5\right)$

Wherein, T is given observation vector, and S is the set of candidate target path segment.

Because estimate that the space of S is very huge, be difficult to direct optimized-type (5).Consider that a target only can belong to unique path segment, therefore, this characteristic can be utilized to effectively reduce the size of search volume, namely the constraint condition of this characteristic equivalence can be expressed as a target can not multiple track label simultaneously:

Wherein, T _krepresent the observation vector of a kth target in scene, T represents the observation vector sum total of each target in scene.

According to the relation of target being observed, candidate target path segment S set can be divided into two parts, such as shown in Figure 6 a, observed object is divided into pinpoint target (marking by dashed rectangle) and joint objective (marking with dot-and-dash line square frame), Fig. 6 b is depicted as the position coordinates of observed object under laser scanner.

Then formula (5) can be write as further:

$S^{*}=\underset{S}{\arg }\max P\left(S^{\mathrm{\α}+\mathrm{\β}}\right)P\left(T\right|S^{\mathrm{\α}+\mathrm{\β}}),---\left(7\right)$

$S^{*}\≈\underset{S}{\arg \max }\mathrm{\ΠP}\left(S^{\mathrm{\α}}\right)\mathrm{\ΠP}\left(T\right|S^{\mathrm{\α}})+\mathrm{\ΠP}\left(S^{\mathrm{\β}}\right)\mathrm{\ΠP}\left(T\right|S^{\mathrm{\β}}).---\left(8\right)$

Wherein, by path segment S set ^alpha+betabe divided into the region S with joint objective track ^αwith pinpoint target track regions S ^β.

Then do further hypothesis: the motion between target and target is separate, then formula (8) can be rewritten as:

$\begin{array}{c}S=\underset{S^{\mathrm{\α}}}{\arg \min }\underset{i,j}{\mathrm{\Σ}}{C}_{j,i}{t}_{j,i}+\underset{i}{\mathrm{\Σ}}{C}_j{t}_i+\underset{S^{\mathrm{\β}}}{\arg \min }\underset{i,j}{\mathrm{\Σ}}{C}_{j,i}{t}_{j,i}+\underset{i}{\mathrm{\Σ}}{C}_i{t}_i\\ =\underset{S^{\mathrm{\α}}}{\arg \min }\underset{i,j}{\mathrm{\Σ}}{C}_{j,i}{t}_{j,i}+\underset{i}{\mathrm{\Σ}}{C}_i{t}_i+\underset{S^{\mathrm{\β}}}{\arg \min }\underset{i,j}{\mathrm{\Σ}}{C}_{j,i}{t}_{j,i}\end{array}---\left(9\right)$

Parameter wherein can be expressed as:

$t_i=\underset{j}{\mathrm{\Σ}}{t}_{j,i}\≤1,---\left(10\right)$

$C_{j,i}=-\log P_{\mathrm{sim}}\left(T_{j_{k-1}}\right|T_{i_k})---\left(11\right)$

$C_i=-\log P\left(T_i\right|S)-\log \frac{p_i}{1-p_i}=\log \frac{1-p_i}{p_i}.---\left(12\right)$

Wherein, t _j,iand t _irepresent the two-value indieating variable of 0,1, for t _j,iif equal 1, then represent observation track T _jwith T _ibe connected, if equal 0, then represent and do not connect.For indieating variable t _iif equal 1, represent that this observation track is in associating track regions, vice versa.C _j,irepresent observation track T _jwith T _ibetween transfer similarity cost value, P _simi () represents the similarity of two sections of tracks, by the external performance f of target _ithe Euclidean distance of vector is tried to achieve, C _irepresent observation track T _jdegree of depth cost value, wherein p _irepresent actual grade value.

Formula (9) is the target equation of Real-time Tracking Model, and the present invention uses the Bipartite Matching method of improvement to solve this problem.The Bipartite Matching method improved belongs to the conventional derivation algorithm in this area, and therefore not to repeat here.

Experiment proves, in continuous print frame of video, the inventive method can trace into each target exactly.As shown in Figure 7, as can be seen from Figure 7, in different traffic scenes, even if blocking repeatedly appears in pedestrian, the present invention still can identify each pedestrian to the detecting and tracking result obtained according to the present invention.

The present invention well combines the advantage of laser sensor and vision sensor, effectively improves single-sensor and carries out the degree of accuracy of detecting and tracking and ageing to pedestrian.By laser spots filtering, cluster and laser spots pattern analysis complete the Primary Location to pedestrian, and the potential site of pedestrian is projected to image layer by the Registration of Measuring Data then between laser scanner and video camera; And then utilize quick pedestrian detection technology to complete pedestrian position to determine and carry out feature extraction, finally between continuous multiple frames, carrying out data correlation, reach the target to the real-time detecting and tracking of pedestrian.The road environment of algorithm to surrounding makes analysis, and judges the threat that may cause vehicle safe driving, carries out danger early warning, realizes the function that auxiliary security is driven.

Above-described specific embodiment; object of the present invention, technical scheme and beneficial effect are further described; be understood that; the foregoing is only specific embodiments of the invention; be not limited to the present invention; within the spirit and principles in the present invention all, any amendment made, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.

Claims (9)

1., based on the quick detecting and tracking method of pedestrian of information fusion, it is characterized in that, the method comprises the following steps:

Step S1: utilize laser scanner with fixing frequency sweeping surrounding environment, obtain laser data;

Step S2: the invalid data in laser data described in filtering, obtains candidate target data;

Step S3: demarcate for the coordinate parameters between laser scanner and video camera, obtains the coordinate transformation parameter between two coordinate systems;

Step S4: based on described candidate target data, candidate target is wherein confirmed;

Step S5: set up Real-time Tracking Model, and follow the tracks of according to the candidate target that described Real-time Tracking Model confirms for described step S4.

2. method according to claim 1, is characterized in that, described laser scanner is two dimensional laser scanning instrument.

3. method according to claim 1, is characterized in that, order set L={l _zrepresent one group of laser data, wherein, z=1 ..., Z, Z represent that laser data is counted, then described step S2 is further comprising the steps:

First, will gather L and core template [-1,1] carries out convolution, retention point spacing data point within the specific limits, obtains thick denoising result set: C={c _n, n=1 ... .N, wherein, N represents that in thick denoising result set, laser data is counted;

Then, for thick denoising result set carry out cluster operation, obtain one group of candidate target data acquisition S.

4. method according to claim 3, is characterized in that, described cluster operation is specially:

Cluster is initial, using each data point as a class, calculates the distance between new class and single sample number strong point, if adjacent two class c _n-1and c _nbetween spacing belong in a certain predetermined threshold range, then think that they belong to same class, otherwise just think and belong to inhomogeneity, and using current single sample number strong point as the starting point of newly-increased class.

5. method according to claim 1, is characterized in that, in described step S3, the method utilizing scaling board simultaneously to be caught by laser scanner and video camera is to try to achieve rotation between laser coordinate system and camera coordinate system and translation matrix.

6. method according to claim 1, is characterized in that, described step S4 is further comprising the steps:

Training sample image is several elementary cells according to predetermined size pixel block comminute by step S41;

Step S42, is a region unit block by an often adjacent m dividing elements, is on average divided by gradient direction, obtains multiple basic orientation bins within the scope of the direction of 180 °;

Step S43, for each elementary cell, carries out projecting to set up respective gradient orientation histogram by wherein all pixels in all basic orientation;

Step S44, couples together the gradient orientation histogram of the elementary cell contained in each region unit, obtains vector;

Step S45, then be concatenated after the vectorial normalization of all region units, obtain the HOG proper vector of each training sample;

Step S46, obtains a svm classifier model based on extracting the HOG proper vector training obtained;

Step S47, for certain candidate region in training sample image described in camera coordinate system, by judging whether there is target in this region by the mode in pixel region scanning multi-scale image region.

7. method according to claim 1, is characterized in that, described Real-time Tracking Model is expressed as:

$S^{*}=\underset{S}{\arg \max }P\left(S\right|T)=\underset{S}{\arg \max }P\left(T\right|S)P\left(S\right),$

Wherein, T is given observation vector, and S is candidate target set.

8. method according to claim 7, is characterized in that, described Real-time Tracking Model is also provided with constraint condition:

Wherein, T _krepresent the observation vector of a kth target in scene, T represents the observation vector sum total of each target in scene.

9. method according to claim 1, is characterized in that, the target equation of described Real-time Tracking Model is expressed as:

$\begin{array}{c}S=\underset{S^{\mathrm{\α}}}{\arg \min }\underset{i,j}{\mathrm{\Σ}}{C}_{j,i}{t}_{j,i}+\underset{i}{\mathrm{\Σ}}{C}_{j,i}{t}_{j,i}+\underset{i}{\mathrm{\Σ}}{C}_j{t}_i+\underset{S^{\mathrm{\β}}}{\arg \min }\underset{i,j}{\mathrm{\Σ}}{C}_{j,i}{t}_{j,i}+\underset{i}{\mathrm{\Σ}}{C}_i{t}_i\\ =\underset{S^{\mathrm{\α}}}{\arg \min }\underset{i,j}{\mathrm{\Σ}}{C}_{j,i}{t}_{j,i}+\underset{i}{\mathrm{\Σ}}{C}_i{t}_i+\underset{S^{\mathrm{\β}}}{\arg \min }\underset{i,j}{\mathrm{\Σ}}{C}_{j,i}{t}_{j,i}\end{array},$

Wherein, S ^αrepresent the region with joint objective track; S ^βrepresent pinpoint target track regions; t _j,iand t _irepresent two-value indieating variable, represent observation track T _jwith T _ibetween transfer similarity cost value, C _j,i=-logP _sim(T _jk-1| T _ik), C _irepresent observation track T _jdegree of depth cost value

$C_i=-\log P\left(T_i\right|S)-\log \frac{p_i}{1-p_i}=\log \frac{1-p_i}{p_i}.$