CN104751119A - Rapid detecting and tracking method for pedestrians based on information fusion - Google Patents

Abstract

The invention discloses a fast detection and tracking method for pedestrians based on information fusion, which is characterized in that the method comprises the following steps: Step S1: use a laser scanner to scan the surrounding environment at a fixed frequency to obtain laser data; Step S2: filter out Invalid data in the laser data is used to obtain candidate target data; step S3: calibrate the coordinate parameters between the laser scanner and the camera to obtain coordinate conversion parameters between the two coordinate systems; step S4: based on the candidate Target data, confirming the candidate targets therein; Step S5: Establish a real-time tracking model, and track the candidate targets confirmed in the step S4 according to the real-time tracking model. The invention perfectly integrates the advantages of the laser sensor and the visual sensor, and effectively improves the accuracy and timeliness of the detection and tracking of pedestrians by a single sensor.

Description

Fast pedestrian detection and tracking method based on information fusion

technical field

The invention belongs to the field of multi-sensor and multi-source information fusion technology, and relates to digital image processing, pattern recognition, data association and other aspects, especially a fast detection and tracking method for pedestrians based on information fusion, which is used in the important technology of intelligent vehicles. The component—vehicle assisted safety driving system—perceives, detects, tracks, analyzes and warns the most uncontrollable factor in the driving environment—pedestrians, so as to ensure the safety of vehicles.

Background technique

Intelligent vehicle technology is mainly divided into two aspects: autonomous navigation and safety assurance. The application of autonomous navigation technology depends on the establishment and improvement of the entire intelligent transportation system (ITS), and it is difficult to achieve practical application in the short term. However, safety assurance technology can be independently applied to the assisted driving system. By detecting and tracking the surrounding driving environment, In order to judge the possible threat to the driver, it can provide technical support for solving traffic accidents caused by the driver's subjective factors.

Intelligent vehicle safety assurance technology is divided into safety monitoring and early warning and active safety assurance. Safety monitoring and early warning mainly refers to monitoring the status of vehicle drivers, vehicle hidden dangers, special environments, etc. by means of sensors and alarm systems, so as to help drivers achieve safe driving. Among them, the detection and tracking of human targets in front of the vehicle is dedicated to non-contact detection of the surrounding environment through sensors to improve driving safety.

Using laser and visual image fusion to detect and track obstacles in front can not only overcome the shortcomings of being easily affected by changes in weather conditions and lighting conditions and not being able to obtain depth information for detection and tracking objects when visual image sensors are used alone for detection, but also can overcome the shortcomings of laser measurement. The distance cannot judge the obstacle type, cannot be processed visually, and has the disadvantages of redundant alarms.

Contents of the invention

The present invention realizes the accurate and fast detection and tracking of pedestrian targets by fusing the depth data obtained from the laser scanner and the picture information obtained from the CCD camera, and provides analysis and early warning, so as to achieve the purpose of assisting the driver to drive safely. The present invention utilizes two different types of sensors at the same time, utilizes their respective advantages to complete rapid detection and tracking of pedestrians on the road surface, realizes complementary advantages, and can provide drivers with maximum protection under complex conditions such as nighttime, poor lighting conditions, and foggy weather. Provide correct and fast driving reference to realize assisted safe driving.

In order to realize the purpose of the present invention, the method for rapid detection and tracking of pedestrians based on information fusion provided by the present invention fully utilizes the advantages of laser and image to perform rapid detection and tracking of pedestrians in the traffic driving environment in real time, which includes the following steps:

Step S1: Use a laser scanner to scan the surrounding environment at a fixed frequency to obtain laser data;

Step S2: filtering out invalid data in the laser data to obtain candidate target data;

Step S3: Calibrate the coordinate parameters between the laser scanner and the camera to obtain coordinate conversion parameters between the two coordinate systems;

Step S4: Based on the candidate target data, confirm the candidate targets therein;

Step S5: Establish a real-time tracking model, and track the candidate targets confirmed in step S4 according to the real-time tracking model.

The beneficial effects of the present invention are: the present invention makes full use of the advantages of the combination of laser sensors and visual sensors in the detection and tracking of pedestrians in traffic scenes, which replaces the specific theories and methods involved in traditional vision, and at the same time, for complex traffic roads, There is a reliable solution to the complex occlusion problem. Compared with the traditional single-sensor detection and tracking method, the present invention has a qualitative leap in terms of real-time performance and accuracy.

Description of drawings

Fig. 1 is the flow chart of the pedestrian rapid detection and tracking method based on information fusion in the present invention;

Figure 2a is the laser data set without filtering out invalid data, and Figure 2b is the candidate target data set obtained after filtering out invalid data;

Fig. 3 is a schematic diagram of installation of laser scanner and camera correction parameters according to an embodiment of the present invention;

Fig. 4 is a schematic diagram of candidate region generation results according to an embodiment of the present invention;

Fig. 5a is a schematic diagram of HOG features according to an embodiment of the present invention, and Fig. 5b is a schematic diagram of human body detection according to an embodiment of the present invention;

Fig. 6 is a schematic diagram of dividing an observation space according to an embodiment of the present invention;

Fig. 7 is a schematic diagram of detection and tracking results using the method of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

Fig. 1 is the flow chart of the pedestrian quick detection and tracking method based on information fusion of the present invention, as shown in Fig. 1, described method comprises the following steps:

Step S1: Use a laser scanner to scan the surrounding environment at a fixed frequency to obtain laser data;

In one embodiment of the present invention, the selected laser scanner is a two-dimensional laser scanner SICKLMS291, with a scanning range of 100°, an angular resolution of 0.25°, and a distance within 80 meters.

The data returned by the two-dimensional laser scanner is a set of discrete data points with a limited length in its two-dimensional scanning plane, and these data can reflect the geometric position and shape of surrounding objects, wherein each data can also represent The distance to the closest target. The number of specific discrete data points is related to the angular resolution of the laser scanner. In an embodiment of the present invention, 400 discrete data are returned each time, and these discrete data are given by polar coordinates, that is: _lz =( _dz , φ _z ) ^T , z=1...400, can be expressed in the Cartesian coordinate system as:

u _z =(x _z ,y _z ) ^T ,z=1...400,

Among them, x _z =d _z cosφ _z , y _z =d _z sinφ.

Step S2: filtering out invalid data in the laser data to obtain candidate target data;

Due to the measurement errors of laser scanners for objects of different materials and colors, invalid data may appear in the returned laser data, and a large part of the invalid data appears as zero values and values beyond the maximum range , In addition, there are noise points caused by sunlight radiation, vehicle bumps and reflective object materials, which can be removed by filtering.

Said step S2 further comprises the following steps:

Let the set L={l _z } represent a set of laser data, where z=1,...,Z, Z represents the number of laser data points.

First, the set L is convolved with the kernel template [-1,1], and the data points within a certain range are retained to obtain a rough denoising result set: C={c _n },n=1,... .N, where N represents the number of laser data points in the rough denoising result set;

Then, for the coarse denoising result set Perform a clustering operation to obtain a set of candidate target data sets S.

Wherein, the clustering operation is specifically:

In the initial clustering, each data point is regarded as a class, and the distance between the new class and a single sample data point is calculated. If the distance between two adjacent classes c _n-1 and c _n falls within a certain predetermined threshold range, They are considered to belong to the same class, otherwise they are considered to belong to different classes, and the current single sample data point is used as the starting point of the new class.

In the process of clustering, a large number of invalid laser data points are excluded, and finally a set of candidate target data sets S={s _m } representing similar obstacles are obtained, where m=1,....M, represents The number of candidate targets. These candidate targets are similar to the detected targets in horizontal length, and their characteristics include depth, length and position information.

Figure 2a is the laser data set without filtering out invalid data, and Figure 2b is the candidate target data set after filtering out invalid data.

Step S3: Calibrate the coordinate parameters between the laser scanner and the camera to obtain the coordinate transformation parameters (φ, Δ) between the two coordinate systems, where φ is the rotation matrix, and Δ is the translation vector;

This step is based on the calibration method of the mobile plane template in the laser scanner and image calibration technology, and the laser coordinate system (two-dimensional scanning surface) where the laser scanner itself is located is added. Simultaneous capture method to obtain the rotation and translation matrix between the laser coordinate system and the camera coordinate system.

Figure 3 shows the installation positions of the laser scanner and the camera during parameter calibration. In the figure, X _f Y _f Z _f represents the three-dimensional coordinate plane of the laser scanner, and X _c Y _c Z _c represents the three-dimensional coordinates of the camera. The transformation relationship from the camera coordinate system p to the laser coordinate system pf is shown in formula (1), where φ is the rotation matrix from the camera coordinate system to the laser coordinate system, which represents the relative direction between the laser scanner and the camera; Δ is the translation vector transformed from the camera coordinate system to the laser coordinate system, representing the relative position between the two:

p ^f =φp+Δ, (1)

In the camera coordinate system, the calibration plate is parameterized as a three-dimensional vector N, the direction of this vector is parallel to the normal direction of the calibration plate, and ||N|| is equal to the distance from the camera to the calibration plate. Take any point p on the calibration board in the camera coordinate system, since p is located on the calibration board parameterized as N, then:

Np＝||N|| ² , (2)

Among them, N can be obtained by the known external parameters [R, t] of the camera (R is the orthogonal rotation matrix of the external parameters of the camera, and t is the translation matrix of the external parameters of the camera):

N=-r ₃ (r ₃ ^T .t) (3) where, r ₃ represents the third column vector of the camera extrinsic parameter orthogonal rotation matrix R.

By changing the position of the calibration plate, a set of different vectors N and the position coordinates of the corresponding laser points under p ^f can be obtained, that is, a set of constraints can be obtained:

N.φ ^-1 (p ^f -Δ)＝||N|| ² (4)

Solving this equation can obtain the rotation and translation relationship between the camera coordinate system and the laser coordinate system √ and This completes the joint calibration of the laser scanner and the camera. The projection results are shown in Figure 4, where Figure 4a is the imaging of the object in the camera coordinate system, where numbers 1 to 6 represent six targets; Figure 4b is the imaging of the object in the laser coordinate system, and Figure 4c is the object in the three-dimensional coordinate system The location coordinates below.

Step S4: Based on the candidate target data, confirm the candidate targets therein;

After the candidate target data is obtained, it is necessary to use the method of visual detection to detect the area where the candidate target data is located, so as to confirm the detection of the candidate target. In the visual detection part, different classifiers need to be trained for different targets (such as human body, vehicle, etc.).

Said step S4 further comprises the following steps:

Step S41, the training sample image (such as a human body training sample of 64×128 pixels) is divided into several basic units (cells) according to a predetermined size, such as 8×8, of pixel blocks;

Step S42, divide each adjacent m units (for example, 4 adjacent units in a cross-shaped structure) into an area block, and divide the gradient direction of (-90°, 90°) equally, and stipulate that every 20 degrees As a basic direction bin, 9 basic direction bins can be obtained within the direction range of 180°;

Step S43, for each basic unit cell, project all pixels therein on all basic direction bins to establish respective gradient direction histograms;

Step S44, connect the gradient orientation histograms of the 4 basic unit cells contained in each block to obtain a 36-dimensional vector;

Step S45, normalizing and concatenating the 36-dimensional vectors of all the block blocks to obtain the HOG feature vector of each training sample;

Figure 5a is a schematic diagram of the extracted HOG features, which reflect the contours of the human body in two different poses.

Step S46, after extracting the HOG feature vector of the training sample image, training based on the extracted HOG feature vector to obtain an SVM classification model for target detection;

Step S47 , for a certain candidate area on the training sample image in the camera coordinate system, it is judged whether there is a target in the area by means of pixel-by-pixel area multi-scale image scanning.

Pixel-by-pixel area multi-scale image method: as shown in Figure 5b, in an area with a size of 192*96 pixels, scan through a basic template with a size of 128*64 pixels, and each scan moves the basic template row by row, And use the SVM classifier to judge whether there is a target under this area.

Step S5: Establish a real-time tracking model, and track the candidate targets confirmed in step S4 according to the real-time tracking model.

The model proposed by the present invention starts with the problem of maximum a posteriori probability. Assume that the total observation vector of each target in the scene is: T=(T ₁ ,T ₂ ,...,T _n ), and the observation vector of each target is expressed as: T _i ={p _i ,o _i ,v _i ,f _i }, where p _i represents the position coordinates of the target; o _i represents the moving direction of the target; _{v i represents} the moving speed of the target; ). The tracking trajectory estimation for the same target can be represented by a series of trajectory segment sets, for example, S={s _k }, where s _k represents the trajectory segment.

For a given observation vector T, the goal of data association is to maximize the posterior probability P(S|T), namely:

${\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

Among them, T is a given observation vector, and S is a set of candidate target trajectory segments.

Because the estimated space of S is very large, it is difficult to directly optimize Equation (5). Considering that a target will only belong to a unique track segment, this feature can be used to effectively reduce the size of the search space, that is, the equivalent constraint of this feature can be expressed as a target cannot have multiple track labels at the same time:

Among them, T _k represents the observation vector of the kth target in the scene, and T represents the sum of the observation vectors of each target in the scene.

According to the relationship of the observed target, the candidate target trajectory segment set S can be divided into two parts. For example, as shown in Fig. box marked), Figure 6b shows the position coordinates of the observation target under the laser scanner.

Then formula (5) can be further written as:

${\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}}\mathrm{<span\; class="notranslate">\; max</span>}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 7</span>}<span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; \&ap;</span>\underset{\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; \&Pi;P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; \&Pi;P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&Pi;P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; \&Pi;P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 8</span>}<span\; class="notranslate">\; )</span>$

Among them, the track segment set S ^α+β is divided into a region S ^α with joint target trajectories and an area S ^β with independent target trajectories.

Then make a further assumption: the movement between the target and the target is independent of each other, then formula (8) can be rewritten as:

$\begin{array}{c}\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; min</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; +</span>\underset{{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; min</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ <span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; min</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; +</span>\underset{{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; min</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}\end{array}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 9</span>}<span\; class="notranslate">\; )</span>$

The parameters can be expressed as:

${\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 10</span>}<span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; log</span>}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; sim</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{{\mathrm{<span\; class="notranslate">\; j</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{{\mathrm{<span\; class="notranslate">\; i</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 11</span>}<span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; log</span>}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; log</span>}\frac{{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; log</span>}\frac{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; .</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 12</span>}<span\; class="notranslate">\; )</span>$

Among them, t _j,i and t _i represent binary indicator variables of 0,1. For t _j,i , if it is equal to 1, it means that the observed trajectory T _j is connected to T _i , and if it is equal to 0, it means that it is not connected. For the indicator variable t _i , if it is equal to 1, it means that the observed track is in the joint track area, and vice versa. C _j,i represents the transfer similarity cost value between the observed trajectory T _j and T _i , P _sim (i) represents the similarity of the two trajectories, which is obtained from the Euclidean distance of the target's appearance feature f _i vector, C _i Indicates the depth cost value of the observed trajectory T _j , where p _i represents the actual depth value.

Formula (9) is the target equation of real-time tracking model, and the present invention uses improved bipartite graph matching method to solve this problem. The improved bipartite graph matching method is a commonly used solution algorithm in this field, and will not be described in detail here.

Experiments have proved that in continuous video frames, the method of the present invention can accurately track each target. The detection and tracking results obtained according to the present invention are shown in FIG. 7 . It can be seen from FIG. 7 that in different traffic scenes, even if pedestrians are occluded multiple times, the present invention can still identify each pedestrian.

The invention well integrates the advantages of the laser sensor and the vision sensor, and effectively improves the accuracy and timeliness of a single sensor for detecting and tracking pedestrians. The preliminary positioning of pedestrians is completed through laser point filtering, clustering and laser point pattern analysis, and then the potential position of pedestrians is projected to the image layer through data registration between the laser scanner and the camera; and then the fast pedestrian detection technology is used to complete Pedestrian position is determined and feature extraction is performed, and finally data association is performed between consecutive multiple frames to achieve the goal of real-time detection and tracking of pedestrians. The algorithm analyzes the surrounding road surface environment, and judges the possible threats to the safe driving of the vehicle, carries out danger warning, and realizes the function of assisting safe driving.

The specific embodiments described above have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (9)

1. A fast detection and tracking method for pedestrians based on information fusion, characterized in that the method may further comprise the steps: Step S1: Use a laser scanner to scan the surrounding environment at a fixed frequency to obtain laser data; Step S2: filtering out invalid data in the laser data to obtain candidate target data; Step S3: Calibrate the coordinate parameters between the laser scanner and the camera to obtain coordinate conversion parameters between the two coordinate systems; Step S4: Based on the candidate target data, confirm the candidate targets therein; Step S5: Establish a real-time tracking model, and track the candidate targets confirmed in step S4 according to the real-time tracking model. 2. The method according to claim 1, wherein the laser scanner is a two-dimensional laser scanner. 3. method according to claim 1, it is characterized in that, let set L={l _z } represent a group of laser data, wherein, z=1,..., Z, Z represent laser data point number, then described step S2 further comprising the following steps: First, the set L is convolved with the kernel template [-1,1], and the data points with a point spacing within a certain range are retained to obtain a rough denoising result set: C={c _n },n=1,... .N, where N represents the number of laser data points in the rough denoising result set; Then, for the coarse denoising result set Perform a clustering operation to obtain a set of candidate target data sets S. 4. The method according to claim 3, wherein the clustering operation is specifically: In the initial clustering, each data point is regarded as a class, and the distance between the new class and a single sample data point is calculated. If the distance between two adjacent classes c _n-1 and c _n falls within a certain predetermined threshold range, It is considered that they belong to the same class, otherwise they are considered to belong to different classes, and the current single sample data point is used as the starting point of the new class. 5. The method according to claim 1, characterized in that, in the step S3, the rotation and translation between the laser coordinate system and the camera coordinate system are obtained by using the method that the calibration plate is captured simultaneously by the laser scanner and the camera matrix. 6. The method according to claim 1, wherein said step S4 further comprises the following steps: Step S41, dividing the training sample image into several basic units according to predetermined size pixel blocks; Step S42, divide each adjacent m units into a block, divide the gradient direction on average, and obtain multiple basic direction bins within the direction range of 180°; Step S43, for each basic unit, project all pixels in all basic directions to establish their respective gradient direction histograms; Step S44, connect the gradient orientation histograms of the basic units contained in each block to obtain a vector; Step S45, normalizing and concatenating the vectors of all the regional blocks to obtain the HOG feature vector of each training sample; Step S46, training an SVM classification model based on the extracted HOG feature vector; Step S47, for a certain candidate area on the training sample image in the camera coordinate system, it is judged whether there is a target in the area by scanning the multi-scale image area pixel by pixel. 7. The method according to claim 1, wherein the real-time tracking model is expressed as: ${\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$ Among them, T is a given observation vector, and S is a set of candidate targets. 8. The method according to claim 7, wherein the real-time tracking model is also provided with constraints: Among them, T _k represents the observation vector of the kth target in the scene, and T represents the sum of the observation vectors of each target in the scene. 9. method according to claim 1, is characterized in that, the objective equation of described real-time tracking model is expressed as: $\begin{array}{c}\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; min</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; +</span>\underset{{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; min</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ <span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; min</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; +</span>\underset{{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; min</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}\end{array}<span\; class="notranslate">\; ,</span>$ Among them, S ^α represents the region with joint target trajectory; S ^β represents the region of independent target trajectory; t _j,i and t _i represent binary indicator variables, Indicates the transition similarity cost value between the observed trajectory T _j and T _i , C _j,i =-logP _sim (T _jk-1 |T _ik ), C _i represents the depth cost value of the observed trajectory T _j ${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; log</span>}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; log</span>}\frac{{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; log</span>}\frac{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; .</span>$