CN115410162A - Multi-target detection and tracking algorithm under complex urban road environment - Google Patents

Abstract

The invention discloses a multi-target detection and tracking method in a complex urban road environment: step 1: construct a training set and a test set; step 2: add feature fusion modules layer by layer on the basis of the existing DLA34 backbone network to realize input images Deep and shallow network feature fusion; Step 3: Use the Transformer coding module to extract long-distance feature dependencies in the feature map; Step 4: Through further feature fusion and logistic regression processing; Step 5: Use the multi-target tracking module for target association processing and tracking , to obtain the tracking feature map with the target detection frame; step 6, to obtain the trained multi-target detection and tracking model; step 7, input the video data to be detected into the trained multi-target detection and tracking model, and obtain the target Tracking feature maps for detection boxes. The invention can accurately detect and track multiple targets in a complex urban road environment, and can stably identify targets with large changes in appearance scale.

Description

A Multi-target Detection and Tracking Algorithm in Complex Urban Road Environment

technical field

The invention belongs to the technical field of automatic driving, and relates to a method for detecting and tracking a traffic target.

Background technique

Smart transportation has become an important direction of future transportation development. Its typical representative automatic driving is a multi-disciplinary and multi-field intertwined comprehensive technology. The development of automatic driving technology requires not only the ability of traffic participating vehicles to have automatic driving capabilities, but also the need to master complex traffic environments. Advanced precise perception technology, high-precision maps, vehicle navigation and positioning, vehicle dynamics control and other technologies to build a complete vehicle-road collaborative traffic system. In recent years, the 5G network superimposed cloud computing solution can use advanced artificial intelligence technology to endow traditional infrastructure with the ability to perceive the road, and then improve the environmental awareness of bicycles through the Internet of Things and cloud computing. Whether it is single-vehicle intelligence or vehicle-road coordination, sensors are required to collect external environmental information. Currently commonly used sensors include lidar, millimeter-wave radar and cameras. Compared with other sensors, cameras have become the preferred visual sensor for environmental perception due to their unique cost performance. Camera-based artificial intelligence technology has become indispensable for the development of smart transportation. Therefore, multi-target detection and tracking is of great significance to the perception of the complex traffic environment.

First of all, the objects in traffic scenes are mostly captured by cameras fixed at high places. There are problems such as small objects farther away in the picture, less feature information, and a large number of objects in the same picture in traffic scenes with large differences in size. The convolutional neural network commonly used in current research will down-sample and encode the image during the forward propagation process, which makes the model easy to lose the target with a small area and makes it more difficult for the model to capture the target. Secondly, with the development of deep learning, a lot of research results have been achieved in multi-target tracking. However, due to the influence of factors such as changes in the appearance and size of targets, occlusion, and blurring caused by rapid movement during the tracking process, existing tracking algorithms cannot achieve this goal. Ideal state. Aiming at the problem of multi-target detection and tracking in traffic scenarios, target detection algorithms and two-stage tracking networks based on Kalman filter and Hungarian algorithm are commonly used in the industry today. Such a model has some problems: the target detection and tracking modules are independent of each other and cannot be simultaneously At the same time, the accuracy of target detection determines the performance of target tracking, resulting in bottlenecks in network training and optimization, and stable tracking of targets with large inter-frame displacements cannot be achieved.

Contents of the invention

The purpose of the present invention is to provide a multi-target detection and tracking algorithm in a complex urban road environment to solve the problem of low target detection accuracy and the inability to achieve stable tracking of targets with large inter-frame displacements in the prior art The problem.

To achieve the above object, the present invention provides the following technical solutions:

A multi-target detection and tracking method in a complex urban road environment, specifically comprising the following steps:

Step 1: Select a public dataset for data augmentation, obtain the dataset, and build a training set and a test set;

Step 2: On the basis of the existing DLA34 backbone network, add the feature fusion module layer by layer to realize the deep and shallow network feature fusion of the input image, and obtain the two-dimensional feature map after three feature fusion;

Step 3: According to the two-dimensional feature map after feature fusion, the Transformer coding module is used to extract the long-distance feature dependency relationship in the feature map, and the feature map after the dependency relationship is extracted;

Step 4: Through further feature fusion and logistic regression processing, generate a heat map and target bounding box;

Step 5: Use the multi-target tracking module to perform target association processing and tracking, and obtain a tracking feature map with a target detection frame;

Step 6, using the training set of step 1 to train the multi-target detection and tracking model composed of steps 2, 3, 4, and 5, and using the test set to test, and finally obtain the trained multi-target detection and tracking model;

Step 7: Input the video data to be detected into the trained multi-target detection and tracking model to obtain the tracking feature map with the target detection frame.

Further, in the step 1, VisDrone_mot in the mainstream traffic object detection data set VisDrone is selected as the data set of the present invention.

Further, the step 2 specifically includes the following sub-steps:

Step 21, input the images in the training set to the DLA34 network, perform two convolution operations on the original image with a convolution kernel of 3×3 size through the BatchNorm layer and the ReLU layer to obtain two feature maps, and the two convoluted two The feature map is input to the aggregation node for feature fusion, and a feature map with a resolution of 1/4 the size of the original input feature map is obtained;

Step 22, perform 2 times downsampling on the 1/4 size feature map obtained in step 21 to obtain a new feature map, repeat the convolution operation and aggregation operation in step 21 twice on the feature map to obtain two feature maps, And perform the aggregation operation again with the aggregation node obtained in step 21 as a common input, and obtain a feature map with a resolution of 1/8 of the size of the original input feature map;

Step 23: Obtain a feature map of size 1/16 from the feature map of size 1/8 in the same way as in step 22 to obtain a feature map of size 1/8 of the original input feature map from a feature map of size 1/4, and then A feature map of size 1/32 is obtained from a feature map of size 1/16;

Step 24, as shown in Figure 2, the obtained 1/4 size feature map, 1/8 size feature map, 1/16 size feature map, and 1/32 size feature map are sequentially used in the feature fusion module for correlation The adjacent feature map features are fused to obtain new feature maps of 1/4 size, 1/8 size and 1/16 size respectively.

Further, in the step 24, the feature fusion module is used to realize the following operations:

Step 241, perform deformable convolution processing with a convolution kernel of 3×3 size on the feature map F1, and pass the processed result through the BatchNorm layer and the ReLU layer to obtain a mapped feature map;

Step 242, using direct interpolation upsampling plus convolution processing to replace the transposed convolution in the DLA34 backbone network, and perform 2 times upsampling on the mapped feature map obtained in step 241 to obtain the feature map F1';

Step 243, adding the feature map F1' obtained in step 242 to the corresponding channel value of the feature map F2 to obtain a merged feature map;

Step 244, after the merged feature map obtained in step 243 is processed by a deformable convolution of 3×3 size, it passes through the BatchNorm layer and the ReLU layer in turn to obtain a two-dimensional feature map F2';

When the feature map F1 and feature map F2 are 1/4 size feature map and 1/8 size feature map respectively, the obtained two-dimensional feature map F2' is a 1/4 size feature map;

When the feature map F1 and feature map F2 are 1/8 size feature map and 1/16 size feature map respectively, the obtained two-dimensional feature map F2' is a 1/8 size feature map;

When the feature map F1 and the feature map F2 are feature maps with a size of 1/16 and feature maps with a size of 1/32 respectively, the obtained two-dimensional feature map F2' is a feature map with a size of 1/16.

Further, the step 3 specifically includes the following sub-steps:

Step 31, collapse the 1/16-size two-dimensional feature map finally obtained in step 2 into a one-dimensional sequence, and convolute to form K, V, Q feature maps;

Step 32, add the position code and the feature map K and feature map Q obtained in step 31 pixel by pixel to obtain two feature maps with position information, and the two feature maps and feature map V are used as common input into the multi-head attention The force module is processed to obtain a new feature map;

Step 33, performing a fusion operation and a LayerNorm operation of adding corresponding values between the feature maps to the new feature map obtained in step 32 and the V, K, and Q feature maps obtained in step 31;

The results obtained in step 34 and step 33 enter the feedforward neural network for processing, and output through the residual connection to obtain a new feature map.

Further, the position code in the step 32 is obtained by the following formula:

PE _(pos,2i) = sin(pos/10000 ^2i/d )

pE _(pos,2i+1) ＝cos(pos/10000 ^2i/d )

Among them, PE _{( )} is the matrix of position encoding, and its resolution is the same as the resolution of the input feature map, pos indicates the position of the vector in the sequence, and i is the index of the channel, and d indicates the channel number of the input feature map.

Further, the step 4 specifically includes the following sub-steps:

Step 41, perform double upsampling on the feature map finally obtained in step 3 to obtain a new feature map.

Step 42, use the same feature fusion module as in step 24 to perform feature fusion on the 1/4 size and 1/8 size feature map obtained in step 24, and obtain a 1/4 size new feature map;

Step 43, use the feature fusion module to perform feature fusion on the 1/8 size and 1/16 size feature map obtained in step 24, and add pixel by pixel to the feature map obtained in step 41 to obtain a new feature of 1/8 size picture;

Step 44, using the feature fusion module to perform feature fusion on the 1/4 size feature map obtained in step 42 and the 1/8 size feature map obtained in step 43 to generate a heat map with a resolution of 1/4 the size of the original image;

Step 45, perform logistic regression on the heat map obtained in step 44 and the heat map label containing the center point of the target in the data set obtained in step 1, to obtain the center point of the predicted target

Step 46, obtain the coordinates of the upper left point and the lower right point of the frame corresponding to each target through formula (3), and generate the target bounding box:

即步骤45得到预测目标的中心点，

表示中心点与目标中心点的偏移量，

表示目标对应的边框的尺寸。in,

That is, step 45 obtains the center point of the predicted target,

Indicates the offset between the center point and the target center point,

Indicates the size of the bounding box corresponding to the target.

Further, the step 5 specifically includes the following sub-steps:

Step 51, use the same image input in step 2 as the T-1th frame image, and select the next frame image, that is, the T-th frame image, and use the T-th frame and the T-1th frame image as input, and pass through the CenterTrack backbone network Processing generates feature maps f _T and f _T-1 respectively;

Step 52, send the feature maps f _T and f _T-1 to the cost space module as shown in Figure 5 for target association processing, and obtain the output feature map f'_T;

将

与步骤52得到的特征图f′_T一起进行可变形卷积生成特征图

Step 53, perform the Hadamard product of the heat map obtained in step 4 and the feature map f _T-1 obtained in step 51 to generate a feature map

Will

Perform deformable convolution together with the feature map f' _T obtained in step 52 to generate a feature map

依次使用3个1×1卷积操作、下采样操作，生成第T-1帧特征图；将步骤51中得到的特征图f_T使用3个1×1卷积进行操作，生成第T帧特征图；Step 54, will

Use three 1×1 convolution operations and downsampling operations in turn to generate the feature map of the T-1th frame; use the feature map f _T obtained in step 51 to operate with three 1×1 convolutions to generate the feature map of the T-th frame picture;

Step 55, input the T-th frame feature map obtained in step 54 together with the T-1-th frame feature map into the attention propagation module for feature propagation to obtain a tracking feature map V′ _T with a target detection frame.

Further, the step 52 specifically includes the following operations:

Step 521, send the feature maps f _T and f _T-1 to the three-layer weight-shared convolution structure in the cost space module to generate feature maps e _T and e _T-1 , which are the appearance encoding vectors of the target;

Step 522, perform the maximum pooling operation on the feature maps e _T and e _T-1 to obtain e′ _T and e′ _T-1 to reduce the complexity of the model, and use the transposition calculation of the product of e′ _T and e′ _T-1 Get the cost space matrix C, the position of the target on the cost space matrix C in the current frame is (i, j), and extract the two-dimensional image containing the position information of the target in the current frame in the previous frame image from the cost space matrix C The cost matrix C _i,j takes the maximum value for the horizontal and vertical directions of C _i,j respectively to obtain the feature map of the corresponding direction

Step 523, define two offset templates by formula (4) and (5)

G _i,j,l =(lj)×s1≤l≤W _C (4)

M _i,j,k ＝(ki)× _s1≤k≤HC (5)

Among them, s is the downsampling multiple of the feature map relative to the original image, W _C , H _C are the width and height dimensions of the feature map, G _i,j,l is the target (i,j) in the T frame image at T- The offset that appears at horizontal position l in 1 frame image, M _{i, j, k} is the offset that T frame target (i, j) appears in vertical position k in T-1 frame image;

与步骤523中定义的偏移模板G和M相乘之后进行通道上的叠加，得到特征图O_T，代表目标在水平和竖直两个方向上的偏移模板；之后将O_T进行2倍上采样恢复为H_F×W_F大小，同时，将O_T特征图的水平与竖直两个通道分别与步骤51 得到的f_T、f_T-1进行通道上的叠加，再经过卷积形成水平和竖直方向上特征图大小不变、通道数为9的2个特征图，将这2个特征图进行通道上的叠加得到输出特征图f′_T。Step 524, the step 522 obtained

After multiplying the offset templates G and M defined in step 523, superimpose on the channel to obtain the feature map O _T , which represents the offset template of the target in both horizontal and vertical directions; then O _T is doubled The upsampling is restored to the size of H _F ×W _F. At the same time, the horizontal and vertical channels of the O _T feature map are respectively superimposed on the channels with f _T and f _T-1 obtained in step 51, and then formed by convolution The size of the feature map in the horizontal and vertical directions is constant, and the number of channels is 9. The two feature maps are superimposed on the channels to obtain the output feature map f′ _T .

Compared with prior art, the beneficial effect of the present invention is:

①In the present invention, the input picture resolution of the data set used is appropriately increased to ensure the size of the final feature map to retain more detailed information;

②In the multi-target detection module of the present invention, the deep feature map containing more semantic information and the shallow feature map containing more detailed information are fused through the feature fusion module to improve the detection ability of the model for small targets;

③In the multi-target detection module, the present invention introduces the self-attention mechanism of the Transformer coding module to capture long-distance dependencies and explore the potential relationship of features in the feature map, so that it can stably identify targets with large changes in appearance scale;

④Propose a multi-target tracking algorithm based on cost space and inter-frame information fusion, use the cost space matrix to predict the position of the target in the current frame in the previous frame, and associate the targets between the two frames to achieve the tracking effect;

⑤ In the multi-target tracking module, the attention propagation module is introduced to fuse the features of multi-frame targets to make up for the target space misalignment problem caused by the target movement between frames, so that the model can still accurately track the target when the target is occluded .

Description of drawings

Fig. 1 is a schematic diagram of a multi-target detection module of the present invention;

Fig. 2 is a schematic diagram of the feature fusion module in the multi-target detection module;

Fig. 3 is a schematic diagram of the Transformer encoding module in the multi-target detection module;

Fig. 4 is a schematic diagram of a multi-target tracking module of the present invention;

Fig. 5 is a schematic diagram of the cost space module in the multi-target tracking module;

6 is a schematic diagram of the experimental results of the multi-target detection module of the present invention; they are respectively schematic diagrams of the target center point and target bounding box results obtained by the module for detecting small targets and large targets.

Fig. 7 is a schematic diagram of the experimental results of the multi-target tracking module of the present invention. They are four pictures of two test cases respectively, and the four pictures of each test case are frame 0, frame 5, frame 10 and frame 15 respectively.

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

The multi-target detection and tracking model of the present invention is divided into two parts. First, it is a multi-target detection module framework as shown in FIG. Introduce the Transformer encoding module to perform self-attention encoding on the fused feature map, so as to solve the limitation of the network's ability to extract the semantics of large targets caused by the large difference in target feature scale; finally generate a target heat map and return it to obtain the boundary of the corresponding target The frame implements traffic object detection. Figure 4 shows the framework of the target tracking module. The feature map is generated through the CenterTrack backbone network, and the target association and tracking between the two frames are realized by using the cost space matrix; the target information of the two frames before and after is fused and complemented by using the attention propagation module. , to achieve accurate tracking when the target is blurred or occluded.

The multi-target detection and tracking method under the complex urban road environment of the present invention specifically comprises the following steps:

Step 1: Select a public dataset for data augmentation, obtain a dataset, and construct a training set and a test set.

Specifically: select VisDrone_mot in the mainstream traffic object detection data set VisDrone as the data set of the present invention. The VisDrone_mot data set collects aerial aerial views of streets in multiple cities in China by drones, and provides 96 video sequences, including 56 training video sequences containing 24201 frame images, 7 verification video sequences containing 2819 frame images, 33 The test sequence contains 12968 frames of images, and the bounding boxes of the recognized objects are manually annotated in each video frame. Increase the resolution of the input pictures in the VisDrone_mot dataset to 1024×1024, ensure that the final feature map output by the multi-target detection module has a size of 256×256, and retain more detailed information. At the same time, use random flipping, and the resolution size is The data enhancement method combining random scaling, random cropping and color dithering between 0.6 and 1.3 times is used as an extended training sample.

Step 2: On the basis of the existing DLA34 backbone network, add a feature fusion module layer by layer to realize the deep and shallow network feature fusion of the input image, and obtain a two-dimensional feature map after three feature fusions. As shown in Figure 1, it specifically includes the following sub-steps:

Step 21, input the images in the training set to the DLA34 network, perform two convolution operations on the original image with a convolution kernel of 3×3 size through the BatchNorm layer and the ReLU layer to obtain two feature maps, and the two convoluted two The feature map is input to the aggregation node for feature fusion, and a feature map with a resolution of 1/4 the size of the original input feature map is obtained. The feature fusion of the aggregation node is as formula (1):

N(X ₁ ,...,X _n )=σ(BN(∑w _i x _i +b),...,BN(∑w _i x _i +b)) (1)

Among them, N(·) represents aggregation node, σ(·) represents feature aggregation, w _i x _i +b represents convolution operation, BN represents BatchNorm operation, Xi _=1...N corresponds to the output of the convolution module.

Step 22, perform 2 times downsampling on the 1/4 size feature map obtained in step 21 to obtain a new feature map, repeat the convolution operation and aggregation operation in step 21 twice on the feature map to obtain two feature maps, And perform the aggregation operation again with the aggregation node obtained in step 21 as a common input, and obtain a feature map with a resolution of 1/8 of the size of the original input feature map. The purpose of this step is to transfer the feature information of the shallow layer of the network to the deep layer of the network.

Step 23: Obtain a feature map of size 1/16 from the feature map of size 1/8 in the same way as in step 22 to obtain a feature map of size 1/8 of the original input feature map from a feature map of size 1/4, and then A feature map of size 1/32 is obtained from a feature map of size 1/16;

Step 24, as shown in Figure 2, the obtained 1/4 size feature map, 1/8 size feature map, 1/16 size feature map, and 1/32 size feature map are sequentially used in the feature fusion module for correlation Neighboring feature map features are fused to obtain new feature maps of 1/4 size, 1/8 size and 1/16 size respectively;

The feature fusion module is used to realize the following operations:

Step 241, perform deformable convolution processing with a convolution kernel of 3×3 size on the feature map F1, and pass the processed result through the BatchNorm layer and the ReLU layer to obtain a mapped feature map;

Step 242, use the form of direct interpolation upsampling plus convolution to replace the transposed convolution in the DLA34 backbone network, perform 2 times upsampling on the mapped feature map obtained in step 241, and obtain the feature map F1′ to obtain a more Multi-target location information and reduce the amount of model parameters;

Step 243, adding the feature map F1' obtained in step 242 to the corresponding channel value of the feature map F2 to obtain a merged feature map;

Step 244, after the merged feature map obtained in step 243 is processed by a deformable convolution with a size of 3 × 3, the two-dimensional feature map F2' is obtained through the BatchNorm layer and the ReLU layer in turn;

When the feature map F1 and feature map F2 are 1/4 size feature map and 1/8 size feature map respectively, the obtained two-dimensional feature map F2′ is a 1/4 size feature map;

When the feature map F1 and feature map F2 are respectively 1/8 size feature map and 1/16 size feature map, the obtained two-dimensional feature map F2′ is a 1/8 size feature map;

When the feature map F1 and feature map F2 are feature maps of 1/16 size and 1/32 size respectively, the obtained two-dimensional feature map F2′ is a feature map of 1/16 size.

Step 3: According to the feature map after feature fusion obtained in step 2, the Transformer encoding module is used to extract the long-distance feature dependencies in the feature map, and the feature map after the dependency relationship is extracted. As shown in Figure 3, it specifically includes the following sub-steps:

Step 31, collapse the 1/16 size two-dimensional feature map finally obtained in step 2 into a one-dimensional sequence, and convolute to form three K (Key), V (Value), Q (Query) feature maps;

Step 32, add the position code and the feature map K and feature map Q obtained in step 31 pixel by pixel to obtain two feature maps with position information, and the two feature maps and feature map V are used as common input into the multi-head attention The force module is processed to obtain new feature maps to capture long-distance dependencies in images. The position code is obtained by formula (1) (2):

PE _{(pos, 2i)} = sin(pos/10000 ^2i/d ) (1)

PE _{(pos, 2i+1)} = cos(pos/10000 ^2i/d ) (2)

Among them, PE _{( )} is the matrix of position encoding, and its resolution is the same as the resolution of the input feature map, pos indicates the position of the vector in the sequence, and i is the index of the channel, and d indicates the channel number of the input feature map.

Step 33, the fusion operation and the LayerNorm (LN) operation of adding corresponding values between the feature maps to the new feature map obtained in step 32 and the V, K, and Q feature maps obtained in step 31 to avoid information loss;

The results obtained in step 34 and step 33 enter the feedforward neural network for processing, and output through the residual connection to obtain a new feature map.

Step 4: According to the feature maps obtained in steps 2 and 3, through further feature fusion and logistic regression processing, generate heat maps and target bounding boxes. Specifically include the following sub-steps:

Step 41, perform double upsampling on the feature map finally obtained in step 3 to obtain a new feature map.

Step 42, use the same feature fusion module as in step 24 to perform feature fusion on the 1/4 size and 1/8 size feature map obtained in step 24, and obtain a 1/4 size new feature map;

Step 43, use the feature fusion module to perform feature fusion on the 1/8 size and 1/16 size feature map obtained in step 24, and add pixel by pixel to the feature map obtained in step 41 to obtain a new feature of 1/8 size picture;

Step 44, using the feature fusion module to perform feature fusion on the 1/4 size feature map obtained in step 42 and the 1/8 size feature map obtained in step 43 to generate a heat map with a resolution of 1/4 the size of the original image;

Step 45, perform logistic regression on the heat map obtained in step 44 and the heat map label containing the center point of the target in the data set obtained in step 1, to obtain the center point of the predicted target

Step 46, obtain the coordinates of the upper left point and the lower right point of the frame corresponding to each target through formula (3), and generate the target bounding box:

即步骤45得到预测目标的中心点，

表示中心点与目标中心点的偏移量，

表示目标对应的边框的尺寸。in,

That is, step 45 obtains the center point of the predicted target,

Indicates the offset between the center point and the target center point,

Indicates the size of the bounding box corresponding to the target.

Step 5: According to the input image in step 2 and the heat map obtained in step 4, use the multi-target tracking module to perform target association processing and tracking, and obtain a tracking feature map with a target detection frame. As shown in Figure 4, it specifically includes the following sub-steps:

Step 51, use the same image input in step 2 as the T-1th frame image, and select the next frame image, that is, the T-th frame image, and use the T-th frame and the T-1th frame image as input, and pass through the CenterTrack backbone network Processing generates feature maps f _T and f _T-1 respectively;

In step 52, the feature maps f _T and f _T-1 are respectively sent to the cost space module shown in Fig. 5 for target association processing, and the output feature map f' _T is obtained. Specifically include the following operations:

Step 521, send the feature maps f _T and f _T-1 to the three-layer weight-shared convolution structure in the cost space module to generate feature maps e _T and e _T-1 , which are the appearance encoding vectors of the target;

Step 522, perform the maximum pooling operation on the feature maps e _T and e _T-1 to obtain e′ _T and e′ _T-1 to reduce the complexity of the model, and use the transposition calculation of the product of e′ _T and e′ _T-1 Get the cost space matrix C to save the similarity of corresponding points between the feature maps of the two frames. The position of the target on the cost space matrix C in the current frame is (i, j), and extract from the cost space matrix C that contains The two-dimensional cost matrix C _{i, j} of the position information of the target in the previous frame image, take the maximum value for the horizontal direction and vertical direction of C _{i, j} respectively to obtain the feature map of the corresponding direction

Step 523, define two offset templates by formula (4) and (5)

G _i,j,l =(lj)×s1≤l≤W _C (4)

M _i,j,k =(ki)× _s1≤k≤HC (5)

Among them, s is the downsampling multiple of the feature map relative to the original image, W _C , H _C are the width and height dimensions of the feature map, G _{i, j, l} is the target (i, j) in the T frame image at T- The offset that appears at horizontal position l in 1 frame of image, M _{i, j, k} is the offset of T-frame target (i, j) that appears at vertical position k in T-1 frame of image.

与步骤523中定义的偏移模板G和M相乘之后进行通道上的叠加，得到特征图O_T，代表目标在水平和竖直两个方向上的偏移模板；之后将O_T进行2倍上采样恢复为H_F×W_F大小，同时，将O_T特征图的水平与竖直两个通道分别与步骤51得到的f_T、f_T-1进行通道上的叠加，再经过卷积形成水平和竖直方向上特征图大小不变、通道数为9的2个特征图，将这2个特征图进行通道上的叠加得到输出特征图f′_T。Step 524, the step 522 obtained

After multiplying the offset templates G and M defined in step 523, superimpose on the channel to obtain the feature map O _T , which represents the offset template of the target in both horizontal and vertical directions; then O _T is doubled The upsampling is restored to the size of H _F ×W _F. At the same time, the horizontal and vertical channels of the O _T feature map are respectively superimposed on the channels with f _T and f _T-1 obtained in step 51, and then formed by convolution The size of the feature map in the horizontal and vertical directions is constant, and the number of channels is 9. The two feature maps are superimposed on the channels to obtain the output feature map f′ _T .

将

与步骤52得到的特征图f′_T一起进行可变形卷积生成特征图

Step 53, perform the Hadamard product of the heat map obtained in step 4 and the feature map f _T-1 obtained in step 51 to generate a feature map

Will

Perform deformable convolution together with the feature map f' _T obtained in step 52 to generate a feature map

依次使用3个1×1卷积操作、下采样操作，生成第T-1帧特征图(q_t-1、 k_t-1、v_t-1)；将步骤51中得到的特征图f_T使用3个1×1卷积进行操作，生成第T帧特征图(Q_t、K_t、V_t)；Step 54, will

Use three 1×1 convolution operations and downsampling operations sequentially to generate the feature map (q _t-1 , k _t-1 , v _t-1 ) of frame T-1; the feature map f _T obtained in step 51 Use three 1×1 convolutions to generate the T-th frame feature map (Q _t , K _t , V _t );

Step 55, input the T-th frame feature map obtained in step 54 together with the T-1-th frame feature map into the attention propagation module for feature propagation to obtain a tracking feature map V′ _T with a target detection frame. Among them, the calculation process of the attention propagation module is shown in formula (6):

为1×1卷积，d_k为特征图Q和K的维度，Q_t、k_t-1、v_t-1、V_t为步骤54中得到的特征图。in,

is a 1×1 convolution, d _k is the dimension of the feature maps Q and K, and Q _t , k _t-1 , v _t-1 , V _t are the feature maps obtained in step 54.

Step 6: Use the training set in step 1 to train the multi-target detection and tracking model composed of steps 2, 3, 4, and 5, and use the test set to test, and finally obtain the trained multi-target detection and tracking model.

Step 7: Input the video data to be detected into the trained multi-target detection and tracking model to obtain the tracking feature map with the target detection frame.

For verifying feasibility and effectiveness of the present invention, the present invention has carried out following experiment:

First, for the multi-object detection module (ie step 2 to step 4), the model is evaluated using the average precision and recall. The average precision rate is obtained from the precision rate, and the formulas of the precision rate P and the recall rate R are specifically shown in formulas (7) and (8).

Among them, P is the percentage of the target that should be retrieved (TP) to all retrieved targets (TP+FP). R is the percentage of the target (TP) that should be retrieved to all the targets (TP+FN) that should be retrieved.

In the detection task, the precision rate reflects the ability of the model to be accurate, and the recall rate reflects the ability of the model to be recalled. The two indicators restrict each other, find the relative balance between the precision rate and the recall rate through the average precision rate (AP) under different confidence thresholds, and make a two-dimensional PR curve with the precision rate and recall rate as the horizontal and vertical coordinates . The average precision rate (AP) is the area surrounded by the PR curve, which is equal to the average operation of the precision rate.

The present invention first quantitatively analyzes the multi-target detection module, compares it with the baseline model on the VisDrone_mot data set, and increases the performance comparison between the method of the present invention and various baseline methods specific to each category in the experiment, from the results. Therefore, compared with other models, the method proposed by the present invention can ensure the recognition performance of some large targets while ensuring the correct recognition of smaller targets. Compared with the commonly used models with superior performance, the method of the present invention has the best recognition performance for large targets, with the accuracy reaching 42.16 and 33.10, and has good detection ability.

At the same time, in order to intuitively reflect the performance of the overall multi-target detection module, a qualitative analysis was performed on the module. The results are shown in Figure 6. It can be seen that the model of the present invention has good detection performance for targets of different scales. After adding the Transformer module, The model captures the long-distance dependencies more stably, and has a good recognition ability for small targets while the recognition effect for large targets is still relatively robust.

Secondly, for the multi-target tracking module (ie step 5), use MOTA(↑), MOTP(↑), IDF1(↑), MT(↑), ML(↓), FP(↓), FN(↓), Frag (↓) and IDSW (↓) and other indicators for evaluation. ↑ indicates that the larger the index value is, the better the model performance is, and ↓ indicates that the smaller the index value is, the better the model performance is.

Among them, MOTA stands for multi-target tracking accuracy, which measures the ability of the algorithm to continuously track targets, and is used to count the accumulation of errors in tracking, and its formula is shown in (9).

Among them, m _t corresponds to FP, which represents the false positives (number of false detections) in the prediction results, that is, there is no corresponding tracking target matching the predicted position in the t-th frame. fp _t corresponds to FN, which represents false negatives (number of missed detections), that is, the target does not have a corresponding predicted position to match it in the tth frame. mme _t corresponds to IDSW, which represents the number of mismatches, that is, the number of ID switching times of the tracking target in the tth frame, g _t refers to the total number of real targets in the frame, MOTA comprehensively considers the false detection, missed detection and ID in the target trajectory exchange.

The MOTP expression also directly reflects the effect of model tracking, reflecting the distance between the tracking result and the label trajectory, and the formula is expressed as (10).

再进行求和得到最终的数值，该指标越大表示模型性能越好，轨迹误差越小。Among them, c _t represents the number of matches in the tth frame, and the trajectory error is calculated for each pair of matches

Then sum to get the final value. The larger the index, the better the model performance and the smaller the trajectory error.

MT is the number of most tracked (Mostly tracked), which refers to the number of tracks whose hit track is greater than 80% of the tag track, and the larger the value, the better. ML is the most lost number (Mostly lost), which refers to the number of lost tracks that are greater than 80% of the label tracks. The smaller the value, the better. Frag is the number of jumps, which refers to the number of changes in the tracking track from the "tracking" state to the "not tracking" state.

For a multi-target tracking detector, ID-related indicators are equally important, specifically the following three important indicators: IDP, IDR, IDF1. IDP stands for Identification Precision, which refers to the ID recognition accuracy rate of each target frame, and its formula is shown in (11).

where IDTP and IDFP are the number of true positive cases and false positive cases predicted by ID, respectively. IDR stands for Identification Recall, which refers to the ID recognition recall rate of each target box, and its formula is shown in (12).

where IDFN is the false negative of ID prediction. IDF1 represents the ID prediction F-score (Identification F-Score), which refers to the ID identification F-score of each target frame. The larger the index value, the better, and its calculation formula is shown in (13).

IDF1 is the first default indicator used to evaluate the quality of the tracker. Any two of the above three indicators can be used to infer the other.

First, comparing the quantitative experiments of the multi-target tracking module with the mainstream baseline model in recent years, on the VisDrone_mot data set, compared with the second best model, the tracking method proposed by the present invention is higher than the MOTA and MOTP indicators respectively. 3.2 and 1.8, and achieved relatively good results on other indicators, but because the false detection rate of the model of the present invention is less, it leads to disturbances in the normal range of ML and MT indicators. Compared with the TBD model, the JDT model can perform end-to-end optimization during the training process because the detection and tracking tasks promote each other, and can achieve better results on the tracking task.

Secondly, qualitatively analyze the model in the above data set, as shown in Figure 7, which shows two sections of test samples, and each section selects four of the pictures for display, which are frame 0 and frame 5 in the time dimension. frame, 10-frame and 15-frame images. It can be seen from the figure that the model can stably track multiple targets in traffic scenes, especially has excellent detection and tracking capabilities for small targets in traffic scenes.

Claims (9)

1. A multi-target detection and tracking method under a complex urban road environment is characterized by specifically comprising the following steps:

step 1: selecting a public data set for data enhancement to obtain a data set, and constructing a training set and a test set;

step 2: adding a feature fusion module layer by layer on the basis of the existing DLA34 backbone network to realize the deep and shallow network feature fusion of an input image and obtain a two-dimensional feature map after three feature fusion;

and 3, step 3: extracting the long-distance feature dependency relationship in the feature graph by adopting a Transformer coding module according to the two-dimensional feature graph after feature fusion to obtain the feature graph after the dependency relationship is extracted;

and 4, step 4: generating a heat map and a target boundary frame through further feature fusion and logistic regression processing;

and 5: performing target association processing and tracking by using a multi-target tracking module to obtain a tracking characteristic diagram with a target detection frame;

step 6, training the multi-target detection and tracking model formed by the steps 2, 3, 4 and 5 by adopting the training set of the step 1, and testing by adopting the test set to finally obtain the trained multi-target detection and tracking model;

and 7, inputting the video data to be detected into the trained multi-target detection and tracking model to obtain a tracking characteristic diagram with a target detection frame.

2. The multi-target detection and tracking method in a complex urban road environment according to claim 1, wherein in step 1, visDrone _ mot in the main-flow traffic target detection data set VisDrone is selected as the data set of the present invention.

3. The multi-target detection and tracking method in a complex urban road environment according to claim 1, wherein said step 2 comprises the following substeps:

step 21, inputting the images in the training set into a DLA34 network, performing convolution operation with convolution kernel of 3 × 3 twice on the original image through a BatchNorm layer and a ReLU layer to obtain two feature maps, inputting the two feature maps after convolution into an aggregation node for feature fusion, and obtaining a feature map with resolution of 1/4 of the original input feature map;

step 22, carrying out 2-time down-sampling on the feature map with the size of 1/4 obtained in the step 21 to obtain a new feature map, repeating the convolution operation and the aggregation operation in the step 21 on the feature map twice to obtain two feature maps, and carrying out the aggregation operation again by taking the aggregation node obtained in the step 21 as a common input to obtain the feature map with the resolution of 1/8 of the original input feature map;

step 23, obtaining a feature map with the size of 1/16 from the feature map with the size of 1/8 according to the same manner of obtaining the feature map with the size of 1/8 from the feature map with the size of 1/4 in the step 22, and obtaining a feature map with the size of 1/32 from the feature map with the size of 1/16;

and 24, as shown in fig. 2, sequentially adopting a feature fusion module to perform adjacent feature fusion on the obtained feature map with the size of 1/4, the feature map with the size of 1/8, the feature map with the size of 1/16 and the feature map with the size of 1/32 to respectively obtain new feature maps with the size of 1/4, the size of 1/8 and the size of 1/16.

4. The multi-target detection and tracking method in a complex urban road environment according to claim 3, wherein in step 24, the feature fusion module is configured to implement the following operations:

step 241, performing deformable convolution processing with a convolution kernel of 3 × 3 on the feature map F1, and obtaining a mapped feature map by passing a result obtained by the processing through a BatchNorm layer and a ReLU layer;

step 242, replacing the transposed convolution in the DLA34 backbone network with a direct interpolation upsampling and convolution processing mode, and performing 2-fold upsampling on the feature map obtained after mapping in step 241 to obtain a feature map F1';

step 243, adding the channel values corresponding to the characteristic diagram F1' and the characteristic diagram F2 obtained in step 242 to obtain a combined characteristic diagram;

step 244, performing 3 × 3 deformable convolution processing on the merged feature map obtained in step 243, and then sequentially passing through the BatchNorm layer and the ReLU layer to obtain a two-dimensional feature map F2';

when the characteristic diagram F1 and the characteristic diagram F2 are respectively a characteristic diagram with the size of 1/4 and a characteristic diagram with the size of 1/8, the obtained two-dimensional characteristic diagram F2' is a characteristic diagram with the size of 1/4;

when the characteristic diagram F1 and the characteristic diagram F2 are respectively a characteristic diagram with the size of 1/8 and a characteristic diagram with the size of 1/16, the obtained two-dimensional characteristic diagram F2' is a characteristic diagram with the size of 1/8;

when the feature maps F1 and F2 are feature maps of 1/16 size and 1/32 size, respectively, the two-dimensional feature map F2' obtained is a feature map of 1/16 size.

5. The multi-target detection and tracking method in a complex urban road environment according to claim 1, wherein said step 3 comprises the following substeps:

step 31, collapsing the two-dimensional characteristic diagram with the size of 1/16 obtained finally in the step 2 into a one-dimensional sequence, and convoluting to form a K, V and Q characteristic diagram;

step 32, respectively adding the position codes and the feature maps K and Q obtained in the step 31 pixel by pixel to obtain two feature maps with position information, inputting the two feature maps and the feature map V into a multi-head attention module as common input, and processing to obtain a new feature map;

step 33, performing fusion operation and LayerNorm operation of adding corresponding values among feature maps on the new feature map obtained in step 32 and the V, K and Q feature maps obtained in step 31;

and step 34, processing the result obtained in the step 33 in a feed-forward neural network, and connecting and outputting through residual errors to obtain a new characteristic diagram.

6. The multi-target detection and tracking method in a complex urban road environment according to claim 5, wherein the position code in step 32 is obtained by the following formula:

PE _(pos，2i) ＝sin(pos/10000 ^2i/d )

PE _(pos，2i+1) ＝cos(pos/10000 ^2i/d )

wherein, PE _(·) The matrix for position coding has the same resolution size as the input feature map, pos represents the position of the vector in the sequence, i is the index of the channel, and d represents the number of channels of the input feature map.

7. The multi-target detection and tracking method in the complex urban road environment according to claim 1, wherein the step 4 specifically comprises the following substeps:

and step 41, performing 2 times of upsampling on the feature map finally obtained in the step 3 to obtain a new feature map.

Step 42, performing feature fusion on the feature map with the size of 1/4 and the size of 1/8 obtained in the step 24 by using the same feature fusion module as that in the step 24 to obtain a new feature map with the size of 1/4;

step 43, performing feature fusion on the feature maps with the size of 1/8 and the size of 1/16 obtained in the step 24 by using a feature fusion module, and performing pixel-by-pixel addition on the feature maps obtained in the step 41 to obtain a new feature map with the size of 1/8;

step 44, performing feature fusion on the feature map with the size of 1/4 obtained in the step 42 and the feature map with the size of 1/8 obtained in the step 43 by using a feature fusion module to generate a heat map with the resolution being the size of 1/4 of the original image;

step 45, performing logistic regression on the heat map obtained in the step 44 and the heat map labels containing the target center points in the data set obtained in the step 1 to obtain the center points of the predicted targets

Step 46, obtaining coordinates of a left upper point and a right lower point of a frame corresponding to each target through the formula (3), and generating a target boundary frame:

wherein,

i.e. step 45 obtains the center point of the predicted target,

representing the offset of the center point from the target center point,

indicating the size of the corresponding bounding box of the object.

8. The multi-target detection and tracking method in the complex urban road environment according to claim 1, wherein the step 5 specifically comprises the following substeps:

step 51, using the same image input in step 2 as the T-1 frame image, selecting the next frame image, namely the T-frame image, using the T-frame and the T-1 frame image as input, and respectively generating the feature map f through the CenterTrack backbone network processing _T And f _T-1 ；

Step 52, the feature map f _T And f _T-1 Respectively sending the data to a cost space module shown in FIG. 5 for target correlation processing to obtain an output feature map f' _T ；

Step 53, comparing the heat map obtained in step 4 with the feature map f obtained in step 51 _T-1 Performing a hadamard product to generate a feature map

Will be provided with

And the feature map f 'obtained in step 52' _T Together withPerforming a deformable convolution to generate a feature map

Step 54, will

Generating a T-1 frame feature map by sequentially using 31 × 1 convolution operations and a downsampling operation; the characteristic diagram f obtained in the step 51 is processed _T Performing operation by using 31 × 1 convolutions to generate a Tth frame feature map;

step 55, inputting the T-th frame feature map obtained in the step 54 and the T-1 th frame feature map into the attention propagation module together for feature propagation to obtain a tracking feature map V 'with a target detection frame' _T 。

9. The multi-target detection and tracking method in a complex urban road environment according to claim 8, wherein said step 52 specifically comprises the following operations:

step 521, the feature map f is processed _T And f _T-1 Three layers of convolution structure generation characteristic graphs e shared by weights respectively sent to cost space modules _T And e _T-1 I.e. the appearance coding vector of the target;

step 522, for the feature map e _T And e _T-1 Performing a max pooling operation to obtain e' _T And e' _T-1 To reduce model complexity, e 'is used' _T And e' _T-1 The transposition calculation of the product obtains a cost space matrix C, the position of the target on the cost space matrix C in the current frame is (i, j), and a two-dimensional cost matrix C containing the position information of the target in the current frame in the previous frame image is extracted from the cost space matrix C _i,j To C, to _i,j Respectively taking the maximum value in the horizontal direction and the vertical direction to obtain a characteristic diagram in the corresponding direction

Step 523, define two biases by equations (4) and (5)Movable template

G _i,j,l ＝(l-j)×s1≤l≤W _C (4)

M _i,j,k ＝(k-i)×s1≤k≤H _C (5)

Wherein s is the downsampling multiple of the feature map relative to the original image, W _C 、H _C Is the size of the width and height dimensions of the feature map, G _i,j,l M is an offset of an object (i, j) in the T frame image appearing in the horizontal position l in the T-1 frame image _i,j,k An offset for a T frame object (i, j) appearing at vertical position k in the T-1 frame image;

step 524, the result of step 522 is processed

Multiplying the offset templates G and M defined in step 523, and then superimposing the offset templates G and M on the channel to obtain a feature map O _T Representing offset templates of the target in both horizontal and vertical directions; then O is introduced _T 2 times up sampling recovery to H _F ×W _F Size, simultaneously, adding O _T The horizontal and vertical channels of the characteristic map are respectively compared with f obtained in step 51 _T 、f _T-1 Superposing on channels, forming 2 feature maps with unchanged feature map sizes in the horizontal direction and the vertical direction and 9 channels by convolution, and superposing the 2 feature maps on the channels to obtain an output feature map f' _T 。

Images