WO2020181685A1 - Vehicle-mounted video target detection method based on deep learning - Google Patents

Abstract

Disclosed is a vehicle-mounted video target detection method based on deep learning. The method employs an improved Faster R-CNN algorithm to implement target detection in a complex traffic environment, and to provide a driving safety auxiliary function. A serious problem of small target missed detection exists in an existing target tracking algorithm. According to the present invention, by adding a depth information channel, connecting the depth information channel to an original color image channel in parallel, performing fusion in a channel dimension, and performing candidate box extraction and target detection on a fused feature image, the detection rate of small targets is improved; in addition, by adding training of a difficult sample in training, the overall target recognition rate of the algorithm is improved. According to the present invention, the problem of small target missed detection existing in the Faster R-CNN algorithm can be fully considered, and the accuracy of vehicle recognition in a complex traffic scene is improved by means of depth image feature fusion and a difficult sample mining method.

Description

A vehicle-mounted video target detection method based on deep learning Technical field

The invention relates to a vehicle-mounted video target detection method based on deep learning, which belongs to the technical field of video image processing.

Background technique

In the driving process, the target detection and tracking of vehicles, pedestrians and other obstacles in front of the vehicle, and on this basis, the behavior analysis of the preceding vehicle is the basis of the driving safety assistance system. The main steps of traditional target detection methods are generally: extracting target features, training corresponding classifiers, sliding window search, repetition and false positive filtering. This sliding window selection strategy for target detection is not targeted, has high time complexity, window redundancy, and hand-designed features have poor robustness and unreliable classifiers; at the same time, existing target detection algorithms cannot be flexibly trained The data is used to learn effective features according to different needs to complete specific detection tasks.

Summary of the invention

The purpose of the present invention is to solve the above shortcomings of the prior art and provide a vehicle-mounted video target detection method based on deep learning.

In order to achieve the above objectives, the present invention adopts the following technical solutions:

A vehicle-mounted video target detection method based on deep learning includes the following steps:

Step 1) Align the pixels under the depth coordinates to the color coordinates; then extract the features of the depth image and the color image through CNN respectively, and concatenate the feature maps output by the respective convolutional layers in the channel dimension to obtain the final RGB -D feature as the convolution feature map after convolution;

Step 2) Construct a regional proposal network RPN, which includes a 3×3 convolutional layer and two 1×1 parallel convolutional layers; input the fused convolution feature map into a 3×3 volume Multilayer, sliding a network of a preset size in pixels on the input feature map, and each sliding position generates an anchor point of a specific scale;

Input the generated anchor points into two 1×1 parallel convolutional layers for position regression and foreground and background judgment, respectively output the anchor point’s foreground and background confidence and the position of all candidate frames, and select frames from the resulting rectangle according to preset conditions Filter out the first specific number of regions with the highest foreground confidence, and get the final regional suggestion set C;

Construct the Fast R-CNN model. The Fast R-CNN model consists of two ROI pooling layers, a fully connected layer and two parallel fully connected layers, respectively outputting the confidence of the region and the candidate frame after the border regression Location; input the fused convolutional features into the Fast R-CNN model, and output the location of the target in the image and its category and confidence;

Step 3): Construct the cost function for training the RPN network and the cost function for training the Fast R-CNN network;

Step 4) Use a standard ZF model to train and fine-tune the parameters of the network, and randomly initialize all new layers by extracting weights from the zero-mean Gaussian distribution of the set standard deviation;

Step 5) Use the back propagation algorithm and the stochastic gradient descent algorithm to train the model by alternately training the RPN and Fast R-CNN networks, and adjust the weight of each layer of neural network in turn according to the preset parameters;

Step 6) Use the pre-obtained training set to test the pre-trained Faster R-CNN model, and screen out difficult samples according to the discriminant formula of difficult samples;

Step 7) Add the difficult samples generated in step 6) to the training set, train the network again, repeat steps 5)-step 7), and finally get the best Faster R-CNN model;

Step 8) Process the vehicle-mounted video image collected in practice, input it into the trained Faster R-CNN model, and output the target category, confidence and target position in the image.

The beneficial effects achieved by the present invention:

First, on the basis of the proposed convolutional neural network model, the present invention proposes a target detection model based on depth information completion. The improved Faster R-CNN adds a depth information channel to separate color images and depth images. Feature extraction is carried out through CNN with the same structure. Two CNNs are connected in parallel, and the original color image feature map and depth image feature map are combined in series to obtain the final image feature. Compared with the original algorithm, the present invention The obtained image features are richer, supplement the detailed information of the vehicle, and will not increase time overhead, and meet the requirements of improving target detection in complex scenes.

Second, the present invention adds a difficult sample mining strategy in the training phase, so that the model pays more attention to difficult samples on the original basis, and better distinguishes the background of vehicles and suspected vehicles, so as to achieve the purpose of improving accuracy;

Third, the Faster R-CNN algorithm that uses the shared convolutional network to extract suggested candidate frames in the present invention has been significantly improved in real-time. This algorithm abandons the traditional region suggestion algorithm and uses the convolutional layer in the deep network to Extracting candidate frames saves a lot of time and overhead.

Description of the drawings

Fig. 1 is a schematic flowchart of a method according to a specific embodiment of the present invention.

Fig. 2 is a training flow chart of the improved Faster R-CN algorithm in a specific embodiment of the present invention.

detailed description

The present invention will be further described below in conjunction with the drawings. The following embodiments are only used to explain the technical solutions of the present invention more clearly, and cannot be used to limit the protection scope of the present invention.

The purpose of the present invention is to propose a vehicle-mounted video target detection method based on deep learning. On the basis of Faster R-CNN, feature maps of depth images are added to supplement vehicle detail information, and convolutional nerves that are the same as extracting color image features are selected. In the network, the color image channel and the depth image channel adopt a parallel structure, and the extracted features are fused in series to obtain the final RGB-D feature, and a difficult sample mining strategy is added to the training to improve the algorithm's ability to deal with small targets and small targets in complex traffic scenes. Detection accuracy of difficult targets.

Fig. 1 is a flowchart of a method according to a specific embodiment of the present invention.

When implementing the method of the present invention, it can be based on pre-acquired training set sample sets and test set samples, or training set and test set can be made according to requirements. In this embodiment, using the KITTI data set to construct the training sample set and the test sample set includes the following step 1: adopting the PASCAL VOC data set format and evaluation algorithm tool. First, convert the categories of KITTI: PASCAL VOC has a total of 20 categories. In urban traffic scenarios, the key detection objects are vehicles, pedestrians, and traffic signs. Therefore, the data set is divided into the above three categories; second, the label information is converted: The annotation file is converted from txt to xml, and other information in the annotation is removed, leaving only 3 categories; finally, a training verification set and a test set are generated.

As shown in Figure 1, the method of the present invention includes the following steps:

Step 2) Construct an improved Faster R-CNN model, which integrates the Regional Proposal Network (RPN) and Fast R-CNN network;

2.1) Convolutional Neural Networks (CNN)

First, align the pixels under the depth coordinates to the color coordinates; CNN selects the feature extraction network in the ZF model, and connects two CNNs with the same structure in parallel (the original color image channel is channel 1, and the parallel depth image channel is Channel 2); After the two types of images are extracted by CNN features, the size of the feature map is hwc (where h and w represent the height and width of the feature map respectively, and c is the three channels of RGB), and the color image feature and the depth image feature are taken as Feature fusion of the two channels, the size of the fused feature map is 2hwc;

2.2) The regional proposal network RPN includes a 3×3 convolutional layer and two 1×1 parallel convolutional layers;

Input the fused convolutional feature map into a 3×3 convolutional layer, and slide a small network in pixels on the input feature map. In this embodiment, 3 scales and 3 aspect ratios are used, and each If k=3×3=9 anchor points of different scales are generated from one sliding position, a total of hwk anchor points are generated, and hwk rectangular candidate frames are obtained;

Two 1×1 parallel convolutional layers perform position regression and background scene judgment on the anchor points of the upper layer, and respectively output the foreground and background scene confidence of the anchor points and the candidate frame position. The position of the candidate frame contains four parameters including the coordinates x and y of the center point of the candidate frame, as well as the width w’ and the height h’;

2.2) Filter the rectangular candidate frame obtained in 2.1) according to the preset condition to select a predetermined number of regions that meet the preset condition. In this embodiment, the obtained rectangular candidate frames are sorted in descending order according to the softmax score, and the first 2000 regions are retained, and then the non-maximum suppression algorithm (Non-Maximum Suppression, NMS) is used to select the top 300 with the highest foreground confidence. Regions, get the final regional suggestion set C;

2.3) Fast R-CNN consists of two ROI pooling layers, one fully connected layer and two parallel fully connected layers, respectively outputting the confidence of the region and the position of the candidate frame after the border regression;

The ROI pooling layer performs a pooling operation on the region suggestion set C and the fused convolution feature map. The ROI pooling layer maps the ROI to the corresponding position of the feature map according to the input image, and divides the mapped area into sections of the same size , Perform maximum pooling operation on each section;

The fully connected layer merges the output results of the ROI pooling layer, and finally inputs two fully connected layers in parallel, performs region classification and border regression on the candidate frame, and outputs the location of the target in the image and its category and confidence;

Step 3) Construct the cost function of training RPN network and the cost function of training Fast R-CNN network:

The cost function of training the RPN network in this embodiment is:

Among them, the anchor point and the ground truth (ie, the calibrated real data) have the largest Intersection over Union (IoU) or not less than 0.7 marked as positive samples, and P _i is the prediction confidence;

Is the label value, when it is 1, it means positive sample, and when it is 0, it means negative sample; i means the index of anchor point; N _cls is the total number of anchor points; N _reg is the number of positive samples; t _i is the predicted anchor point bounding box Correction value

Is the actual anchor point bounding box correction value; L _cls is the classification cost; L _reg is the border regression cost; λ is the balance weight;

The cost function of training the Fast R-CNN network in this embodiment is:

L(p,u,t ^u ,v)=L _cls (p,u)+λ[u≥1]L _reg (t ^u ,v)

Among them, u is the u-th category; ^tu is the correction value of the u-th frame regression prediction; v is the actual correction value; L _cls is the classification cost; L _reg is the frame regression cost; λ is the balance weight;

Step 4) Use a standard ZF model to train and fine-tune the parameters of the network, and randomly initialize all new layers by extracting weights from the zero-mean Gaussian distribution of the set standard deviation;

Step 5) Use back propagation algorithm and stochastic gradient descent algorithm to train the model by alternately training RPN and Fast R-CNN networks, adjust the weights of each layer of neural network in turn, and set the initial network learning rate to 0.01, the minimum learning rate is set to 0.0001, the momentum is set to 0.9, the weight attenuation coefficient is set to 0.0005, and the dropout value is set to 0.5. The specific steps are as follows:

(1) Use back propagation algorithm and stochastic gradient algorithm to train RPN model, and iterate 80,000 times in this stage;

(2) Use the candidate frame generated by RPN as the input of Fast R-CNN, and conduct independent training, and iterate 40,000 times in this stage;

(3) Use the results in (2) to initialize the RPN network structure, fix the shared convolutional layer (set the learning rate of the shared convolutional layer to 0), update the parameters of the RPN network, and iterate 80,000 times in this stage;

(4) Fix the shared convolutional layer (set the learning rate of the shared convolutional layer to 0), fine-tune the Fast R-CNN network structure, update the parameters of its fully connected layer, and iterate 40,000 times in this stage;

Step 6) Use the training set to test the pre-trained Faster R-CNN model, and select difficult samples according to the difficult sample discrimination formula of the present invention;

Step 7) Add the difficult samples generated in step 6) to the training set, train the network again, repeat step 5) to strengthen the network's ability to discriminate difficult samples, and finally get the best Faster R-CNN model; training process See Figure 2.

Step 8) Process the vehicle-mounted video image collected in practice, input it into the trained Faster R-CNN model, and output the target category, confidence and target position in the image.

The present invention can fully consider the problem of missed detection of small targets in the Faster R-CNN algorithm, and improve the accuracy of vehicle recognition in complex traffic scenes through deep image feature fusion and difficult sample mining methods.

The target detection algorithm based on the convolutional neural network used in the present invention can learn effective features according to different requirements under the condition of flexible training data to complete specific detection tasks. The R-CNN algorithm is a target detection algorithm based on the combination of candidate frame suggestions and convolutional neural networks. Due to the large number of recommended candidate frames generated by the region recommendation algorithm and the large time overhead, the algorithm is in real-time and accuracy. There is still a lot of room for improvement. The Faster R-CNN algorithm, which uses the shared convolutional network to extract the suggested candidate frames, has been significantly improved in real-time. The algorithm discards the traditional region suggestion algorithm and uses the convolutional layer in the deep network to extract the candidate frames, saving A lot of time is spent, but in scenes with more small targets and more complex, the missed detection is more serious, and there is still much room for improvement.

The above are only the preferred embodiments of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the technical principles of the present invention, several improvements and modifications can be made. These improvements and modifications It should also be regarded as the protection scope of the present invention.

Claims (10)

1. A vehicle-mounted video target detection method based on deep learning is characterized by including the following steps:

   Step 1) Align the pixels under the depth coordinates to the color coordinates; then extract the features of the depth image and the color image through CNN respectively, and concatenate the feature maps output by the respective convolutional layers in the channel dimension to obtain the final RGB -D feature as the convolution feature map after convolution;

   Construct a regional suggestion network RPN, which includes a 3×3 convolutional layer and two 1×1 parallel convolutional layers; input the merged convolution feature map into a 3×3 convolutional layer, Slide a network with a preset size in pixels on the input feature map, and each sliding position will generate an anchor point of a specific scale;

   Input the generated anchor points into two 1×1 parallel convolutional layers for position regression and foreground and background judgment, respectively output the anchor point’s foreground and background confidence and the position of all candidate frames, and select frames from the resulting rectangle according to preset conditions Select a preset number of areas that meet specific conditions in the, and obtain the final area suggestion set C;

   Step 2) Construct Fast R-CNN model:

   The Fast R-CNN model is composed of two ROI pooling layers, a fully connected layer, and two parallel fully connected layers. The confidence of the region and the position of the candidate frame after the border regression are respectively output; the merged volume The product features are input to the Fast R-CNN model, and the position of the target in the output image and its category and confidence are output;

   Step 3): Construct the cost function for training the RPN network and the cost function for training the Fast R-CNN network;

   Step 4) Use a standard ZF model to train and fine-tune the parameters of the network, and randomly initialize all new layers by extracting weights from the zero-mean Gaussian distribution of the set standard deviation;

   Step 5) Use the back propagation algorithm and the stochastic gradient descent algorithm to train the model by alternately training the RPN and Fast R-CNN networks, and adjust the weight of each layer of neural network in turn according to the preset parameters;

   Step 6) Use the pre-obtained training set to test the pre-trained Faster R-CNN model, and screen out difficult samples according to the discriminant formula of difficult samples;

   Step 7) Add the difficult samples generated in step 6) to the training set, train the network again, repeat steps 5)-step 7) to obtain the optimal Faster R-CNN model;

   Step 8) Process the vehicle-mounted video image collected in practice, input it into the trained Faster R-CNN model, and output the target category, confidence and target position in the image.
2. A vehicle-mounted video target detection method based on deep learning according to claim 1, wherein the RGB-D feature in step 2) is used as a convolution feature map shared by RPN and Fast R-CNN, which The matrix form is:

   

   Among them, i, j, k are intermediate variables, i～[0,h-1], j～[0,w-1], k～[0,2c-1], h is the height of the feature map, and w is The width of the feature map, c is the three channels of RGB; Y _RGB (i,j,k) is the image feature after channel concatenation; Y _depth (i,j,kc) is the color image feature; Y _merge (i,j, k) is the depth image feature.
3. A vehicle-mounted video target detection method based on deep learning according to claim 1, wherein the cost function of training RPN network is:

   

   Among them, the anchor point with the largest intersection ratio with the calibrated real data or not less than 0.7 is marked as a positive sample, and P _i is the prediction confidence;

   

   Is the label value, when it is 1, it means positive sample, and when it is 0, it means negative sample; i means the index of anchor point; N _cls is the total number of anchor points; N _reg is the number of positive samples; t _i is the predicted anchor point bounding box Correction value

   

   Is the actual anchor bounding box correction value; L _cls is the classification cost; L _reg is the border regression cost; λ is the balance weight.
4. A vehicle-mounted video target detection method based on deep learning according to claim 1, characterized in that the cost function of training the Fast R-CNN network is:

   L(p,u,t ^u ,v)=L _cls (p,u)+λ[u≥1]L _reg (t ^u ,v)

   Among them, u is the u-th category; ^tu is the correction value of the u-th frame regression prediction; v is the actual correction value; L _reg is the frame regression cost and p is the classification prediction result.
5. A vehicle-mounted video target detection method based on deep learning according to claim 1, characterized in that, in said step 6), said difficult sample discriminant function is as follows:

   L(o,p)=L _IoU (o)+L _score (p),

   

   L _score (p)=(1-p),

   Among them, L _IoU is the frame error; L _score is the classification error; o is the intersection rate between the sample and the target; k is the sensitivity coefficient to the threshold; the value ranges of o and p are both 0 to 1.
6. A vehicle-mounted video target detection method based on deep learning according to claim 1, wherein the specific steps of step 5) are as follows:

   (1) Use back propagation algorithm and stochastic gradient algorithm to train RPN model, and iterate 80,000 times in this stage;

   (2) Use the candidate frame generated by RPN as the input of Fast R-CNN, and conduct independent training, and iterate 40,000 times in this stage;

   (3) Use the results in (2) to initialize the RPN network structure, fix the shared convolutional layer (set the learning rate of the shared convolutional layer to 0), update the parameters of the RPN network, and iterate 80,000 times in this stage;

   (4) Fix the shared convolutional layer (set the learning rate of the shared convolutional layer to 0), fine-tune the Fast R-CNN network structure, update the parameters of its fully connected layer, and iterate 40,000 times in this stage.
7. A vehicle-mounted video target detection method based on deep learning according to claim 1, wherein the parameter setting in step 5) includes the initial network learning rate is set to 0.01, the minimum learning rate is set to 0.0001, and the momentum is set to 0.9 , The weight attenuation coefficient is 0.0005, and the Dropout value is set to 0.5.
8. A vehicle-mounted video target detection method based on deep learning according to claim 1, wherein the method of obtaining a training set and a test set in advance comprises the following steps:

   Use KITTI data set to construct training sample set and test sample set;

   According to the PASCAL VOC format conversion KITTI category, the KITTI data set is divided into three categories: vehicle, pedestrian and traffic;

   Convert annotation information: Convert the annotation file from txt to xml, remove other information in the annotation, and leave only 3 categories; finally, generate the training validation set and the test set.
9. A vehicle-mounted video target detection method based on deep learning according to claim 1, characterized in that the method of filtering out a preset number of regions meeting specific conditions is as follows:

   The obtained rectangular candidate boxes are sorted in descending order according to the softmax score, the first 2000 regions are retained, and the non-maximum suppression algorithm is further used to filter out the first specific number of regions with the highest foreground confidence.
10. A vehicle-mounted video target detection method based on deep learning according to claim 1, wherein the standard deviation set in step 4) is 0.01.

Images