CN115171074A - Vehicle target identification method based on multi-scale yolo algorithm - Google Patents

Abstract

A vehicle target recognition method based on a multi-scale yolo algorithm belongs to the field of target recognition methods. The problems of instantaneity, accuracy, robustness and the like of the existing target detection method in a complex environment are to be improved. A vehicle target identification method based on a multi-scale yolo algorithm is realized by the following steps: preprocessing a data set; extracting features of the backbone network; performing feature fusion on the PANet; a step of non-maximum inhibition of NMS; outputting a target calibration decision; in addition, based on the influence of the class imbalance problem of the sample data set on the classification precision, a multi-loss function alternative training strategy is adopted, and the cross entropy loss function and the focusing loss function are alternately used at different stages of network training, so that the problem of sample imbalance is solved.

Description

A vehicle target recognition method based on multi-scale yolo algorithm

technical field

The invention relates to a vehicle identification method, in particular to a vehicle target identification method based on a multi-scale yolo algorithm.

Background technique

Deep learning has good self-learning ability, and also has strong expression and processing ability. Today's target detection field is usually done using deep learning. Convolutional Neural Networks (CNN) is one of the most widely used expressions in deep learning. Convolutional Neural Networks R-CNN [6] to Fast R-CNN, Faster R-CNN, Cascade R-CNN, etc. The model and algorithm have been continuously developed and improved, and the detection accuracy and detection efficiency have been greatly improved. Therefore, the algorithm model based on deep learning will become one of the most extensive in the field of object detection.

Intelligent assisted driving technology requires a lot of image recognition and processing work, often the information collected by the vehicle is input in the form of video or image, and the on-board computer needs to identify valuable targets and content based on these visual information. Provide a guarantee for the next vehicle behavior decision. Therefore, the target in the image can be correctly and quickly identified, which is the basis of intelligent assisted driving technology. Although target detection technology has achieved some results, how to improve its real-time, accuracy, robustness and other issues in a more complex environment is still a hot area that needs to be studied urgently.

SUMMARY OF THE INVENTION

The purpose of the present invention is to solve the problems of real-time, accuracy, robustness and other problems that the existing target detection methods show in a relatively complex environment to be improved, and propose a vehicle based on the multi-scale yolo algorithm. target recognition method.

A vehicle target recognition method based on multi-scale yolo algorithm, the method is realized by the following steps:

Data set preprocessing steps;

The steps of backbone network feature extraction;

The steps of feature fusion in PANet;

The step of NMS non-maximal suppression;

The steps of target calibration decision output;

In addition, based on the impact of the class imbalance problem of the sample data set on the classification accuracy, an alternate training strategy of multiple loss functions is adopted, and the cross-entropy loss function and the focus loss function are used alternately in different stages of network training to improve the sample imbalance. The problem.

Preferably, the described backbone network feature extraction step specifically includes:

(1) The steps of designing the convolution algorithm;

The convolution operation means that each pixel in the output image is obtained by the weighted average of the pixels in the small area of the corresponding position of the input image, and this area is called the convolution kernel. Through the convolution operation between the image and the convolution kernel, some features of the image are extracted;

A pixel array of an 8*8 two-dimensional grayscale image and a 3*3 convolution kernel; if the convolution kernel moves with a step size of 1, that is, only one small grid is moved each time it moves, and when the convolution kernel moves When reaching the i-th row and the j-th column, the input image and the corresponding position values of the convolution kernel are multiplied in turn and then averaged, and the output value of the i-th row and the j-th column of the output image can be determined. For example, the value of the second row and the third column of the output value should be: [1*1+2*0+3*(-1)+4*1+5*0+6*(-1)+ 7*1+8*0+9*(-1)], which is -6; the number of layers of the convolution kernel is equal to the number of layers of the input data. If the input image is a three-dimensional color image, the number of layers of the convolution kernel is also It should be three layers, the three-dimensional convolution operation is basically the same as the two-dimensional convolution operation, and the output value is the weighted average of the value of the input value and the corresponding position of the convolution kernel;

A convolution layer contains multiple convolution kernels. The number of layers of the output pixel array after the convolution layer is related to the number of convolution kernels. If the convolution layer contains n convolution kernels, the convolution layer The number of layers of the output array is also n;

(2) the steps of designing the activation function;

After the convolution layer, the activation function is used to nonlinearize the data. If the input and output are always linear, then after many layers, the total input and output are still not linear, so no matter how many layers there are in the middle, it is equivalent to one layer, as shown in the formula;

Y=aX+b; Z=cY+d

Z=c(aX+b)+d=(ac)X+(bc+d)

Select the ReLU function as the activation function;

(3) The steps of designing the pooling algorithm;

After the input image is subjected to the convolution operation, the pooling operation is performed;

If the output data is the maximum value of the input data at the corresponding position of the pooling window, it is maximum pooling; if the output data is the average value of the input data at the corresponding position of the pooling window, it is average pooling;

(4) the steps of performing a spatial pyramid pooling structure;

Introduce the SPP structure to establish the mapping relationship between the candidate region and the input feature map;

(5) The steps of designing the MobileNetv3 network structure;

The input D×D×3 feature map is convolved with a 3×3 convolution kernel to output a D×D×N feature map; the standard convolution process is N 3×3 convolution kernels and input features Each channel of the graph is convolved, and finally a new feature map with N channels is obtained;

The depthwise separable convolution first uses three 3×3 convolution kernels to convolve with each channel of the input feature map respectively, and obtains a feature map whose input channel is equal to the output channel, and then uses N 1×1 convolution kernels to check. This feature map is convolved to obtain a new feature map of N channels;

Calculate the amount of parameters used by the two convolutions respectively, and the results are shown in the following formulas:

P ₁ =D×D×3×N (1)

P ₂ =D×D×3+D×D×1×N (2)

P ₁ is the amount of parameters used in standard convolution, P ₂ is the amount of parameters used when using depthwise separable convolution, D is the length and width of the input feature map, and N is the number of convolution kernels;

When performing standard convolution, the number of input channels is much smaller than the number of output channels, and formula (3) is obtained by comparing formula (1) and formula (2):

It can be seen that the result of P ₂ /P ₁ is much less than 1. After using the depthwise separable convolution, the amount of parameters used in the convolution can be greatly reduced while the effect is similar to that of the standard convolution;

(6) The steps of designing the structure of RFBs;

The hole convolution is introduced into the structure module; the RFBs structure first performs 1×1 convolution on the feature map for channel transformation, and then performs multi-branch hole structure processing to obtain the multi-scale information features of the target; the multi-branch structure uses a common convolution layer. Combined with the structure of the hole convolution layer, the 3×3 convolution kernel in the original RFB structure in the ordinary convolution layer is replaced by the parallel 1×3 and 3×1 convolution kernels, and the 5×5 convolution kernel is replaced by two series 1 convolution kernels. Instead of ×3 and 3×1 convolution kernels, the hole convolution layer consists of 3 convolution kernels with a size of 3 × 3, and the expansion rates of the convolution kernels are 1, 3, and 5, respectively, to prevent the expansion rate from being too high. The convolution layer is degraded due to the large size; finally, the feature layers of different sizes processed by the multi-branch hole structure are subjected to the Concat operation, and a new fusion feature layer is output;

(7) The steps of the improved SPPNet;

Inspired by the backbone feature extraction network CSPDarknet in YOLOv4, the CSP structure is introduced into SPP. Before fusing multi-scale features, the network is divided into two parts, and some features are connected by shortcuts and directly merged with the features after SPP fusion.

Preferably, the step that the described PANet performs feature fusion is specifically:

The CSPDarknet-53 network contains a large number of convolution operations, using a large number of 3 × 3, 1 × 1 convolutional residual modules to stack, and use a 3 × 3 convolutional layer with a stride of 2 in it to combine features. The size of the graph is reduced by 1/2; Residual is a residual module, and the n× on the right side of the rectangular box is the number of times the residual module is reused; after each pass through the residual module, it can be found that the step size used is 2 of the 3×3 convolutional layers are used 5 times in total. Each time the convolutional layer is used, the length and width of the feature map will become 1/2 of the original, instead of the convolutional network using the pooling layer for downsampling. ; In the prediction stage, the feature layers F1, F2 and F3 obtained by the CSPDarknet-53 feature extraction network are respectively input into the multi-scale prediction network, and F3 obtains the coarse-scale feature layer 3 through the convolution operation, which is used to detect large-scale targets; After upsampling, layer 3 is first fused with F2, and then convolved to obtain mesoscale feature layer 2, which is used to detect mesoscale targets; feature layer 2 is then upsampled and convolved with F1 fusion, and then fine-scale feature layer 1 is obtained. In order to detect small-scale targets, this Feature Pyramid Network (FPN) structure enables the algorithm to achieve better detection results for targets of different sizes and scales; Combination, the final detection result is obtained through the non-maximum suppression post-processing algorithm;

The obtained 9 different sizes of prediction boxes are (10×13), (16×30), (33×23), (30×61), (62×45), (59×119), (116× 90), (156×198), (373×326); apply (116×90), (156×198), (373×326) on the 13×13 coarse-scale feature layer 3 to detect larger objects ;Apply medium-sized prediction boxes (30×61), (62×45), (59×119) on the 26×26 mesoscale feature layer 2 to detect medium-sized objects; The scale features (10×13), (16×30), (33×23) are applied on layer 1 for detecting smaller objects.

The beneficial effects of the present invention are:

The present invention understands the basic experimental method and experimental principle basis of YOLOv4 through the research on the YOLOV4 network, and clarifies the basic structure of YOLOv4 and how each network directly transmits data information. According to the feasibility of the experimental environment and its network structure The existing problems such as missed detection of small targets, occlusion problems, detection problems in complex backgrounds, and multi-scale target detection are modified to the network to improve the target detection accuracy of the network, and the network is pruned to reduce network parameters and improve model training. speed.

1. Compared with YOLOv4's backbone feature extraction network structure CSPDarknet residual skip connection method, this paper uses MobileNetv3 instead of the original backbone feature extraction network because the depth separable convolution of MobileNetv3 is conducive to optimizing the network structure and reducing network training parameters, so In this paper, MobileNetv3 is used as the backbone feature extraction network to strengthen the effective use and transmission of features in the network. The network can learn more feature information, making the structure of the network more lightweight, thereby improving the speed of target detection.

2. Inspired by the backbone feature extraction network CSPDarknet in YOLOv4, this paper introduces the CSP structure into SPP. Before fusing multi-scale features, the network is divided into two parts, and some features are connected by shortcuts and directly fused with SPP features Combined, this operation reduces the amount of computation by 40%. This structure is formed by the fusion of multiple shallow networks. The shallow network does not have the problem of vanishing gradient during training, which can accelerate the convergence of the network.

3. Since the low-level feature map has less semantic information, the target location is accurate; the high-level feature map has rich semantic information and rough target location. By introducing RFBNet, the original receptive field of the network is multi-integrated and the hole convolution is introduced. To achieve the purpose of increasing the receptive field and fusing features of different sizes.

Description of drawings

Fig. 1 is the method flow chart of the present invention;

Fig. 2 is the basic method of convolution operation involved in the present invention;

Fig. 3 is the output layer number of the pixel array involved in the present invention after passing through the convolution layer;

Fig. 4 is two kinds of activation functions involved in the present invention;

Fig. 5 is two kinds of pooling methods involved in the present invention;

6 is the number of output layers of the pixel array involved in the present invention after passing through the pooling layer;

Fig. 7 is the SPP structure of the fixed size (21 dimensions) output involved in the present invention;

FIG. 8 is the network structure of YOLOv3 involved in the present invention.

Detailed ways

Specific implementation one:

A vehicle target recognition method based on the multi-scale yolo algorithm of the present embodiment, as shown in FIG. 1 , the method is implemented through the following steps:

Data set preprocessing steps;

The steps of backbone network feature extraction;

The steps of feature fusion in PANet;

The step of NMS non-maximal suppression;

The steps of target calibration decision output;

In addition, based on the impact of the class imbalance problem of the sample data set on the classification accuracy, an alternate training strategy of multiple loss functions is adopted, and the cross-entropy loss function and the focus loss function are used alternately in different stages of network training to improve the sample imbalance. The problem.

Among them, the sample imbalance mainly includes two aspects. On the one hand, it means that most of the samples in the data set are relatively easy to learn, that is, the so-called "simple samples". Such samples occupy the majority of the entire dataset, and their features are relatively distinct and concentrated, and their surrounding background interference is not strong, so they can be easily learned and identified by the network. On the contrary, it is a "difficult sample". This kind of sample has weak features due to its small size and close distance from other samples, or is subject to lack of main features due to surrounding occlusion and changes in light and shadow. It is difficult to learn its discriminative features, and the detection effect is not good. In addition, the frequency of difficult samples in the data set is low, and the proportion of difficult samples is not balanced with that of simple samples. It is difficult to obtain sufficient training during training, which further exacerbates the problems of difficult learning and inaccurate detection.

The present invention does not use the focus loss function to completely replace the cross-entropy loss function for network training, because we found in the actual training and testing of the XDUAV data set that simply using the focus loss function function can be used for some rare categories. (such as buses and oil tankers), the effect is greatly improved, but the effect is not obvious for categories with few and small sizes, and their own characteristics are also weak (such as bicycles and motorcycles). Our analysis believes that these samples have very limited characteristics. Although the focus loss function increases the training weight of these samples, the lack of data still makes it difficult for the network to learn the feature expressions of these categories. Therefore, we use the cross-entropy loss function in the early stage of training, the purpose of which is to learn the overall feature distribution of the samples. For categories with less data such as bicycles and motorcycles, since their features are relatively similar, the cross-entropy loss function enables the network to learn these similar categories as a whole. Then we switch to a focused loss function and train the network to learn discriminative features between these samples of similar categories, so as to better distinguish these samples.

Specific implementation two:

Different from the first embodiment, in a vehicle target recognition method based on a multi-scale yolo algorithm in this embodiment, the steps of the backbone network feature extraction specifically include:

(1) The steps of designing the convolution algorithm;

The convolution operation means that each pixel in the output image is obtained by the weighted average of the pixels in the small area of the corresponding position of the input image, and this area is called the convolution kernel. Through the convolution operation between the image and the convolution kernel, some features of the image are extracted. The basic method of the convolution operation is shown in Figure 2.

In the figure, there is an 8*8 two-dimensional grayscale image pixel array and a 3*3 convolution kernel; if the convolution kernel moves with a step size of 1, that is, only one small cell is moved each time, when the convolution kernel moves When the product kernel moves to the ith row and the jth column, the input image and the corresponding position values of the convolution kernel are multiplied in turn and then averaged, and the output value of the ith row and the jth column of the output image can be determined, as shown in Figure 3 For example, the value of the second row and the third column of the output value should be: [1*1+2*0+3*(-1)+4*1+5*0+6*(- 1)+7*1+8*0+9*(-1)], which is -6; the number of layers of the convolution kernel is equal to the number of layers of the input data. If the input image is a three-dimensional color image, the convolution kernel The number of layers should also be three layers. The three-dimensional convolution operation is basically the same as the two-dimensional convolution operation. The output value is the weighted average of the value of the input value and the corresponding position of the convolution kernel; when designing the network, set the size of the step size. and whether to fill in the input data;

A convolution layer contains multiple convolution kernels. The number of layers of the output pixel array after the convolution layer is related to the number of convolution kernels. If the convolution layer contains n convolution kernels, the convolution layer The number of layers of the latter output array is also n, as shown in Figure 3.

(2) the steps of designing the activation function;

After the convolutional layer, the activation function is used to non-linearize the data, so that the relationship between input and output is not linear, which can capture more complex changes in the input. If the input and output are always in a linear relationship, then after many layers, the total input and output are still not in a linear relationship, then no matter how many layers there are in the middle, it is equivalent to one layer, as shown in the formula;

Y=aX+b; Z=cY+d

Z=c(aX+b)+d=(ac)X+(bc+d)

Select the ReLU function as the activation function;

Commonly used activation functions include ReLU function and Mish function. The function image is shown in Figure 4. Since the output value of the Mish function does not change much when the input value is too large or too small, the ReLU function can avoid the disappearance of the gradient and improve the training speed. greater advantage;

(3) The steps of designing the pooling algorithm;

After the input image is subjected to the convolution operation, each pixel in the graph may affect multiple output points, which may cause information redundancy and reduce the performance of the algorithm. Therefore, a pooling operation is required. The most common algorithm in the pooling operation is the average pooling. As shown in Figure 5, the most commonly used is the maximum pooling;

If the output data is the maximum value of the input data at the corresponding position of the pooling window, it is maximum pooling; if the output data is the average value of the input data at the corresponding position of the pooling window, it is average pooling;

The pooling algorithm is similar to the convolution algorithm, but the number of layers of the convolution kernel needs to be the same as the number of layers of the input data, but the pooling layer is a two-dimensional data array, and the pooling window moves row by row, column by column, and layer by layer. The number of layers of the output data after the pooling layer is the same as the number of layers of the input data, as shown in Figure 6.

(4) The step of performing the spatial pyramid pooling structure (SPPNet);

The SPP structure is generally between the convolutional layer and the fully connected layer, and is used to generate a fixed-size output for an input of any size. The processing process is as follows: Parallel use of 1×1, 2×2 and 4×4 layer) to pool the input feature map of any size to obtain 1+4+16=21 different image blocks, and then calculate the maximum value (maximum pooling) or average value (average pooling) of each image block respectively. ) to obtain a feature vector with a fixed size of 21 dimensions, which can be directly connected to the fully connected layer. By adjusting the size and number of panes, features are extracted from receptive fields of different sizes; then the output and input feature maps of the pooling layer of three specifications are spliced to obtain a fused feature vector. The output of any size can be generated, which is the most important feature of the SPP structure. Figure 7 shows the SPP structure.

Taking the classic two-stage target detection algorithm R-CNN as an example, the algorithm generates a large number of candidate regions (about 2000) in the region proposal stage, and the candidate regions generate fixed-size input images through operations such as cropping or distortion, and then use CNN. The model processes the input image to obtain a fixed-size feature vector, which can then be fed into the fully connected layer for classification and regression. It can be seen that all the generated candidate regions need to be fed into the CNN model to generate fixed-size feature vectors, and the huge number of candidate regions will inevitably lead to inefficiency of the algorithm. After introducing the SPP structure to optimize the R-CNN algorithm, the input image can be directly sent to the CNN model to obtain the overall input feature map, and then the mapping relationship between the candidate region and the input feature map can be established to obtain all candidate regions. The corresponding feature vectors (without going through the CNN model), and finally perform similar classification and regression on these feature vectors. Introducing the SPP structure, only the mapping relationship between the candidate region and the input feature map needs to be established, and the forward calculation of the CNN model does not need to be repeated. High detection accuracy;

(5) The steps of designing the MobileNetv3 network structure;

The core idea of MobileNet is to decompose a complete convolution operation into two steps, namely depthwise convolution and pointwise convolution. Efficient neural network mainly through: 1. Reduce the number of parameters; 2. Quantize parameters to reduce the memory occupied by each parameter. The current research summary is divided into two directions: one is to compress the trained complex model to obtain a small model; the other is to directly design and train the small model (Mobile Net belongs to this category).

The input D×D×3 feature map is convolved with a 3×3 convolution kernel to output a D×D×N feature map; the standard convolution process is N 3×3 convolution kernels and input features Each channel of the graph is convolved, and finally a new feature map with N channels is obtained;

The depthwise separable convolution first uses three 3×3 convolution kernels to convolve with each channel of the input feature map respectively, and obtains a feature map whose input channel is equal to the output channel, and then uses N 1×1 convolution kernels to check. This feature map is convolved to obtain a new feature map of N channels;

Calculate the amount of parameters used by the two convolutions respectively, and the results are shown in the following formulas:

P ₁ =D×D×3×N (1)

P ₂ =D×D×3+D×D×1×N (2)

P ₁ is the amount of parameters used in standard convolution, P ₂ is the amount of parameters used when using depthwise separable convolution, D is the length and width of the input feature map, and N is the number of convolution kernels;

When performing standard convolution, the number of input channels is much smaller than the number of output channels, and formula (3) is obtained by comparing formula (1) and formula (2):

It can be seen that the result of P ₂ /P ₁ is much less than 1. After using the depthwise separable convolution, the amount of parameters used in the convolution can be greatly reduced while the effect is similar to that of the standard convolution;

(6) The steps of designing the structure of RFBs;

In order to capture the multi-scale feature information of pedestrians, the RFBs structural module is connected after the improved backbone network; the hollow convolution is introduced into the structural module to increase the receptive field and fuse features of different sizes;

The RFBs structure first performs 1×1 convolution on the feature map for channel transformation, and then performs multi-branch hole structure processing to obtain the multi-scale information features of the target; the multi-branch structure adopts the structure of ordinary convolution layer combined with hole convolution layer, The 3×3 convolution kernel in the original RFB structure in the ordinary convolution layer is replaced by parallel 1×3 and 3×1 convolution kernels, and the 5×5 convolution kernel is replaced by two series 1×3 and 3×1 convolution kernels Instead of cores, this can effectively reduce the computational load of the network and ensure that the entire network is lightweight. Among them, the hole convolution layer is composed of three convolution kernels with a size of 3 × 3, and the expansion rates of the convolution kernels are 1, 3, and 5, respectively, to prevent the convolution layer from degrading due to too much expansion rate; The feature layers of different sizes processed by the branch hole structure are subjected to Concat operation, and a new fusion feature layer is output; in order to retain the original information of the input feature map, the new fusion feature layer is transformed through a 1×1 convolutional layer. The large residual edges formed by the original feature map are accumulated and output.

(7) The steps of the improved SPPNet;

Inspired by the backbone feature extraction network CSPDarknet in YOLOv4, the CSP structure is introduced into SPP. Before fusing multi-scale features, the network is divided into two parts. Some features are connected by shortcuts and directly merged with the features after SPP fusion. Operations reduce the amount of computation by 40%. This structure is formed by the fusion of multiple shallow networks. The shallow network will not have the problem of vanishing gradient during training, which can accelerate the convergence of the network.

Specific implementation three:

Different from the second embodiment, in a vehicle target recognition method based on the multi-scale yolo algorithm in this embodiment, the steps of the PANet for feature fusion are specifically:

The YOLOv4 algorithm is an improved algorithm based on YOLOv2 and YOLOv23. It is different from the Two-stage target detection algorithm such as Faster R-CNN. It divides the image into different grids, each grid is responsible for the corresponding object, supports multi-category target detection, and can Achieve faster detection while maintaining accuracy. The network structure of YOLOv3 is shown in Figure 8. It is an end-to-end real-time target detection framework, which mainly includes Darknet-53 backbone feature extraction network and multi-scale feature fusion prediction network.

The CSPDarknet-53 network contains a large number of convolution operations, using a large number of 3 × 3, 1 × 1 convolutional residual modules to stack, and use a 3 × 3 convolutional layer with a stride of 2 in it to combine features. The size of the figure is reduced by 1/2; the Residual in the rectangular box in the figure is a residual module, and the n× on the right side of the rectangular box is the number of times the residual module is reused; after each residual module, you can It was found that a 3×3 convolutional layer with a stride of 2 was used for a total of 5 times. Each time a convolutional layer was used, the length and width of the feature map would become 1/2 of the original, thus replacing the traditional convolutional network. Use the pooling layer for downsampling; for example, input a 256×256 image, after such a fully convolutional network, it will perform 1/2 reduction 5 times, and a feature map with a size of 8×8 can be obtained ( 1/2 to the fifth power is 1/32). Its main feature is the use of residual unit Residual Block. By using the residual skip connection structure, the learning channel between the input layer and the output layer is increased, and the problem of gradient disappearance when deepening the number of network layers is reduced. The convolution layer in the Residual Block alternately uses two convolution kernels of different sizes, 1×1 and 3×3. After each convolution, the BN batch normalization layer is connected, and the input data is set to have a mean value of 0 and a variance of 1. Normalization processing, using Leaky Re LU activation function for nonlinear operation after the BN layer, enables the network to be applied in nonlinear models.

In the prediction stage, the feature layers F1, F2 and F3 obtained by the CSPDarknet-53 feature extraction network are respectively input into the multi-scale prediction network, and F3 is obtained through the convolution operation to obtain the coarse-scale feature layer 3 for detecting large-scale targets; 3 After upsampling, it is first fused with F2, and then convolved to obtain a mesoscale feature layer 2, which is used to detect mesoscale targets; the feature layer 2 is then upsampled and convolved with F1 fusion, and then a fine-scale feature layer 1 is obtained. Detecting small-scale targets, this feature pyramid network (Feature Pyramid Network, FPN) structure enables the algorithm to achieve better detection results for targets of different sizes and scales; Combination, the final detection result is obtained through the non-maximum suppression post-processing algorithm;

Since the more convolutional layers, the more the feature information of the input image is lost, so the YOLOv4 network uses K-means clustering to obtain the size of the prediction box, and sets 3 different prediction boxes for each scale in the feature pyramid network, a total of The obtained 9 different sizes of prediction boxes are (10×13), (16×30), (33×23), (30×61), (62×45), (59×119), (116× 90), (156×198), (373×326); apply (116×90), (156×198), (373×326) on the 13×13 coarse-scale feature layer 3 to detect larger objects ;Apply medium-sized prediction boxes (30×61), (62×45), (59×119) on the 26×26 mesoscale feature layer 2 to detect medium-sized objects; Apply (10×13), (16×30), (33×23) on scale feature layer 1 for detecting smaller objects;

Improve detection accuracy and speed with YOLOv4 network.

Simulation:

The hardware configuration used to create the experimental environment is Inter(R)Core i7-9700K CPU, NVIDIA Geforce RTX 2080Ti graphics card, Windows operating system, software environment is CUDA11.0, Cudnn8.0, using Tensorflow1.14 deep learning framework. The Adam optimizer is used for training, the initial learning rate is 0.001, the momentum size is 0.9, the Batchsize is set to 8, the number of iterations is 500, and the Mosaic data enhancement technique and Dropblock regularization method are used. The data are randomly divided into training set, validation set, and test set according to 6:2:2. Before training, the input image size was adjusted to 608×608, and the bounding box width, height and center point coordinates of the annotation information were normalized in accordance with the PASCAL VOC dataset format to reduce the impact of abnormal samples on the data.

In the detection process, the prediction result and the Io U value of the actual target are used to measure whether the target position is successfully predicted. The IoU threshold is set to 0.5, that is, if the IoU is greater than 0.5, the prediction is correct, otherwise, the prediction is wrong. Using Mean Average Precision (mAP) and recall as the precision metrics of the network,

P stands for Precision, which refers to the probability of correct detection among all detected targets, and R stands for Recall, which refers to the probability of correct detection among all positive samples. The calculation formulas of Precision and Recall are as follows:

TP refers to the number of correctly predicted positive samples, FP refers to the number of predicted positive samples but actually were negative samples, and FN refers to the number of predicted negative samples but actually positive samples. In practical applications, the network is often deployed on mobile devices, so the scale and detection speed of the network cannot be ignored. The size of the network is determined by the amount of parameters or weights, and the detection speed is determined by FPS (Frames Per Second), which is defined as the number of images that can be detected per second.

Compared with YOLOv4's backbone feature extraction network structure CSPDarknet residual skip connection, MobileNetv3's depth separable convolution is beneficial to optimize network structure and reduce network training parameters. Therefore, this paper uses MobileNetv3 structure to replace CSPDarknet module to enhance the effectiveness of features in the network. Utilization and transmission enable the network to learn more feature information, thereby improving the speed of target detection; because the low-level feature map has less semantic information, the target position is accurate; the high-level feature map has richer semantic information, and the target position is relatively rough. RFBNet multi-integrates the original receptive field of the network, and introduces atrous convolution to achieve the purpose of increasing the receptive field and fusing features of different sizes; inspired by the backbone feature extraction network CSPDarknet in YOLOv4, this paper introduces the CSP structure into SPP In , before fusing multi-scale features, the network is divided into two parts, and a part of the features is connected by shortcuts and directly merged with the features after SPP fusion, this operation reduces the amount of computation by 40%. This structure is formed by the fusion of multiple shallow networks. The shallow network does not have the problem of vanishing gradient during training, which can accelerate the convergence of the network.

The embodiment of the present invention announces the preferred embodiment, but is not limited to this, those of ordinary skill in the art can easily understand the spirit of the present invention according to the above-mentioned embodiment, and make different extensions and changes, However, as long as they do not depart from the spirit of the present invention, they are all within the protection scope of the present invention.

Claims (3)

1. a vehicle target recognition method based on multi-scale yolo algorithm, is characterized in that: described method realizes by following steps: Data set preprocessing steps; The steps of backbone network feature extraction; The steps of feature fusion in PANet; The step of NMS non-maximal suppression; The steps of target calibration decision output; In addition, based on the impact of the class imbalance problem of the sample data set on the classification accuracy, an alternate training strategy of multiple loss functions is adopted, and the cross-entropy loss function and the focus loss function are used alternately in different stages of network training to improve the sample imbalance. The problem. 2. a kind of vehicle target identification method based on multi-scale yolo algorithm according to claim 1, is characterized in that: the step of described backbone network feature extraction, specifically comprises: (1) The steps of designing the convolution algorithm; The convolution operation means that each pixel in the output image is obtained by the weighted average of the pixels in the small area of the corresponding position of the input image. certain characteristics; A pixel array of an 8*8 two-dimensional grayscale image and a 3*3 convolution kernel; if the convolution kernel moves with a step size of 1, that is, only one small grid is moved each time it moves, and when the convolution kernel moves When reaching the i-th row and the j-th column, the input image and the corresponding position values of the convolution kernel are multiplied in turn and then averaged, and the output value of the i-th row and the j-th column of the output image can be determined. For example, the value of the second row and the third column of the output value should be: [1*1+2*0+3*(-1)+4*1+5*0+6*(-1)+ 7*1+8*0+9*(-1)], which is -6; the number of layers of the convolution kernel is equal to the number of layers of the input data. If the input image is a three-dimensional color image, the number of layers of the convolution kernel is also It should be three layers, the three-dimensional convolution operation is basically the same as the two-dimensional convolution operation, and the output value is the weighted average of the value of the input value and the corresponding position of the convolution kernel; A convolution layer contains multiple convolution kernels. The number of layers of the output pixel array after the convolution layer is related to the number of convolution kernels. If the convolution layer contains n convolution kernels, the convolution layer The number of layers of the output array is also n; (2) the steps of designing the activation function; After the convolution layer, the activation function is used to nonlinearize the data. If the input and output are always linear, then after many layers, the total input and output are still not linear, so no matter how many layers there are in the middle, it is equivalent to one layer, as shown in the formula; Y=aX+b; Z=cY+d Z=c(aX+b)+d=(ac)X+(bc+d) Select the ReLU function as the activation function; (3) The steps of designing the pooling algorithm; After the input image is subjected to the convolution operation, the pooling operation is performed; If the output data is the maximum value of the input data at the corresponding position of the pooling window, it is maximum pooling; if the output data is the average value of the input data at the corresponding position of the pooling window, it is average pooling; (4) the steps of performing a spatial pyramid pooling structure; Introduce the SPP structure to establish the mapping relationship between the candidate region and the input feature map; (5) The steps of designing the MobileNetv3 network structure; The input D×D×3 feature map is convolved with a 3×3 convolution kernel to output a D×D×N feature map; the standard convolution process is N 3×3 convolution kernels and input features Each channel of the graph is convolved, and finally a new feature map with N channels is obtained; The depthwise separable convolution first uses three 3×3 convolution kernels to convolve with each channel of the input feature map respectively, and obtains a feature map whose input channel is equal to the output channel, and then uses N 1×1 convolution kernels to check. This feature map is convolved to obtain a new feature map of N channels; Calculate the amount of parameters used by the two convolutions respectively, and the results are shown in the following formulas: P ₁ =D×D×3×N (1) P ₂ =D×D×3+D×D×1×N (2) P ₁ is the amount of parameters used in standard convolution, P ₂ is the amount of parameters used when using depthwise separable convolution, D is the length and width of the input feature map, and N is the number of convolution kernels; When performing standard convolution, the number of input channels is much smaller than the number of output channels, and formula (3) is obtained by comparing formula (1) and formula (2):

It can be seen that the result of P ₂ /P ₁ is much less than 1. After using the depthwise separable convolution, the amount of parameters used in the convolution can be greatly reduced while the effect is similar to that of the standard convolution; (6) The steps of designing the structure of RFBs; The hole convolution is introduced into the structure module; the RFBs structure first performs 1×1 convolution on the feature map for channel transformation, and then performs multi-branch hole structure processing to obtain the multi-scale information features of the target; the multi-branch structure uses a common convolution layer. Combined with the structure of the hole convolution layer, the 3×3 convolution kernel in the original RFB structure in the ordinary convolution layer is replaced by the parallel 1×3 and 3×1 convolution kernels, and the 5×5 convolution kernel is replaced by two series 1 convolution kernels. Instead of ×3 and 3×1 convolution kernels, the hole convolution layer consists of 3 convolution kernels with a size of 3 × 3, and the expansion rates of the convolution kernels are 1, 3, and 5, respectively, to prevent the expansion rate from being too high. The convolution layer is degraded due to the large size; finally, the feature layers of different sizes processed by the multi-branch hole structure are subjected to the Concat operation, and a new fusion feature layer is output; (7) The steps of the improved SPPNet; Inspired by the backbone feature extraction network CSPDarknet in YOLOv4, the CSP structure is introduced into SPP. Before fusing multi-scale features, the network is divided into two parts, and some features are connected by shortcuts and directly merged with the features after SPP fusion. 3. a kind of vehicle target recognition method based on multi-scale yolo algorithm according to claim 1 and 2, is characterized in that: the step that described PANet carries out feature fusion is specifically: The CSPDarknet-53 network contains a large number of convolution operations, using a large number of 3 × 3, 1 × 1 convolutional residual modules to stack, and use a 3 × 3 convolutional layer with a stride of 2 in it to combine features. The size of the graph is reduced by 1/2; Residual is a residual module, and the n× on the right side of the rectangular box is the number of times the residual module is reused; after each pass through the residual module, it can be found that the step size used is 2 of the 3×3 convolutional layers are used 5 times in total. Each time the convolutional layer is used, the length and width of the feature map will become 1/2 of the original, instead of the convolutional network using the pooling layer for downsampling. ; In the prediction stage, the feature layers F1, F2 and F3 obtained by the CSPDarknet-53 feature extraction network are respectively input into the multi-scale prediction network, and F3 obtains the coarse-scale feature layer 3 through the convolution operation, which is used to detect large-scale targets; After upsampling, layer 3 is first fused with F2, and then convolved to obtain mesoscale feature layer 2, which is used to detect mesoscale targets; feature layer 2 is then upsampled and convolved with F1 fusion, and then fine-scale feature layer 1 is obtained. In order to detect small-scale targets, this feature pyramid network (Feature Pyramid Network, FPN) structure enables the algorithm to achieve better detection results for targets of different sizes and scales; Combination, the final detection result is obtained through the non-maximum suppression post-processing algorithm; The obtained 9 different sizes of prediction boxes are (10×13), (16×30), (33×23), (30×61), (62×45), (59×119), (116× 90), (156×198), (373×326); apply (116×90), (156×198), (373×326) on the 13×13 coarse-scale feature layer 3 to detect larger objects ;Apply medium-sized prediction boxes (30×61), (62×45), (59×119) on the 26×26 mesoscale feature layer 2 to detect medium-sized objects; The scale features (10×13), (16×30), (33×23) are applied on layer 1 for detecting smaller objects.

Images