CN115690704A - LG-CenterNet model-based complex road scene target detection method and device - Google Patents

Abstract

The invention discloses a complex road scene target detection method and a device based on an LG-CenterNet model, which are used for collecting an original road image data set to prepare a data set, extracting a feature pair by constructing the LG-CenterNet network model and using ResNet50 as a backsbone of the model, and guiding the features of different levels while improving the sense of the feature pattern of a main network by adopting a level guiding attention mechanism; inputting the feature graph processed by the level guide mechanism into a scaleseEncoder module for processing; performing characteristic pixel reduction by adopting a deconvolution module; a new feature enhancement module is adopted for the restored features to solve the problem of feature information loss in the pixel restoration process; and finally, inputting the enhanced feature map into a Center points prediction module for road target class identification and position positioning. The average recognition precision of the self-built complex road scene data set is 86.93%, the road scene target image detection speed reaches 50 frames/s, and the requirements on accurate detection and real-time detection of a road scene can be met.

Description

Object detection method and device for complex road scenes based on LG-CenterNet model

technical field

The invention belongs to the fields of semantic segmentation, image processing and intelligent driving, and in particular relates to a complex road scene object detection method and device based on an LG-CenterNet model.

Background technique

In recent years, the steady increase in the number of cars has led to frequent traffic accidents, which seriously threaten the safety of people's lives. Nowadays, with the development of autonomous driving technology, researchers have also shifted from research on passive safety technology to active safety technology. To realize the automation of the car, some advanced technical means must be used to complete part of the car driving task. Using deep learning method to intelligently detect road scene objects is the key to solve the vehicle active safety technology. The target detection network at this stage mainly uses the backbone network for feature extraction, but does not give too much consideration to the underlying multi-scale problems, which may lead to insufficient multi-scale target detection capabilities.

Contents of the invention

Purpose of the invention: To provide a complex road scene target detection method and device based on the LG-CenterNet model in view of the poor application effect of complex road scene target detection at the present stage, and conventional detection methods that cannot meet the detection requirements of the actual road environment.

Technical solution: The present invention proposes a complex road scene target detection method based on the LG-CenterNet model, which specifically includes the following steps:

(1) Process the images of complex road scenes, obtain road target images containing multiple categories, mark the categories and positions of the road targets in the images, construct complex road scene datasets and perform preprocessing;

(2) build target detection LG-CenterNet model, and above-mentioned road target data set is trained to obtain model S by LG-CenterNet model; Described LG-CenterNet model comprises Backbone module, hierarchical guidance attention module, Scales Encoder module, Deconvolution module, feature enhancement module and Centerpoints prediction module;

(3) Use the trained model S to perform target positioning, frame size division and category prediction in the form of heat maps for complex road targets through the Center points prediction module, and display the obtained results on videos or images and input corresponding effects .

Further, the preprocessing of the road scene data set described in step (1) is to normalize the image size of the image to 512×512 pixel size by normalizing the images of different pixels and complex road scenes, and then Uniformly distributed feature target samples are obtained through batch normalization, ReLU activation function, and maximum pooling operations.

Further, the implementation process of the step (2) is as follows:

(21) In the LG-CenterNet model, a new MresneIt50 is proposed as the Backbone module. MresneIt50 is composed of multiple residual blocks. The feature map extracted by the 4 residual modules is marked as E1, and the number of channels is 512; the 6 residual blocks are The feature map extracted from the difference block is marked as E2, and the number of channels is 1024; the feature map extracted from 3 channels is marked as E3, and the number of channels is 2048;

(22) Input the feature maps E1, E2, and E3 extracted by Backbone into the hierarchical guided attention module. Its main structure includes two branches: the global pooling branch and the hierarchical guided branch, and the feature map with 512 channels E1 is input to the global pooling branch, and EC1 is obtained through the operation of the global maximum pooling layer and the up-sampling layer; the feature maps E1, E2, and E3 with the number of channels of 512, 1024, and 2048 are input into the hierarchical guidance branch, through a series of The average pooling and convolution operations are combined with upsampling to obtain EC2; EC1 and EC2 are combined using add to obtain EC3, thereby reducing calculation parameters;

(23) Input the extracted EC3 to the Scales Encoder module, and perform a series of convolution and residual module operations to obtain EC4;

(24) Input the extracted EC4 to the deconvolution module. The deconvolution module is composed of 3 deconv groups. The size of the feature map is continuously enlarged through the convolution operation of each deconv group, and the number of channels is continuously reduced to obtain the scale. The feature map of 128×128×64 is marked as EC5;

(25) Input the feature map EC5 to the feature enhancement module for convolution operation to obtain a feature map EC6 with a scale size of 128×128×64. P-FEM consists of 3×3 Poly-Scale Convolution, batch normalization, ReLU activation function and Sigmoid The composition of the activation function is mainly to improve the correlation of local information in the feature map and enhance its ability to express features.

Further, the implementation process of the step (3) is as follows:

The Centerpoints prediction module classifies and predicts the input images through the trained model S, generates a heatmap image with the same scale as EC6 from the original image, and then calculates the loss value of the heat map separately as L _h , the loss of the target length and width The value is recorded as L _s and the loss value of the center point offset is recorded as L _f to determine the position and size of the target and generate the final heatmap for classification and positioning; the overall network loss is:

L _d = L _k +λ _s L _s +λ _f L _f

Where λ _s =0.1, λ _f =1; for an image with an input image size of 512×215, the feature map generated by the network is H×W×C, then the calculation formulas of L _k , L _s and L _f are respectively for:

Among them, A _HWC is the actual value of the target label in the image, A' _HWC is the predicted value of the image, α and β are 2 and 4 respectively, N is the number of key points in the image, s' _pk is the predicted size, s _k is the real size, and p is the center point position of the target in the image.

Based on the same inventive concept, the present invention also provides a complex road scene object detection device based on the LG-CenterNet model, comprising a memory, a processor and a computer program stored on the memory and operable on the processor, the computer program When loaded into the processor, the above-mentioned complex road scene target detection method based on the LG-CenterNet model is realized.

Beneficial effect: compared with prior art, the beneficial effect of the present invention: 1, by improving the backbone network of LG-CenterNet model, propose MresneIt50 to strengthen feature extraction effect; 3. A new Scales Encoder module and a feature enhancement module are proposed to focus on the extraction of local features to avoid the problem of feature loss in the deconvolution module; 4. The improved LG-CenterNet target detection Compared with the average precision mAP (meanAveragePrecision) of the original CenterNet framework, the model has improved by 5 percentage points; 5. The present invention also has higher detection precision in dealing with complex road scenes.

Description of drawings

Figure 1 is a flowchart of a complex road scene target detection method based on the LG-CenterNet model;

Fig. 2 is a schematic diagram based on the LG-CenterNet target detection model proposed by the present invention;

Fig. 3 is a structural schematic diagram of the residual block structure Mblock proposed by the present invention;

Figure 4 is a schematic diagram of the structure of the hierarchical guided attention model;

Figure 5 is a schematic diagram of the Scales Encoder module structure;

Fig. 6 is a schematic structural diagram of a feature enhancement module;

Figure 7 is a detection effect diagram obtained after using the LG-CenterNet target detection model.

Detailed ways

The present invention will be described in further detail below in conjunction with the accompanying drawings.

A large number of variables are involved in this embodiment, and each variable is described as follows. As shown in Table 1.

Table 1 variable description table

variable variable specification S 3×3 convolution kernel with 1024 channels E1 Feature map extracted from 4 residual blocks in the Backbone module E2 The feature map extracted from the 6 residual blocks in the Backbone module E3 Feature map extracted from 3 residual blocks in the Backbone module EC1 The feature map obtained by E1 through the global pooling branch EC2 Feature maps obtained by E1, E2, and E3 via hierarchical guidance branches EC3 The feature map of EC2 processed by the ScalesEncoder module EC4 The feature map of EC3 processed by the ScalesEncoder module EC5 The feature map of EC4 processed by the deconvolution module EC6 The feature map of EC5 processed by the feature enhancement module

The present invention provides a complex road scene target detection method based on the LG-CenterNet model. By collecting different target images of the road scene and marking them into complex road scene data sets, the proposed MresneIt50 is used as the backbone network for feature extraction. For the backbone network The feature maps of different scales extracted in are input into the hierarchical guidance attention module, and then multiple receptive field features are obtained through the Scales Encoder module, and then the feature pixels are restored by the deconvolution module, and the Poly-Scale Convolution (PSConv for short) is used. Build a feature enhancement module to improve the information relevance of local features. Finally, the center points prediction module is used to predict the position of the center point of the target, the scale size of the prediction frame and the offset of the center point, and at the same time identify the target category. As shown in Figure 1, it specifically includes the following steps:

Step 1: Process the image of the complex road scene, obtain the road target image containing multiple categories for preprocessing, and mark the category and position of the road target in the image to construct a complex road scene dataset.

The preprocessing of the road scene dataset is mainly by normalizing the images of different pixels and complex road scenes, normalizing the size of the image to 512×512 pixel size, and then through batch normalization (BatchNormalizaition), ReLU activation The function and the maximum pooling operation get the target samples in a relatively uniform distribution in the image.

Step 2. Build the target detection LG-CenterNet model. The structure of the LG-CenterNet model is shown in Figure 2, and the above-mentioned road target data set is trained through the LG-CenterNet model to obtain the model S. The LG-CenterNet network mainly includes the Backbone module , Levels guide attention module (Levels guide attention, LGA for short), Scales Encoder module, deconvolution module, feature enhancement module (P-Feature enhancement module, P-FEM) and Centerpoints prediction module.

(21) In the LG-CenterNet model, a new MresneIt50 is proposed as the Backbone module. MresneIt50 is composed of multiple residual blocks Mblock. The residual block structure Mblock is shown in Figure 3. The feature maps extracted from the four residual blocks are marked is E1, the number of channels is 512; the feature map extracted from 6 residual blocks is marked as E2, and the number of channels is 1024; the feature map extracted from 3 channels is marked as E3, and the number of channels is 2048.

(22) Input the feature maps E1, E2, and E3 extracted by Backbone into the Levelsguide attention module (Levelsguide attention, referred to as LGA). The structure of the LGA module is shown in Figure 4. Its main structure includes two branches: global The pooling branch and the hierarchical guidance branch input the feature map E1 with 512 channels into the global pooling branch, and obtain EC1 through the global maximum pooling layer and upsampling layer operation; the feature maps with 512, 1024, and 2048 channels E1, E2, and E3 are input to the hierarchical guidance branch, and EC2 is obtained through a series of average pooling and convolution operations with upsampling. Combine EC1 and EC2 with add to obtain EC3, thereby reducing calculation parameters.

(23) Input the extracted EC3 to the Scales Encoder module. The structure of the Scales Encoder module is shown in Figure 5, and EC4 is obtained after a series of convolution and residual module operations.

(24) Input the extracted EC4 to the deconvolution module. The deconvolution module is composed of 3 deconv groups. The size of the feature map is continuously enlarged through the convolution operation of each deconv group, and the number of channels is continuously reduced to obtain the scale. A feature map of 128×128×64 is marked as EC5.

(25) Input feature map EC5 to P-FEM for convolution operation to obtain feature map EC6 with a scale size of 128×128×64. P-FEM consists of 3×3 Poly-Scale Convolution (PSConv for short), Batch Normalization (BatchNormalization ), ReLU activation function and Sigmoid activation function, mainly to improve the correlation of local information in the feature map and enhance its ability to express features. The P-FEM structure is shown in Fig. 6.

Step 3: Use the trained model S to perform target positioning, frame size division, and category prediction in the form of heat maps through the Centerpoints prediction module for road scene targets, and display the obtained results on videos or images and input corresponding effects.

The Centerpoints prediction module classifies and predicts the input pictures through the trained model S, and generates a heatmap with the same scale as EC6 from the original image, and then calculates the loss value of the heat map separately as L _h , and the target length and width (size ) is recorded as L _s and the loss value of the center point offset (offset) is recorded as L _f to determine the position and size of the target and generate the final heatmap for classification and positioning. where the overall network loss is L _d .

L _d = L _k +λ _s L _s +λ _f L _f

Wherein λ _s =0.1, λ _f =1. For an image with an input image size of 512×215, the feature map generated by the network is H×W×C, then the calculation formulas of L _k , L _s and L _f are respectively:

Among them, A _HWC is the actual value of the target label in the image, A' _HWC is the predicted value of the image, α and β are 2 and 4 respectively, N is the number of key points in the image, s' _pk is the predicted size, s _k is the real size, and p is the center point position of the target in the image.

Based on the same inventive concept, the present invention also provides a complex road scene object detection device based on the LG-CenterNet model, comprising a memory, a processor and a computer program stored on the memory and operable on the processor, the computer program When loaded into the processor, the above-mentioned complex road scene target detection method based on the LG-CenterNet model is realized. As shown in Figure 7.

The self-built complex scene data set is trained through the LG-CenterNet network to obtain a model that can recognize complex scene objects, and the model performance is verified through the verification set in the data set, as shown in Figure 7. The present invention has an average recognition accuracy of 86.93% in the self-built complex road scene data set, and the detection speed of the road scene object image reaches 50 frames/s, which can meet the requirements of accurate detection and real-time detection of the road scene.

Among them, Precision is the precision, Recall is the recall rate, AP is the precision, mAP is the average precision, FPS is the number of frames, and t is the time to detect a single picture. There are many sample categories in the data set (such as car, person, etc.), n represents the number of samples, TP (True Positives) is the number of positive samples and is recognized as positive samples (that is, the total number of samples that are car are recognized as car) ; TN (True Negatives) is the total number of negative samples identified by the negative sample model; FP (FalsePositives) is the total number of positive samples identified by the negative sample model (that is, the total number of samples that are not car, and the model is identified as car); FN (FalsePositives) Negatives) is the total number of positive samples identified by the negative sample model.

The embodiments of the present invention have been described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments, and can also be made without departing from the gist of the present invention within the scope of knowledge possessed by those of ordinary skill in the art. Variations.

Claims (5)

1. A complex road scene target detection method based on an LG-CenterNet model is characterized by comprising the following steps:

(1) Processing images of complex road scenes to obtain road target images containing various categories, marking the categories and positions of road targets in the images, constructing a complex road scene data set and preprocessing the complex road scene data set;

(2) Constructing an LG-CenterNet model for target detection, and training the road target data set through the LG-CenterNet model to obtain a model S; the LG-CenterNet model comprises a Backbone module, a hierarchy attention-directing module, a Scales Encoder module, a deconvolution module, a feature enhancement module and a Centerpoints prediction module;

(3) And (3) performing target positioning, frame size division and category prediction on the complex road target in a thermodynamic diagram mode by using the trained model S through a Center points prediction module, and displaying and inputting the obtained result on a video or an image to obtain a corresponding effect.

2. The method for detecting the complex road scene target based on the LG-centrnet model as claimed in claim 1, wherein the preprocessing of the road scene data set in step (1) is to normalize the image of the complex road scene by pixel inconsistency, normalize the image size to 512 x 512 pixel size, and obtain the uniformly distributed feature target samples by batch normalization, reLU activation function and max pooling.

3. The LG-CenterNet model-based complex road scene target detection method according to claim 1, wherein the step (2) is realized by the following steps:

(21) A new Mresneit50 is proposed in the LG-CenterNet model to serve as a Backbone module, the Mresneit50 is composed of a plurality of residual blocks, a feature diagram extracted by 4 residual blocks is marked as E1, and the number of channels is 512; marking a feature map extracted by the 6 residual blocks as E2, wherein the number of channels is 1024; the feature map extracted by the number of 3 channels is marked as E3, and the number of the channels is 2048;

(22) The character diagrams E1, E2 and E3 extracted from the Backbone are input into a hierarchical attention-directing module, and the main structure of the hierarchical attention-directing module comprises two branches: the method comprises the steps of global pooling branching and level guiding branching, inputting a feature map E1 with the channel number being 512 into the global pooling branching, and obtaining EC1 through operation of a global maximum pooling layer and an upper sampling layer; inputting feature maps E1, E2 and E3 with the channel numbers of 512, 1024 and 2048 into a hierarchical guide branch, and obtaining EC2 through a series of average pooling and convolution operations and matching with upsampling; performing feature combination on EC1 and EC2 by using add to obtain EC3, thereby reducing calculation parameters;

(23) Inputting the extracted EC3 into a Scales Encoder module, and carrying out a series of convolution and residual module operations to obtain EC4;

(24) Inputting the extracted EC4 into a deconvolution module, wherein the deconvolution module consists of 3 deconv groups, the size of the characteristic graph is continuously enlarged through convolution operation of the deconv groups each time, and the number of channels is continuously reduced at the same time, so that the characteristic graph with the dimension of 128 multiplied by 64 is obtained and recorded as EC5;

(25) Inputting the feature map EC5 into a feature enhancement module to carry out Convolution operation to obtain a feature map EC6 with the dimension of 128 × 128 × 64, wherein the P-FEM is composed of 3 × 3 Poly-Scale conversion, batch standardization, reLU activation function and Sigmoid activation function, and is mainly used for improving the correlation of local information in the feature map and enhancing the expression capability of the feature map on the features.

4. The LG-CenterNet model-based complex road scene target detection method according to claim 1, characterized in that the step (3) is realized by the following steps:

the Centerpoints prediction module classifies and predicts the input pictures through the trained model S, generates a heatmap image with the scale consistent with the size of EC6 from the original image, and then records the loss value of the original image as L through respectively calculating the thermodynamic diagrams _h The loss value of the target length and width is expressed as L _s And the loss value of the center point offset is recorded as L _f To determine the position and size of the target and to generate a final sorted positioned heatmap; where the overall network loss is:

L _d ＝L _k +λ _s L _s +λ _f L _f

wherein λ _s ＝0.1，λ _f =1; for an image with an input picture size of 512 × 215, the feature map generated by the network is H × W × C, and L _k 、L _s And L _f The calculation formulas are respectively as follows:

wherein, A _HWC Is the true value, A ', of the target annotation in the image' _HWC Is a predicted value of the image, alpha and beta are respectively 2 and 4, N is the number of key points in the image, s' _pk To predict the size, s _k For true size, p is the location of the center point of the object in the image.

5. An LG-cenernet model-based complex road scene object detection device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the computer program when loaded into the processor implements the LG-cenernet model-based complex road scene object detection method according to any one of claims 1 to 4.

Images