CN112132207A

CN112132207A - Target detection neural network construction method based on multi-branch feature mapping

Info

Publication number: CN112132207A
Application number: CN202010988619.3A
Authority: CN
Inventors: 刘晋; 李怡瑶
Original assignee: Shanghai Maritime University
Current assignee: Shanghai Maritime University
Priority date: 2020-09-18
Filing date: 2020-09-18
Publication date: 2020-12-25

Abstract

The invention discloses a target detection method based on multi-branch feature mapping. The method divides the detected targets into three types according to the size of the detected targets in the image sample data set; inputting three branches of a network respectively, wherein each branch has the same network structure, different expansion rates and different effective prediction frame size ranges; according to the prediction frame range, limiting a scale range matched with the receptive field of each branch, and generating a plurality of prediction frames for each branch; in order to improve the screening precision of the prediction frames, the invention provides a scale perception training scheme, which effectively eliminates the prediction frames with inconsistent sizes, so that the network achieves the aim of light weight; and finally integrating the results of all the branches and outputting a network prediction result in the main branch. The target detection neural network based on the branch feature mapping is used for flexibly adjusting the network branches to enable prediction to be accurate when the size span of a detected target is large under a complex background, and particularly, the target detection neural network can still effectively detect very large or very small targets.

Description

Target detection neural network construction method based on multi-branch feature mapping

Technical Field

The invention belongs to the field of digital image processing and deep learning, and particularly relates to a method for identifying targets in a complex scene with a large target size change range, in particular to a method for constructing a target detection neural network based on multi-branch feature mapping.

Background

At present, the application of artificial intelligence technology in various fields such as industry, science and life is gradually wide. Object detection, as a pre-step of object recognition, plays a crucial role in applications such as robot monitoring, unmanned autonomous driving, etc. Since more than one object is often contained in a scene image and the background is complex, which means that the image contains a huge amount of data, a computer program is required to automate the task of extracting objects from the scene image.

Convolutional neural networks have gained widespread use in the field of target detection. Conventional convolutional neural network-based approaches can be broadly divided into two categories: a one-stage based approach, similar to YOLO or SSD, that directly uses convolutional neural networks to obtain the bounding box of interest; and two-phase based methods, similar to FasterR-CNN or R-FCN, that generate prediction frames and then extract features to generate bounding frames with further refinement. However, the core problem of both methods is how to handle the scale change. With the proposal of AlexNet, the error rate of classification tasks performed using deep learning neural networks has dropped from 15% to 2%, however, the average detection accuracy of the currently best performing detector on the COCO data set is only 62%. The reason for this gap is the hot problem in the field of target detection, i.e., the large span of the target scale. Depending on the actual scenario, the size of the object to be detected may vary over a large range, which may form a significant obstacle for detection, especially for detectors with very small or very large detection sizes.

To solve this multi-scale problem, an intuitive approach is to use a multi-scale image pyramid, which is popular in both manual feature-based approaches and current deep convolutional neural network-based approaches. Recent studies have shown that depth detectors can be well optimized if multi-scale training methods are applied. The basic idea of this training method is to avoid extreme-scale training objects. In contrast, SNIP proposes a scale normalization method, which generates an image pyramid for each image and selectively trains objects of appropriate size in each image scale. However, expanding a single image into various sizes is equivalent to doubling the data amount, which results in a huge amount of calculation, thereby increasing the time for object detection, and is not favorable for practical use. Another approach is to use a pyramid of features within the network instead of the image pyramid, thereby reducing the computational cost. Some researchers have attempted to construct a fast feature pyramid for target detection by interpolating the feature channels of nearby scale layers. Then, the SSD proposes a multi-scale feature mapping of different layers, and not only uses the feature map of the last layer for prediction, but also uses the feature maps of the last six layers for multiple predictions. Because the low-level features and the high-level features are fully utilized, the target detection effect is enhanced. On this basis, to make up for the lack of semantics in low-level features, the FPN further expands the top-down path and cross-joins to incorporate strong semantic information in high-level features. However, since the regional features of the objects of different scales are extracted from different levels of the FPN network, each level of the FPN network is generated from a different set of parameters. This makes the feature pyramid unable to achieve the same performance as the image pyramid with a reduced amount of computation. In summary, the problem of target detection with large target scale span still remains to be solved efficiently.

Disclosure of Invention

In order to solve the above problems, a method for constructing a target detection neural network based on multi-branch feature mapping is provided, which is a target detection neural network that can be flexible and changeable in a complex background, and particularly achieves good adaptability for very large or very small detection targets.

According to the invention, through a multi-branch network structure, the network can adapt to multi-scale target detection, different branches have the same network structure, and each branch is operated in parallel to execute convolution operation, pooling operation and the like, so that the operation time can be reduced, the characteristic information of different scales of the target can be extracted, meanwhile, the different receptive fields caused by expansion rate parameters are similar to the increase of the number of samples, and the over-fitting problem is reduced; each branch adopts a weight sharing mechanism, so that the problem of parameter redundancy in the network is effectively reduced, and the final detection effect can be optimized by improving the single-scale detection precision; by introducing the region set feature unit into each branch, the position and other information of the low-level feature graph and the classification and other information of the high-level feature graph are effectively combined to meet the position classification dual requirements of the detection task; by introducing a scale perception training scheme, the scale perception capability of each branch is improved, and training objects with extreme scales on unmatched branches are avoided; by introducing a non-maximum suppression algorithm (NMS), the optimal prediction box is retained within a defined number of frames.

The multi-branch feature mapping-based target detection neural network construction method provided by the invention has the advantages of multi-scale target detection, scale perception training strategy, combination of feature extraction of different grades, strong adaptability of model structure, light model weight and the like.

In order to achieve the purpose, the method for constructing the target detection neural network based on the multi-branch feature mapping is realized by the following technical scheme:

a construction method of a target detection neural network based on multi-branch feature mapping is characterized by comprising the following steps:

step 1: making an image data set COCO2017-plus, dividing the data set into a training set COCO-train and a testing set COCO-test, wherein the COCO2017-plus data set is constructed by adopting a data enhancement mode of random horizontal turning on the basis of a standard data set COCO 2017;

step 2: constructing a multi-branch characteristic mapping target detection neural network, wherein the network comprises a multi-branch network structure based on weight sharing, a region set characteristic unit combining characteristics of each level and a scale perception training module improving the network operation performance;

and step 3: defining a scale perception training scheme, and screening a plurality of prediction frames generated by each branch and removing the prediction frames outside the range by using the scale perception training scheme provided in the network;

and 4, step 4: preprocessing operations such as size unification and graying are carried out on data to be input;

and 5: inputting the preprocessed training set COCO2017-train into the multi-branch target detection neural network for training, and removing the unqualified prediction boxes by using a well-defined scale perception training scheme;

step 6: the optimal number of prediction blocks obtained on each branch is fed to the NMS module. After NMS processing, the optimal 500 prediction blocks are retained.

Further, the multi-branch feature mapping target detection neural network building method in step 2 is as follows:

firstly, the multi-branch feature mapping network architecture is composed of a plurality of branch blocks, the example defines the multi-branch feature mapping network architecture into three branches, and the number of the branches can be defined according to the variation range of specific target scales in actual operation. The three branches have the same basic backbone network structure based on ResNet-50, each branch block operates in parallel but has different expansion rates, the expansion rates d of the three branches₁,d₂,d ₃1, 2, 3, respectively, a 1 x 1 convolution operation is required before entering each branch. Each branch block comprises a plurality of parallel region set characteristic units, the expansion rate of the same branch is the same, and each unit consists of three parts: BN (BatchNormalization) layer, ReLu (RectisedLinearunits) layer and a 3X 3 convolutional layer. Adding a scale perception training scheme to the tail part of each branch, leaving off unqualified prediction frames, not participating in back propagation, and finally, performing NMS (network management system) conversion on the three-branch results to keep optimal 500 prediction frames, and uniformly outputting the prediction frames in a main network;

further, the scale-aware training scheme in step 3 is defined as follows:

defining valid prediction box ranges for each branch, [ l ]_i,u_i]Let RoI (Region-of-Interest) on the input image be w in width and h in height, and its effective range should be:

in actual operation, the effective prediction range of each branch image can be defined according to the variation range of the target scale. And screening a plurality of prediction frames obtained by each branch by using the effective prediction range defined by the scale perception training scheme, wherein frames which are not in the range are omitted and do not participate in back propagation.

Further, the specific training strategy in step 5 is as follows:

inputting the preprocessed training set COCO2017-train into the multi-branch target detection neural network, wherein training is divided into 12 rounds in total, the input initialization learning rate of the network is defined to be 0.02, the initial learning rate is reduced by 0.1 time after the 8 th round and the 9 th round, the network is converged to the accuracy rate to meet the requirement, the trained model is stored, and the number of the whole training round can be adjusted to be two times or three times according to the number of branches of the actual network.

Drawings

FIG. 1 is a flow chart of an implementation of the neural network construction method for target detection based on multi-branch feature mapping according to the present invention

FIG. 2 is a network architecture diagram of the multi-branch eigen-mapping-based target detection neural network construction method of the present invention

FIG. 3 is a convolution operation of the region set feature unit based on the multi-branch feature mapping target detection neural network construction method of the present invention

FIG. 4 is the internal structure of the region set feature block of the multi-branch feature mapping-based target detection neural network construction method of the present invention

FIG. 5 is an application of the multi-scale training scheme of the multi-branch feature mapping based target detection neural network construction method of the present invention

FIG. 6 is a detection effect diagram of the multi-branch feature mapping-based target detection neural network construction method of the present invention

Detailed Description

The invention is described in further detail below with reference to the figures and the detailed description. It should not be understood that the scope of the above-described subject matter is limited to the following examples, and that any technique realized based on the teachings of the present invention is within the scope of the present invention.

The overall implementation flow of the method for constructing the target detection neural network based on the multi-branch feature mapping is shown in fig. 1, and is specifically described as follows:

the data set in the embodiment is a new image data set COCO2017-plus constructed in a data enhancement mode of random horizontal inversion on the basis of COCO2017, and is divided into a training set COCO-train and a test set COCO-test. And the image size is unified to 800 pixels. The COCO dataset is a large, rich object detection, segmentation and caption dataset. The data set is mainly intercepted from complex daily scenes by taking scenes as targets, and the targets in the images are calibrated through accurate segmentation. The image included 91 types of objects, 328,000 images and 2,500,000 labels.

In this embodiment, the implementation platform of the method for constructing the target detection neural network based on the multi-branch feature mapping is a computer, the operating system is Windows 10, the deep learning architecture is Pytorch and Detectron2, the graphics processing library uses opencv 4.1.0, and the image acceleration unit uses GeForce GTX 1060 GPU.

The overall network architecture of the multi-branch feature mapping-based target detection neural network construction method provided by the invention is shown in fig. 2, and in the example, an input image G is subjected to_inFormalization is defined as follows:

performing graying processing on an input image, setting the gray value of the original image as f (m, n) E [ a, b ], changing the gray value into g (m, n) E [ c, d ] after graying, and operating as follows:

g(m,n)＝c+k[f(m,n)-a]。

the processed image is input to the network, in this example assuming that the output of the l-th layer neurons in the neural network is represented as y^l. For the ith neuron in the l +1 layer neural network, the method

Express their corresponding weights by

Representing its corresponding offset, the convolution operation is defined as:

the expansion rates of the branches in the multi-branch network are different, and in this example, the expansion rates of the 3 branches are set as d₁＝1,d₂＝2,d₃3. The dilation convolution is to fill 0 in a convolution kernel, and assuming that the original convolution kernel has a size of K, the dilated convolution kernel has a size of K, and the dilation rate is d, the size of the equivalent convolution kernel of the dilation convolution can be calculated by the following formula:

K＝k+(k+1)*(d-1)。

the feature map size output after convolution is:

wherein W_inIndicating the size of the input image, W_outRepresenting the output image size, padding represents the number of 0's, tide represents the step size, and F represents the size of the convolution kernel. The expansion rate is 1, which is a normal convolution, and if the expansion rate is greater than 1, which means that the expansion convolution is performed.

The region set feature unit is adopted among the branches to extract features, the convolution operation with the region set feature unit is shown in fig. 3, and the internal structure of each region set feature block is shown in fig. 4. The image extracted region features are defined as a one-dimensional vector:

r＝[r₁,r₂,...,r_n]。

the region set feature extraction network establishes connection between each layer and all the following layers, and the extraction operation is as follows:

x_l＝H_l([x₀,x₁,...,x_l-1])。

wherein H_l([x₀,x₁,...,x_l-1]) Representing the collection of signatures obtained from each layer, from layer 0 to layer l-1. The output image after neural network processing is defined as:

because each branch adopts different expansion rates, but each branch shares the same data set, in order to avoid the degradation of network performance caused by the condition of size mismatching, a training scheme of scale perception is utilized to set different effective ranges for each branch, thereby improving the perception capability of each size. The application of the multi-scale training scheme in the network is shown in fig. 5. The multi-scale branch number is i, and the expansion ratio is d₁,d₂...d_i. The range of each branch is defined as [ l ]_i,u_i]And satisfy

w is the width of the image and h is the height of the image. The effective prediction range for the 3 branches in this example is [0,90 ]]，[30,160]，[90，∞]And each branch limits a scale range matched with the receptive field of the branch, and for a plurality of prediction frames generated in the training process of each branch, the prediction frames outside the range are screened and removed, so that the backward propagation is not involved, and the calculated amount is reduced, so that the network is light.

In this example, the structure of the multi-branch unit is shown in table 1, where MB _ Layer represents the multi-branch unit. Where MBconv consists of 3 dilated convolutions. In the example, a total of 12 rounds of training are carried out, the learning rate is initialized to 0.02 and is reduced by 0.1 times after 8 th round and 9 th round, and the whole training discussion can be adjusted to be two times or three times according to the number of branches of the network in practical operation. In the training process, the optimal 1200 propofol values obtained on each branch are sent to the NMS module. After NMS processing, the optimal 500 propofol are reserved, and finally, a main branch is used for representing the detection result of the network.

TABLE 1 network Multi-Branch Unit architecture

The target detection result obtained by prediction of the present invention is shown in fig. 6, where a, B, and C represent input pictures, and D, E, and F represent targets recognized through the present network and corresponding block diagrams. Experiments prove that the multi-branch feature mapping-based target detection neural network provided by the invention has higher size perception capability on each branch, effectively adapts to targets with different sizes, and is more accurate in detection.

It will be appreciated by those of ordinary skill in the art that the foregoing description provides numerous implementation details. Of course, embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Claims

1. A target detection neural network construction method based on multi-branch feature mapping is characterized by comprising the following steps:

step 2: constructing a multi-branch characteristic mapping target detection neural network, wherein the network comprises a multi-branch network structure based on weight sharing, and a scale perception training module for combining region set characteristic units of each level of characteristics and improving the network operation performance, and the network construction steps are as follows:

step 2.1: the multi-branch feature mapping network architecture is composed of a plurality of branch networks, generally defined as three branches, and the number of the branches can be defined according to the variation range of specific target scales in actual operation;

step 2.2: the three branches have the same basic backbone network structure based on ResNet-50, and the branch networks operate in parallel but have different expansion rates, the expansion rates d of the three branches₁,d₂,d₃1, 2 and 3 respectively, before entering each branchThere is a 1 x 1 convolution operation;

step 2.3: each branch network comprises a plurality of parallel regional set characteristic units, the expansion rate of the same branch is the same, and each unit consists of three parts: BN layer, ReLu layer and a 3 x 3 convolutional layer;

step 2.4: adding a scale perception training scheme to the tail part of each branch;

step 2.5: finally, the three-branch results are transformed and unified in the main network through NMS and output;

and step 3: defining a scale perception training scheme, screening a plurality of prediction frames generated by each branch by using the scale perception training scheme, and removing the prediction frames outside the range:

step 3.1: defining a prediction box range for each branch l_i,u_i]Let RoI on the input image be w wide and h high, its valid range should be:

step 3.2: the effective prediction ranges of the three branches are set to l₁,u₁],[l₂,u₂],[l₃,u₃]In actual operation, the effective prediction range of each branch image can be defined according to the variation range of the target scale;

step 3.3: screening a plurality of prediction frames obtained by each branch by using a scale perception training scheme, removing the prediction frames outside the range and not participating in backward propagation;

and 4, step 4: preprocessing data to be input, adjusting all short edges of an image to 800 pixels, and performing graying processing on the image;

and 5: inputting the preprocessed training set COCO2017-train into the multi-branch target detection neural network, wherein training is totally divided into 12 rounds, the network defines the input initialization learning rate to be 0.02, the input initialization learning rate is reduced by 0.1 time after the 8 th round and the 9 th round, and finally the network is converged, and the trained model is stored, and the number of the whole training round can be adjusted to be two times or three times according to the number of branches of the actual network;

step 6: and sending a certain number of optimal prediction boxes obtained on each branch to an NMS module, and reserving 500 optimal prediction boxes after NMS processing.