CN110796028B - Unmanned aerial vehicle image small target detection method and system based on local adaptive geometric transformation - Google Patents

Abstract

The invention provides an unmanned aerial vehicle image small target detection method and system based on local adaptive geometric transformation, which comprises the following steps: 1) constructing a feature extractor; 2) extracting features; 3) locally stretching and amplifying the branches for processing; 4) obtaining a region needing to be transformed; 5) constructing a transformation function; 6) local transformation and target regression; 7) the result is inverse transformed. The method can adaptively amplify partial areas of the convolutional network characteristic diagram, and effectively detect the small-scale targets in the unmanned aerial vehicle image.

Description

Unmanned aerial vehicle image small target detection method and system based on local adaptive geometric transformation

Technical Field

The invention belongs to the technical field of automatic target identification, and particularly relates to an automatic identification method for a small unmanned aerial vehicle image target.

Background

With the development of the unmanned aerial vehicle technology, unmanned aerial vehicles equipped with cameras have been widely deployed and applied to various industries such as agriculture, traffic, aerial photography and the like. Because the unmanned aerial vehicle has the characteristic of flying in the air, the image formed by the unmanned aerial vehicle has the characteristics that the visual angle is taken down or taken obliquely, and the target size is smaller. There are currently great advances in computer vision target detection, but current computer vision algorithms and data sets are designed and evaluated for laboratory settings with human-centered photographs of close-range natural scene objects taken horizontally. For the unmanned aerial vehicle image shot vertically, the interested objects are relatively small and the features are few, and generally, in the process of processing data, a target detection method based on deep learning easily loses target detail information along with deepening of a neural network, so that the detection positioning is inaccurate or cannot be detected. The FPN (feature pyramid network) network uses multi-scale features including high-resolution feature maps to predict targets, and has a certain effect on small target detection, but the detection effect is not ideal when the target scale is smaller than the down-sampling step of the feature map with the highest resolution.

Disclosure of Invention

Aiming at the problems of the existing unmanned aerial vehicle image small target detection method, the invention provides the unmanned aerial vehicle image small target detection method based on the local adaptive geometric transformation based on the deep convolutional neural network method, and the method can effectively detect the small-scale target in the unmanned aerial vehicle image.

The technical scheme of the invention is an unmanned aerial vehicle image small target detection method based on local adaptive geometric transformation, which mainly comprises the following steps:

1) construct feature extractor

Constructing a feature extractor with a self-adaptive local stretching amplification branch based on a deep full-convolution neural network;

2) extracting features

Performing feature extraction on an input image by using the deep fully-convolutional neural network feature extractor with the self-adaptive local stretching amplification branch constructed in the step 1), wherein the mathematical description is X_n+1＝f_n(x_n) Wherein n belongs to {0,1,. k-1}, k is the number of layers of the deep full convolution neural network, f_nFor the nth layer of feature extraction operation, x_nIs passing through f_nCalculating the obtained characteristics;

3) local stretching amplifying branch for processing

For the shallow feature x extracted in the step 2)_mWherein m is<k, the down sampling step is S, the partial stretching amplification branch is used for processing, and x is obtained through the branch output result C_mWhether each position of the layer characteristic needs to be subjected to stretching transformation, wherein C_i,jWhether the characteristic point with the row position of i and the column position of j needs to be amplified or not is shown;

4) obtaining the required transformation area

Thresholding C using a threshold t, C_i,jIf the number is larger than t, marking the feature points to be amplified as a point set P, connecting the point set P into a plurality of areas by using a flooding filling algorithm, and calculating to obtain that the circumscribed rectangle of each area approximates the area; defining an amplification factor alpha, obtaining an amplified rectangle according to the obtained circumscribed rectangle and the amplification factor, if the two amplified circumscribed rectangles have an overlapped part, removing the circumscribed rectangle before amplification with a smaller area, and forming a region set A to be amplified by the filtered rectangle before amplification, wherein each element in the set is in x_mA rectangle on the feature layer coordinate system represents that the area needs to be enlarged;

5) constructing transformation functions

According to the collectionCalculating the feature map and original feature map x after local amplification of rectangular region by combining the size and amplification factor of rectangle in A_mMapping relation f and inverse transformation f of feature point coordinates^-1；

6) Local transformation and object regression

According to the mapping relation f to x_mThe feature layer is locally transformed to obtain a transformed feature map x_m2Then x is added_m2Continuously sending the convolution layer to obtain the position and the class information of the detection frame;

7) result inverse transformation

Calculating the detection box at x_m2Coordinate information on feature map, x_m2The position information on the feature map is inversely transformed by f^-1Is obtained at x_mAnd finally, calculating the detection result on the characteristic diagram to obtain the detection result under the input scale.

The invention provides an unmanned aerial vehicle image small target detection system based on local adaptive geometric transformation, which comprises the following modules:

the feature extractor construction module: the method is used for constructing a feature extractor with a self-adaptive local stretching amplification branch on the basis of a deep full convolution neural network;

a feature extraction module: the method is used for extracting the features of an input image by utilizing a deep full convolution neural network feature extractor with an adaptive local stretching amplification branch constructed in a feature extractor construction module, and the mathematical description is X_n+1＝f_n(x_n) Wherein n belongs to {0,1,. k-1}, k is the number of layers of the deep full convolution neural network, f_nFor the nth layer of feature extraction operation, x_nIs passing through f_nCalculating the obtained characteristics;

the local stretching and amplifying branch processing module comprises: extracting shallow layer feature x from feature extracting module_mWherein m is<k, the down sampling step is S, the partial stretching amplification branch is used for processing, and x is obtained through the branch output result C_mWhether each position of the layer characteristic needs to be subjected to stretching transformation, wherein C_i,jWhether the characteristic point with the row position of i and the column position of j needs to be amplified or not is shown;

a transformation area acquisition module: for thresholding C using a threshold t, C_i,jIf the number is larger than t, marking the feature points to be amplified as a point set P, connecting the point set P into a plurality of areas by using a flooding filling algorithm, and calculating to obtain that the circumscribed rectangle of each area approximates the area; defining an amplification factor alpha, obtaining an amplified rectangle according to the obtained circumscribed rectangle and the amplification factor, if the two amplified circumscribed rectangles have an overlapped part, removing the circumscribed rectangle before amplification with a smaller area, and forming a region set A to be amplified by the filtered rectangle before amplification, wherein each element in the set is in x_mA rectangle on the feature layer coordinate system represents that the area needs to be enlarged;

the transformation function building module: is used for calculating the feature map after the local amplification of the rectangular region and the original feature map x according to the size and the amplification factor of the rectangle in the set A_mMapping relation f and inverse transformation f of feature point coordinates^-1；

Local transformation and target regression module: according to the mapping relation f to x_mThe feature layer is locally transformed to obtain a transformed feature map x_m2Then x is added_m2Continuously sending the convolution layer to obtain the position and the class information of the detection frame;

a result inverse transformation module: for calculating the detection box at x_m2Coordinate information on feature map, x_m2The position information on the feature map is inversely transformed by f^-1Is obtained at x_mAnd finally, calculating the detection result on the characteristic diagram to obtain the detection result under the input scale.

Compared with the prior art, the invention has the following advantages and beneficial effects: on the premise of not greatly increasing the calculation amount of the detection network, the small-scale target detection performance of the network is increased.

Drawings

Fig. 1 is a flow chart of unmanned aerial vehicle image small target detection based on local adaptive geometric transformation.

Detailed Description

The technical solution of the present invention is further explained with reference to the drawings and the embodiments.

As shown in fig. 1, the present invention provides a method for detecting small targets in an image of an unmanned aerial vehicle based on local adaptive geometric transformation, which mainly comprises the following steps:

1) construct feature extractor

Constructing a feature extractor with an adaptive local stretching amplification branch based on a deep full convolution neural network, wherein, as shown in fig. 1, conv1, conv2, conv3, conv4 and conv5 are used as basic feature extraction networks, and the local stretching transformation branch input features are output results of a conv2 layer;

2) extracting features

Performing feature extraction on an input image by using the depth fully-convolutional neural network feature extractor with the adaptive local stretching amplification branch constructed in the step 1), wherein the mathematical description is X_n+1＝f_n(x_n) Wherein n belongs to {0,1,. k-1}, k is the number of layers of the deep full convolution neural network, f_nFor the n-th layer of feature extraction operation (the feature extraction mode is convolution, pooling and activation function), x_nIs passing through f_nAnd calculating the obtained characteristics.

3) Local stretching amplifying branch for processing

For the shallow feature x extracted in the step 2)_m，(m<k) The down sampling step length is S, the partial stretching amplifying branch is used for processing, and the result C is output through the branch to obtain x_mWhether each position of the layer characteristic needs to be subjected to stretching transformation, wherein C_i,jAnd whether the characteristic point with the row position of i and the column position of j needs to be amplified or not is shown.

3) Obtaining the required transformation area

Thresholding C using a threshold t, C_i,jIf the number is larger than t, marking the feature points to be amplified as a point set P, connecting the point set P into a plurality of areas by using a flooding filling algorithm, and calculating to obtain the circumscribed rectangle of each area to approximate the area. Defining amplification factor alpha, obtaining amplified rectangles according to the circumscribed rectangles obtained in the previous step and the amplification factor, if the two amplified circumscribed rectangles have an overlapped part, removing the circumscribed rectangles before amplification with smaller area, and forming a region set A and a region set A to be amplified by the filtered rectangles before amplificationEach element in the sum being in x_mA rectangle on the feature layer coordinate system represents that this area needs to be enlarged. The flooding filling algorithm is the prior art, and the invention is not described.

4) Constructing transformation functions

According to the size and the amplification factor of the rectangle in the set A, calculating a feature map after local amplification of the rectangular region and an original feature map x_mMapping relation f and inverse transformation f of feature point coordinates^-1。

5) Local transformation and object regression

According to the mapping relation f to x_mThe feature layer is locally transformed to obtain a transformed feature map x_m2Then x is added_m2And continuously feeding the subsequent convolution layer to obtain the position and the class information of the detection frame.

6) Result inverse transformation

According to the detection frame and x_m2The proportion relation of the characteristic diagram can obtain the result of the detection frame in x_m2Coordinate information on feature map, x_m2The position information on the feature map is inversely transformed by f^-1Is obtained at x_mAnd finally, calculating the detection result on the characteristic diagram to obtain the detection result under the input scale.

The utility model provides an unmanned aerial vehicle image small-object detection system based on local adaptive geometric transformation which characterized in that includes following module:

the feature extractor construction module: the method is used for constructing a feature extractor with a self-adaptive local stretching amplification branch on the basis of a deep full convolution neural network;

a feature extraction module: the method is used for extracting the features of an input image by utilizing a deep full convolution neural network feature extractor with an adaptive local stretching amplification branch constructed in a feature extractor construction module, and the mathematical description is X_n+1＝f_n(x_n) Wherein n belongs to {0,1,. k-1}, k is the number of layers of the deep full convolution neural network, f_nFor the nth layer of feature extraction operation, x_nIs passing through f_nCalculating the obtained characteristics;

the local stretching and amplifying branch processing module comprises: extracted for the extracted feature moduleShallow feature x_mWherein m is<k, the down sampling step is S, the partial stretching amplification branch is used for processing, and x is obtained through the branch output result C_mWhether each position of the layer characteristic needs to be subjected to stretching transformation, wherein C_i,jWhether the characteristic point with the row position of i and the column position of j needs to be amplified or not is shown;

a transformation area acquisition module: for thresholding C using a threshold t, C_i,jIf the number is larger than t, marking the feature points to be amplified as a point set P, connecting the point set P into a plurality of areas by using a flooding filling algorithm, and calculating to obtain that the circumscribed rectangle of each area approximates the area; defining an amplification factor alpha, obtaining an amplified rectangle according to the obtained circumscribed rectangle and the amplification factor, if the two amplified circumscribed rectangles have an overlapped part, removing the circumscribed rectangle before amplification with a smaller area, and forming a region set A to be amplified by the filtered rectangle before amplification, wherein each element in the set is in x_mA rectangle on the feature layer coordinate system represents that the area needs to be enlarged;

the transformation function building module: is used for calculating the feature map after the local amplification of the rectangular region and the original feature map x according to the size and the amplification factor of the rectangle in the set A_mMapping relation f and inverse transformation f of feature point coordinates^-1；

Local transformation and target regression module: according to the mapping relation f to x_mThe feature layer is locally transformed to obtain a transformed feature map x_m2Then x is added_m2Continuously sending the convolution layer to obtain the position and the class information of the detection frame;

a result inverse transformation module: for calculating the detection box at x_m2Coordinate information on feature map, x_m2The position information on the feature map is inversely transformed by f^-1Is obtained at x_mAnd finally, calculating the detection result on the characteristic diagram to obtain the detection result under the input scale.

The specific implementation manner and the steps of each module correspond, and the invention is not described.

The specific embodiments described herein are merely illustrative of the spirit of the invention. Various modifications or additions may be made to the described embodiments or alternatives may be employed by those skilled in the art without departing from the spirit or ambit of the invention as defined in the appended claims.

Claims (2)

1. An unmanned aerial vehicle image small target detection method based on local adaptive geometric transformation is characterized by comprising the following steps:

1) construct feature extractor

Constructing a deep full convolution neural network feature extractor based on the deep full convolution neural network;

2) extracting features

Performing feature extraction on an input image by using the deep full convolution neural network feature extractor constructed in the step 1), wherein the mathematical description is X_n+1＝f_n(x_n) Wherein n belongs to {0,1,. k-1}, k is the number of layers of the deep full convolution neural network, f_nFor the nth layer of feature extraction operation, x_nIs passing through f_nCalculating the obtained characteristics;

3) local stretching amplifying branch for processing

For the shallow feature x extracted in the step 2)_mWherein m is<k, the down sampling step is S, the partial stretching amplification branch is used for processing, and x is obtained through the branch output result C_mWhether each position of the layer characteristic needs to be subjected to stretching transformation, wherein C_i,jIs an element of C, C_i,jWhether the characteristic point with the row position of i and the column position of j needs to be amplified or not is shown;

4) obtaining the required transformation area

Thresholding C using a threshold t, C_i,jIf the number is larger than t, marking the feature points to be amplified as a point set P, connecting the point set P into a plurality of areas by using a flooding filling algorithm, and calculating to obtain that the circumscribed rectangle of each area approximates the area; defining amplification factor alpha, obtaining amplified rectangles according to the obtained circumscribed rectangles and the amplification factor, if two amplified circumscribed rectangles have overlapping parts, removing the circumscribed rectangle before amplification with smaller area, and filteringThe large front rectangle is formed by enlarging a region set A, and each element in the set is in x_mA rectangle on the feature layer coordinate system represents that the area needs to be enlarged;

5) constructing transformation functions

According to the size and the amplification factor of the rectangle in the set A, calculating a feature map after local amplification of the rectangular region and an original feature map x_mMapping relation f and inverse transformation f of feature point coordinates^-1；

6) Local transformation and object regression

According to the mapping relation f to x_mThe feature layer is locally transformed to obtain a transformed feature map x_m2Then x is added_m2Continuously sending the convolution layer to obtain the position and the class information of the detection frame;

7) result inverse transformation

Calculating the detection box at x_m2Coordinate information on feature map, x_m2The position information on the feature map is inversely transformed by f^-1Is obtained at x_mAnd finally, calculating the detection result on the characteristic diagram to obtain the detection result under the input scale.

2. The utility model provides an unmanned aerial vehicle image small-object detection system based on local adaptive geometric transformation which characterized in that includes following module:

the feature extractor construction module: the deep full convolution neural network feature extractor is constructed on the basis of the deep full convolution neural network;

a feature extraction module: the method is used for extracting the features of the input image by utilizing the deep full convolution neural network feature extractor constructed in the feature extractor construction module, and the mathematical description is X_n+1＝f_n(x_n) Wherein n belongs to {0,1,. k-1}, k is the number of layers of the deep full convolution neural network, f_nFor the nth layer of feature extraction operation, x_nIs passing through f_nCalculating the obtained characteristics;

the local stretching and amplifying branch processing module comprises: extracting shallow layer feature x from feature extracting module_mWherein m is<k, down-sampling step length of S, using local stretch amplification branchProcessing, by branching off the output C to obtain x_mWhether each position of the layer characteristic needs to be subjected to stretching transformation, wherein C_i,jIs an element of C, C_i,jWhether the characteristic point with the row position of i and the column position of j needs to be amplified or not is shown;

a transformation area acquisition module: for thresholding C using a threshold t, C_i,jIf the number is larger than t, marking the feature points to be amplified as a point set P, connecting the point set P into a plurality of areas by using a flooding filling algorithm, and calculating to obtain that the circumscribed rectangle of each area approximates the area; defining an amplification factor alpha, obtaining an amplified rectangle according to the obtained circumscribed rectangle and the amplification factor, if the two amplified circumscribed rectangles have an overlapped part, removing the circumscribed rectangle before amplification with a smaller area, and forming a region set A to be amplified by the filtered rectangle before amplification, wherein each element in the set is in x_mA rectangle on the feature layer coordinate system represents that the area needs to be enlarged;

the transformation function building module: is used for calculating the feature map after the local amplification of the rectangular region and the original feature map x according to the size and the amplification factor of the rectangle in the set A_mMapping relation f and inverse transformation f of feature point coordinates^-1；

Local transformation and target regression module: according to the mapping relation f to x_mThe feature layer is locally transformed to obtain a transformed feature map x_m2Then x is added_m2Continuously sending the convolution layer to obtain the position and the class information of the detection frame;

a result inverse transformation module: for calculating the detection box at x_m2Coordinate information on feature map, x_m2The position information on the feature map is inversely transformed by f^-1Is obtained at x_mAnd finally, calculating the detection result on the characteristic diagram to obtain the detection result under the input scale.

Images