CN111753677A - Multi-angle remote sensing ship image target detection method based on characteristic pyramid structure - Google Patents

Multi-angle remote sensing ship image target detection method based on characteristic pyramid structure Download PDF

Info

Publication number
CN111753677A
CN111753677A CN202010521967.XA CN202010521967A CN111753677A CN 111753677 A CN111753677 A CN 111753677A CN 202010521967 A CN202010521967 A CN 202010521967A CN 111753677 A CN111753677 A CN 111753677A
Authority
CN
China
Prior art keywords
ship
remote sensing
network
sample
layers
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010521967.XA
Other languages
Chinese (zh)
Other versions
CN111753677B (en
Inventor
李训根
李子璇
潘勉
吕帅帅
马琪
张战
门飞飞
刘爱林
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Dianzi University
Original Assignee
Hangzhou Dianzi University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Dianzi University filed Critical Hangzhou Dianzi University
Priority to CN202010521967.XA priority Critical patent/CN111753677B/en
Publication of CN111753677A publication Critical patent/CN111753677A/en
Application granted granted Critical
Publication of CN111753677B publication Critical patent/CN111753677B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/13—Satellite images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection

Abstract

The invention discloses a multi-angle remote sensing ship image target detection method based on a characteristic pyramid structure, which comprises the following steps: s1, collecting the remote sensing satellite ship image data set, and carrying out sample annotation to obtain an annotation target; s2, preprocessing the sample in the data set extracted by the S1 to form a complete training data set; s3, extracting the features of the preprocessed sample by using the improved feature pyramid network to obtain a feature pyramid with multi-layer feature fusion; s4, generating a candidate area through an RPN; s5, adding ROI Pooling layers with different Pooling sizes; s6, building a Fast R-CNN network; s7, carrying out step S2 preprocessing operation of training phase on the test data collected in S1; s8, the sample processed by S7 is sent to the model constructed by S3, S4, S5 and S6 to be tested to obtain the result, namely the result of classification and regression is finally output by Fast R-CNN.

Description

Multi-angle remote sensing ship image target detection method based on characteristic pyramid structure
Technical Field
The invention relates to the technical field of remote sensing picture target detection, in particular to a multi-angle remote sensing ship image target detection method based on a characteristic pyramid structure.
Background
In recent years, with the overall development of the world economic trade, in addition to the transportation modes such as land transportation and air transportation, the transportation in the water areas such as oceans and lakes has been more and more emphasized and developed due to the unique advantages of large cargo capacity, low cost and the like. In addition, the national supervision and control of sea areas and the protection of ocean rights and interests are also gaining more attention, so that the research and development of ship monitoring are of great significance in the fields of military use, commercial use and civil use, national defense construction, port management, cargo transportation, marine rescue and the attack of illegal ships.
At present, related departments also focus on traditional short-distance monitoring on marine vessel monitoring, the existing traffic management system has some inevitable limitations, and more students begin to apply emerging technologies to research on vessel dynamic monitoring means. With the development of science and technology, the number of global emission satellites increases, and the high-resolution remote sensing image target detection technology is also used as a new technology for ship target detection. The ship satellite image has large area covering the water area and wide time range, and the resolution definition of the image is greatly improved.
Traditional remote sensing ship target detection focuses on gray level statistics, threshold segmentation, edge detection and the like. However, the methods are only suitable for simple and calm seas, and the application scene is single. At the present stage, remote sensing image ship detection has a plurality of models and algorithms. Such as bag-of-words models, sparse representations, feature extraction, etc. However, these methods are computationally intensive, and often miss some small ships, and cannot sufficiently extract high-level semantic features in high-resolution images.
Currently, deep learning techniques have achieved significant performance in target detection. By extracting ship features with higher semantics through a deep neural network, more useful information can be obtained from remote sensing images, and the ship identification accuracy is gradually improved. But there are several characteristics due to the remote sensing of the ship target: (1) the size is diversified, and the length and the width of the large ship are different from those of the small ship by times; (2) the density is that the port ships are parked compactly; (3) the redundancy of the detection area, when the traditional horizontal bounding box is used for a ship with a large length-width ratio, a plurality of redundant pixels which do not belong to the ship target actually can be brought; (4) a complex background. The difficulty of remote sensing ship target detection is increased.
Disclosure of Invention
In view of the technical problems, the invention is used for providing a multi-angle remote sensing ship image target detection framework based on a multi-scale depth neural structure, and the method firstly preprocesses a data set, so that the generalization capability of a model is improved; then, extracting features by improved feature pyramid network extraction, maximizing information flow among all layers of the network and obtaining optimal ship features; and then, adding two Pooling layers with Pooling sizes into an ROI Pooling layer module in a Fast R-CNN network to obtain a more accurate characteristic diagram containing the characteristics of the remote sensing ship, and finally classifying the categories of the targets in the candidate areas and regressing the coordinates of the targets.
In order to solve the technical problems, the invention adopts the following technical scheme:
a multi-angle remote sensing ship image target detection method based on a characteristic pyramid structure comprises the following steps:
s1, collecting the remote sensing satellite ship image data set, and carrying out sample annotation to obtain an annotation target;
s2, preprocessing the sample in the data set extracted by the S1 to form a complete training data set;
s3, extracting the features of the preprocessed sample by using the improved feature pyramid network to obtain a feature pyramid with multi-layer feature fusion;
s4, generating a candidate area through an RPN;
s5, adding ROI Pooling layers with different Pooling sizes;
s6, building a Fast R-CNN network;
s7, performing data clipping operation of step S2 preprocessing of the training phase on the test data acquired in S1;
s8, the sample processed by S7 is sent to the model constructed by S3, S4, S5 and S6 to be tested to obtain the result, namely the result of classification and regression is finally output by Fast R-CNN.
Preferably, the step S1 further includes:
s101, collecting remote sensing satellite pictures of ports, open sea, wharfs, lakes and other places containing ship targets from a Google Earth satellite map to serve as training images;
and S102, marking the category and the position coordinates of all ship targets in the training image to obtain marked targets.
Preferably, the step S2 further includes:
s201, cutting a training image into pictures of 1000 multiplied by 1000, wherein the overlapping rate is 20%;
s202, removing negative samples in the training set after cutting;
and S203, horizontally turning and rotating the picture to form a complete training data set.
Preferably, the step S3 further includes:
firstly, selecting a ResNet-101 residual network to extract the characteristics of an input remote sensing satellite image, selecting the characteristic diagram of the last layer of the last 4 residual modules to construct a network from bottom to top, obtaining a characteristic diagram with higher resolution by transverse connection and dense connection, and constructing the network from top to bottom, wherein the dense connection means that the input of each layer of the characteristic diagram not only comprises the output of down sampling of adjacent characteristic diagrams, but also comprises the output of down sampling of other upper layer characteristic diagrams, and all the inputs are combined by series connection; the input of the horizontal connection refers to the output of the 1 × 1 convolution of the corresponding feature graph in the bottom-up network, and finally the 3 layers of the lowest layer of the top-down network are used as the output of the feature extraction network, so that the process of obtaining the 3 layers of feature layers is expressed as the following form:
Figure BDA0002532456150000031
wherein, CkK-th layer characteristic diagram, P, representing a bottom-up networkkRepresents the feature map of the k-th layer after fusion (k decreases from top to bottom), f1×1And f3×3Represents convolutional layers of 1 × 1 and 3 × 3, Up represents upsampling, 2iAnd representing the size of upsampling, wherein the value of k + i is not more than 5, and through the process, the characteristic pyramid of multi-layer characteristic fusion is obtained.
Preferably, the step S4 further includes:
and respectively building regional generation networks on the 3 layers of feature maps, connecting two 1 × 1 convolutional layers through a 3 × 3 convolutional layer to perform classification prediction of foreground and background and coordinate regression of a boundary frame, wherein the regional generation networks are full convolutional networks, and the parameters of the regional generation networks of the 3 feature maps are independent from each other and do not share the parameters.
Preferably, the step S5 further includes:
for the candidate region output in the previous step, two ROI Pooling layers were added, the Pooling sizes of the added two ROI Pooling layers were 10 × 3 and 3 × 10, the total number of ROI Pooling layers was 3 layers, and the sizes were 7 × 7, 10 × 3, and 3 × 10, respectively. And (3) respectively passing each candidate region through 3 ROI Pooling layers, and finally outputting to obtain feature maps with fixed sizes of 7 × 7 × ConvDepth, 10 × 3 × ConvDepth and 3 × 10 × ConvDepth.
Preferably, the step S6 further includes:
s601, flattening the feature map of each candidate region into a one-dimensional vector, and classifying the candidate regions through a ReLU function, finally respectively passing through two full-connection layers and Softmax; the other performs a more accurate regression of the detection frame coordinates;
s602, the loss function expression of the remote sensing ship target detection network is as follows:
Figure BDA0002532456150000041
wherein liLabels representing objects, piRepresenting the probability distribution, t, of the various classes calculated by the Softmax functioniRepresents the five coordinate vectors of the prediction,
Figure BDA0002532456150000042
the coordinates of a truth detection box corresponding to the prediction area are shown, N represents the number of categories, and only one type of data of the ship is shown in the method, namely NclsThe hyperparameter λ controls the balance between the two losses, the method using λ 1, and in addition the function LclsAnd LregIs defined as:
Lcls(p,l)＝-logpl
Figure BDA0002532456150000043
Figure BDA0002532456150000044
s603, initializing all weights and offsets to be trained in the model, setting training parameters including learning rate, batch _ size, threshold values of positive and negative samples of RPN and Fast R-CNN networks, and starting model training.
Preferably, the step S8 further includes:
and calculating the Recall rate (Recall), Precision rate (Precision), average Precision (Ap) and F-measure of the remote sensing ship target.
S801, the recall rate of the remote sensing ship test sample can be calculated as follows:
Figure BDA0002532456150000051
wherein tp (true positions) is that the positive sample is correctly identified as the positive sample, the picture of the ship is correctly identified as the ship, and P is all true value samples;
s802, the precision rate of the remote sensing ship test sample can be calculated as follows:
Figure BDA0002532456150000052
wherein, tp (true positions) is that the positive sample is correctly identified as the positive sample, and the picture of the ship is correctly identified as the ship; fp (false positives) is a false positive sample, i.e. a negative sample is misidentified as a positive sample, and a picture of a ship is misidentified as other categories;
s803, the average accuracy is calculated from each category under various IoU thresholds (0.5, 0.55.., 0.95), the results are first ranked by confidence, and the area under the interpolated accuracy-recall curve is estimated by the average interpolation accuracy of the recall on 11 equidistant horizontal axes, where the average accuracy can be calculated as:
Figure BDA0002532456150000053
where r is the recall, c is the number of categories given, c is the ship category in the method, t is the threshold value of IoU, and in addition, pinterp(r) can be calculated as:
Figure BDA0002532456150000054
the curve is monotonically decreasing by reassigning the accuracy of each recall r to the maximum accuracy of higher recalls;
s804, calculating the F-measure of the remote sensing ship test sample as follows:
Figure BDA0002532456150000061
precision and recall are the Precision and recall explained above.
The invention has the following beneficial effects:
(1) the implementation of the method is different from the prior model established based on the characteristic pyramid structure, the characteristic pyramid is an effective multi-scale method for fusing multi-level information, the remote sensing ship detection is a task which needs to be considered for both large object detection and small object detection, and meanwhile, the complex diversity of the background in the remote sensing ship image is also considered. Therefore, on the basis of the original characteristic pyramid network, cross-layer connection is added to the lower layer in the top-down network to connect the front layer and the rear layer in the network; with dense concatenation, the input of the lower layer is not just adjacent upper layer down-sampling, but includes cross-layer down-sampling. The features extracted in the way not only enhance the feature propagation, but also encourage the feature reuse, and the extracted feature map maximizes the information flow among all layers in the network and obtains the optimal ship features.
(2) The method adds two ROI Pooling layers with different length-width ratios in the ROI Pooling layer to capture more remote sensing ship features. Based on the size of the target object in the natural scene, the traditional feature map with the same length and width of the pooling generated finally by each candidate area is of fixed size. However, by counting the aspect ratio in the remote sensing ship picture, the feature maps with the same length and width are not suitable for remote sensing ship detection, and finally, other feature maps with the same aspect ratio are generated for further detection in consideration of the ship target with the large length-width ratio. The pooled feature map thus obtained more accurately includes the features of the remote sensing ship, and the feature map can be subjected to more accurate regression adjustment in the next stage of processing.
Drawings
FIG. 1 is a flow chart of steps of a multi-angle remote sensing ship image target detection method based on a characteristic pyramid structure according to an embodiment of the invention;
FIG. 2 is a block diagram of a feature extraction network according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of different Pooling size profiles generated by ROI Pooling layers according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, the invention discloses a multi-angle remote sensing ship image target detection method based on a characteristic pyramid structure, which comprises the following specific implementation steps:
s1, the remote sensing satellite ship images collected on the Google Earth satellite map are publicly collected, ship pictures including scenes of famous ports, civil docks, military bases, near-coast areas, open seas, lakes and the like are selected in the global range in order to reflect diversity and richness of remote sensing ship data set scenes, and sample labeling is carried out on the category and the coordinate of each ship target.
S2, the data set picture of the large scene is cut into 1000 × 1000 pixel pictures with an overlap ratio of 20%, so as to improve the difficulty of reading the large picture by the model. And (4) processing the cut picture by using a negative sample, and deleting the picture without the target ship. In order to improve the generalization capability of the model, data amplification is carried out on the data set by horizontal turning and rotation. And in the process of selecting the training set and the test set, ensuring that the selected training set sample and the selected test set sample cover all remote sensing ship scenes. The ratio of the number of samples in the training set to the number of samples in the test set was 7: 3.
And S3, performing multi-scale feature extraction on the sample. From the characteristics of each layer extracted from the convolutional neural network, the semantic information is richer as the hierarchy is higher, but the position information is less, so that the method is suitable for large target detection and complex target distinguishing; and the low-level characteristic position information is rich, so that the method is more beneficial to target positioning and small target detection. And aiming at the large and small targets coexisting in the remote sensing ship, combining the high and low-level features and carrying out target detection on the feature maps of a plurality of levels of the improved feature pyramid structure. And finally, the output of each layer is independent, and the generated feature graph can generate more information. The characteristic that such characteristic extraction network has smooth characteristic propagation and characteristic to reuse, has greatly improved the detection performance.
S4, generating a candidate area through an RPN;
s5, adding ROI Pooling layers with different Pooling sizes; and for the candidate region, extracting a feature map with a fixed size for each region by using the ROI Pooling layer through reusing the convolution feature map, and finally obtaining a feature map of 7 multiplied by ConvDepth for each region based on the size of a target object in a natural scene. However, feature maps with the same length and width are not very suitable for ship targets, and in consideration of the large length-width ratio of the ship targets, two different pooling sizes are added to capture more remote sensing ship features, so that detection of horizontal samples with the width far larger than the height of the horizontal samples and vertical samples with the height far larger than the width of the vertical samples is facilitated. The pooled features are connected together for further detection, and such a feature map more accurately contains the features of the remotely sensed vessel.
S6, building a Fast R-CNN network; and finally, outputting the classification result passing through Softmax and the regression coordinate parameter result.
S7, performing data clipping operation of step S2 preprocessing of the training phase on the test data acquired in S1; the data tested also required clipping to 1000 x 1000 pixel size to be fed into the network.
S8, the sample processed by S7 is sent to the model constructed by S3, S4, S5 and S6 to be tested to obtain the result, namely the result of classification and regression is finally output by Fast R-CNN.
Through the 8 steps, the multi-angle remote sensing ship image target detection model based on the characteristic pyramid structure can be obtained.
In a specific application example, the step S3 further includes:
firstly, a ResNet-101 residual error network is selected to extract the characteristics of an input remote sensing satellite image, and the characteristic diagram of the last layer of the last 4 residual error modules is selected to construct a network from bottom to top. Obtaining a feature map with higher resolution by transverse connection and dense connection, and constructing a network from top to bottom, wherein the dense connection means that the input of each layer of feature map not only comprises the output of downsampling of adjacent feature maps, but also comprises the output of downsampling of other upper-layer feature maps, and all the inputs are combined by series connection; the horizontally connected input is the output after 1 × 1 convolution with the corresponding feature map in the bottom-up network. Finally, the output of the network is extracted by taking the lowest 3 layers of the top-down network as features. The process of obtaining 3 layers of feature layers can be expressed as follows:
Figure BDA0002532456150000081
wherein, CkK-th layer characteristic diagram, P, representing a bottom-up networkkRepresents the feature map of the k-th layer after fusion (k decreases from top to bottom), f1×1And f3×3Represents convolutional layers of 1 × 1 and 3 × 3, Up represents upsampling, 2iRepresenting the size of the upsampling, the value of k + i does not exceed 5. Through this process, we can obtain a feature pyramid of multi-layer feature fusion, as shown in fig. 2.
The subsequent prediction does not share the classification and regression parameters among each layer, and the output of each layer is independent, so that the generated feature map can generate more multi-scale information. We maximize the information flow between all layers in the network and obtain the optimal ship characteristics.
In a specific application example, the step S4 further includes:
respectively building a region generation network on a 3-layer characteristic diagram, connecting two 1 × 1 convolutional layers through a 3 × 3 convolutional layer to perform classification prediction of foreground and background and coordinate regression of a boundary frame, and aiming at each anchor point, two groups of different outputs are provided: (1) whether a certain target exists, namely a predicted value of a background and a foreground is output; (2) the bounding box regression outputs 4 predicted values. The area generation network is a full convolution network, and the parameters of the area generation networks of the 3 characteristic graphs are mutually independent and do not share the parameters.
In a specific application example, the step S5 further includes:
for the candidate region output in the previous step, two ROI Pooling layers are added, and two ROI Pooling layers with Pooling sizes of 10 × 3 and 3 × 10 are added. The total number of ROI Pooling layers was 3 layers with sizes of 7 × 7, 10 × 3 and 3 × 10, respectively. A 3 x 10 pooling size can capture more horizontal features and facilitate detection of vessel targets that are much wider than they are tall; a pooling size of 10 x 3 can capture more vertical features and is useful for vertical ship targets when the height is greater than the width.
Each candidate region is passed through 3 ROI Pooling layers, and finally output to obtain a feature map with fixed sizes of 7 × 7 × ConvDepth, 10 × 3 × ConvDepth, and 3 × 10 × ConvDepth, as shown in fig. 3. The ROI Pooling layer cascades the three layers of results for further classification and regression.
In a specific application example, the step S6 further includes:
s601, flattening the feature map of each candidate region into a one-dimensional vector, and classifying the candidate regions through a ReLU function, finally respectively passing through two full-connection layers and Softmax; and the other performs a more accurate regression of the detection box coordinates.
S602, the loss function expression of the remote sensing ship target detection network is as follows:
Figure BDA0002532456150000091
wherein liLabels representing objects, piRepresenting the probability distribution, t, of the various classes calculated by the Softmax functioniRepresents the five coordinate vectors of the prediction,
Figure BDA0002532456150000101
coordinates of a truth detection box corresponding to the prediction area are indicated. N represents the number of categories, and the method only has ship class data, namely NclsThe hyperparameter λ controls the balance between the two losses, which is 1 for both methods. In addition, a function LclsAnd LregIs defined as:
Lcls(p,l)＝-logpl
Figure BDA0002532456150000102
Figure BDA0002532456150000103
s603, initializing all weights and offsets to be trained in the model, setting training parameters including learning rate, batch _ size, threshold values of positive and negative samples of RPN and Fast R-CNN networks, and starting model training.
In a specific application example, the step S8 further includes:
and calculating the Recall rate (Recall), Precision rate (Precision), average Precision (Ap) and F-measure of the remote sensing ship target.
S801, the recall rate of the remote sensing ship test sample can be calculated as follows:
Figure BDA0002532456150000104
wherein tp (true positions) is that the positive sample is correctly identified as the positive sample, and the picture of the ship is correctly identified as the ship. P is all true samples.
S802, the precision rate of the remote sensing ship test sample can be calculated as follows:
Figure BDA0002532456150000105
wherein tp (true positions) is that the positive sample is correctly identified as the positive sample, and the picture of the ship is correctly identified as the ship. Fp (false positives) is a false positive sample, i.e. a negative sample is misidentified as a positive sample, and a picture of a ship is misidentified as other classes.
The average accuracy is calculated from each category under various IoU thresholds (0.5, 0.55.., 0.95). The results are first ranked by confidence, and the area under the interpolation accuracy-recall curve is estimated by the average interpolation accuracy of the recall on 11 equidistant horizontal axes. The average accuracy can be calculated as:
Figure BDA0002532456150000111
where r is the recall, c is the number of categories given, c is the ship category in this method, and t is the threshold of IoU. In addition, pinterp(r) can be calculated as:
Figure BDA0002532456150000112
the curve is monotonically decreasing by reassigning the accuracy of each recall r to the maximum accuracy of higher recalls.
S804, the F-measure of the remote sensing ship test sample can be calculated as follows:
Figure BDA0002532456150000113
precision and recall are the Precision and recall explained above.
It is to be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the present invention have been described with reference to the accompanying drawings, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims.

Claims (8)

1. A multi-angle remote sensing ship image target detection method based on a characteristic pyramid structure is characterized by comprising the following steps:
s1, collecting the remote sensing satellite ship image data set, and carrying out sample annotation to obtain an annotation target;
s2, preprocessing the sample in the data set extracted by the S1 to form a complete training data set;
s3, extracting the features of the preprocessed sample by using the improved feature pyramid network to obtain a feature pyramid with multi-layer feature fusion;
s4, generating a candidate area through an RPN;
s5, adding ROI Pooling layers with different Pooling sizes;
s6, building a Fast R-CNN network;
s7, performing data clipping operation of step S2 preprocessing of the training phase on the test data acquired in S1;
s8, the sample processed by S7 is sent to the model constructed by S3, S4, S5 and S6 to be tested to obtain the result, namely the result of classification and regression is finally output by Fast R-CNN.
2. The method for detecting the multi-angle remote sensing ship image target based on the characteristic pyramid structure as claimed in claim 1, wherein the step S1 further comprises:
s101, collecting remote sensing satellite pictures of ports, open sea, wharfs, lakes and other places containing ship targets from a Google Earth satellite map to serve as training images;
and S102, marking the category and the position coordinates of all ship targets in the training image to obtain marked targets.
3. The method for detecting the multi-angle remote sensing ship image target based on the characteristic pyramid structure as claimed in claim 1, wherein the step S2 further comprises:
s201, cutting a training image into pictures of 1000 multiplied by 1000, wherein the overlapping rate is 20%;
s202, removing negative samples in the training set after cutting;
and S203, horizontally turning and rotating the picture to form a complete training data set.
4. The method for detecting the multi-angle remote sensing ship image target based on the characteristic pyramid structure as claimed in claim 3, wherein the step S3 further comprises:
firstly, selecting a ResNet-101 residual error network to extract the characteristics of an input remote sensing satellite image, and selecting the characteristic diagram of the last layer of the last 4 residual error modules to construct a network from bottom to top;
obtaining a feature map with higher resolution by transverse connection and dense connection, and constructing a network from top to bottom, wherein the dense connection means that the input of each layer of feature map not only comprises the output of downsampling of adjacent feature maps, but also comprises the output of downsampling of other upper-layer feature maps, and all the inputs are combined by series connection;
the input of the horizontal connection is the output after 1 multiplied by 1 convolution with the corresponding characteristic diagram in the bottom-up network; the process of obtaining the 3 layers of feature layers is expressed as the following form by taking the lowest 3 layers of the top-down network as the output of the feature extraction network:
Figure FDA0002532456140000021
wherein, CkK-th layer characteristic diagram, P, representing a bottom-up networkkRepresents the feature map of the k-th layer after fusion (k decreases from top to bottom), f1×1And f3×3Represents convolutional layers of 1 × 1 and 3 × 3, Up represents upsampling, 2iAnd representing the size of upsampling, wherein the value of k + i is not more than 5, and through the process, the characteristic pyramid of multi-layer characteristic fusion is obtained.
5. The method for detecting the multi-angle remote sensing ship image target based on the characteristic pyramid structure as claimed in claim 1, wherein the step S4 further comprises:
and respectively building regional generation networks on the 3 layers of feature maps, connecting two 1 × 1 convolutional layers through a 3 × 3 convolutional layer to perform classification prediction of foreground and background and coordinate regression of a boundary frame, wherein the regional generation networks are full convolutional networks, and the parameters of the regional generation networks of the 3 feature maps are independent from each other and do not share the parameters.
6. The method for detecting the multi-angle remote sensing ship image target based on the characteristic pyramid structure as claimed in claim 1, wherein the step S5 further comprises:
and for the candidate region output in the previous step, adding two ROI Pooling layers, wherein the Pooling sizes of the two added ROI Pooling layers are 10 × 3 and 3 × 10, the total number of the ROI Pooling layers is 3 layers, the sizes are 7 × 7, 10 × 3 and 3 × 10 respectively, each candidate region passes through the 3 ROI Pooling layers respectively, and finally outputting feature maps with fixed sizes of 7 × 7 × ConvDepth, 10 × 3 × ConvDepth and 3 × 10 × ConvDepth.
7. The method for detecting the multi-angle remote sensing ship image target based on the characteristic pyramid structure as claimed in claim 1, wherein the step S6 further comprises:
s601, flattening the feature map of each candidate region into a one-dimensional vector, and classifying the candidate regions through a ReLU function, finally respectively passing through two full-connection layers and Softmax; the other performs a more accurate regression of the detection frame coordinates;
s602, the loss function expression of the remote sensing ship target detection network is as follows:
Figure FDA0002532456140000031
wherein liLabels representing objects, piRepresenting the probability distribution, t, of the various classes calculated by the Softmax functioniRepresents the five coordinate vectors of the prediction,
Figure FDA0002532456140000032
the coordinates of the truth detection boxes corresponding to the prediction areas are shown, N represents the number of categories, and only one type of data of the ship is shown, namely NclsThe hyperparameter λ controls the balance between the two losses, where λ is 1 and, in addition, the function L is usedclsAnd LregIs defined as:
Lcls(p,l)＝-logpl
Figure FDA0002532456140000033
Figure FDA0002532456140000034
s603, initializing all weights and offsets to be trained in the model, setting training parameters including learning rate, batch _ size, RPN and threshold values of respective positive and negative samples of Fast R-CNN network, and starting model training.
8. The method for detecting the multi-angle remote sensing ship image target based on the characteristic pyramid structure as claimed in claim 1, wherein the step S8 further comprises:
calculating the Recall rate (Recall), Precision rate (Precision), average Precision (Ap) and F-measure of the remote sensing ship target,
s801, the recall rate of the remote sensing ship test sample can be calculated as follows:
Figure FDA0002532456140000041
wherein tp (true positions) is that the positive sample is correctly identified as the positive sample, the picture of the ship is correctly identified as the ship, and P is all true value samples;
s802, the precision rate of the remote sensing ship test sample can be calculated as follows:
Figure FDA0002532456140000042
wherein, tp (true positions) is that the positive sample is correctly identified as the positive sample, and the picture of the ship is correctly identified as the ship; fp (false positives) is a false positive sample, i.e. a negative sample is misidentified as a positive sample, and a picture of a ship is misidentified as other categories;
s803, the average accuracy is calculated from each category under various IoU thresholds (0.5, 0.55.., 0.95), the results are first ranked by confidence, and the area under the interpolated accuracy-recall curve is estimated by the average interpolation accuracy of the recall on 11 equidistant horizontal axes, where the average accuracy can be calculated as:
Figure FDA0002532456140000043
where r is the recall, c is the number of categories given, c is the ship category in the method, t is the threshold value of IoU, and in addition, pinterp(r) is calculated as:
Figure FDA0002532456140000044
the curve is monotonically decreasing by reassigning the accuracy of each recall r to the maximum accuracy of higher recalls;
s804, calculating the F-measure of the remote sensing ship test sample as follows:
Figure FDA0002532456140000045
precision and recall are as described above.
CN202010521967.XA 2020-06-10 2020-06-10 Multi-angle remote sensing ship image target detection method based on characteristic pyramid structure Active CN111753677B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010521967.XA CN111753677B (en) 2020-06-10 2020-06-10 Multi-angle remote sensing ship image target detection method based on characteristic pyramid structure

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010521967.XA CN111753677B (en) 2020-06-10 2020-06-10 Multi-angle remote sensing ship image target detection method based on characteristic pyramid structure

Publications (2)

Publication Number Publication Date
CN111753677A true CN111753677A (en) 2020-10-09
CN111753677B CN111753677B (en) 2023-10-31

Family

ID=72675908

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010521967.XA Active CN111753677B (en) 2020-06-10 2020-06-10 Multi-angle remote sensing ship image target detection method based on characteristic pyramid structure

Country Status (1)

Country Link
CN (1) CN111753677B (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112183463A (en) * 2020-10-23 2021-01-05 珠海大横琴科技发展有限公司 Ship identification model verification method and device based on radar image
CN112233088A (en) * 2020-10-14 2021-01-15 哈尔滨市科佳通用机电股份有限公司 Brake hose loss detection method based on improved Faster-rcnn
CN112270280A (en) * 2020-11-02 2021-01-26 重庆邮电大学 Open-pit mine detection method in remote sensing image based on deep learning
CN112508848A (en) * 2020-11-06 2021-03-16 上海亨临光电科技有限公司 Deep learning multitask end-to-end-based remote sensing image ship rotating target detection method
CN113536986A (en) * 2021-06-29 2021-10-22 南京逸智网络空间技术创新研究院有限公司 Representative feature-based dense target detection method in remote sensing image
CN113837199A (en) * 2021-08-30 2021-12-24 武汉理工大学 Image feature extraction method based on cross-layer residual error double-path pyramid network
CN114220019A (en) * 2021-11-10 2022-03-22 华南理工大学 Lightweight hourglass type remote sensing image target detection method and system

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109711288A (en) * 2018-12-13 2019-05-03 西安电子科技大学 Remote sensing ship detecting method based on feature pyramid and distance restraint FCN
CN109800716A (en) * 2019-01-22 2019-05-24 华中科技大学 One kind being based on the pyramidal Oceanic remote sensing image ship detecting method of feature
CN111126202A (en) * 2019-12-12 2020-05-08 天津大学 Optical remote sensing image target detection method based on void feature pyramid network

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109711288A (en) * 2018-12-13 2019-05-03 西安电子科技大学 Remote sensing ship detecting method based on feature pyramid and distance restraint FCN
CN109800716A (en) * 2019-01-22 2019-05-24 华中科技大学 One kind being based on the pyramidal Oceanic remote sensing image ship detecting method of feature
CN111126202A (en) * 2019-12-12 2020-05-08 天津大学 Optical remote sensing image target detection method based on void feature pyramid network

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112233088A (en) * 2020-10-14 2021-01-15 哈尔滨市科佳通用机电股份有限公司 Brake hose loss detection method based on improved Faster-rcnn
CN112233088B (en) * 2020-10-14 2021-08-06 哈尔滨市科佳通用机电股份有限公司 Brake hose loss detection method based on improved Faster-rcnn
CN112183463A (en) * 2020-10-23 2021-01-05 珠海大横琴科技发展有限公司 Ship identification model verification method and device based on radar image
CN112183463B (en) * 2020-10-23 2021-10-15 珠海大横琴科技发展有限公司 Ship identification model verification method and device based on radar image
CN112270280B (en) * 2020-11-02 2022-10-14 重庆邮电大学 Open-pit mine detection method in remote sensing image based on deep learning
CN112270280A (en) * 2020-11-02 2021-01-26 重庆邮电大学 Open-pit mine detection method in remote sensing image based on deep learning
CN112508848A (en) * 2020-11-06 2021-03-16 上海亨临光电科技有限公司 Deep learning multitask end-to-end-based remote sensing image ship rotating target detection method
CN112508848B (en) * 2020-11-06 2024-03-26 上海亨临光电科技有限公司 Deep learning multitasking end-to-end remote sensing image ship rotating target detection method
CN113536986A (en) * 2021-06-29 2021-10-22 南京逸智网络空间技术创新研究院有限公司 Representative feature-based dense target detection method in remote sensing image
CN113837199A (en) * 2021-08-30 2021-12-24 武汉理工大学 Image feature extraction method based on cross-layer residual error double-path pyramid network
CN113837199B (en) * 2021-08-30 2024-01-09 武汉理工大学 Image feature extraction method based on cross-layer residual double-path pyramid network
CN114220019A (en) * 2021-11-10 2022-03-22 华南理工大学 Lightweight hourglass type remote sensing image target detection method and system
CN114220019B (en) * 2021-11-10 2024-03-29 华南理工大学 Lightweight hourglass type remote sensing image target detection method and system

Also Published As

Publication number Publication date
CN111753677B (en) 2023-10-31

Similar Documents

Publication Publication Date Title
CN111738112B (en) Remote sensing ship image target detection method based on deep neural network and self-attention mechanism
CN111753677B (en) Multi-angle remote sensing ship image target detection method based on characteristic pyramid structure
CN112308019B (en) SAR ship target detection method based on network pruning and knowledge distillation
Liu et al. Multiscale U-shaped CNN building instance extraction framework with edge constraint for high-spatial-resolution remote sensing imagery
CN108805070A (en) A kind of deep learning pedestrian detection method based on built-in terminal
CN111079739B (en) Multi-scale attention feature detection method
Chen et al. Adaptive effective receptive field convolution for semantic segmentation of VHR remote sensing images
CN114612769B (en) Integrated sensing infrared imaging ship detection method integrated with local structure information
CN107731011B (en) Port berthing monitoring method and system and electronic equipment
CN114565860B (en) Multi-dimensional reinforcement learning synthetic aperture radar image target detection method
CN111898633A (en) High-spectral image-based marine ship target detection method
CN113591617B (en) Deep learning-based water surface small target detection and classification method
Nurkarim et al. Building footprint extraction and counting on very high-resolution satellite imagery using object detection deep learning framework
Soloviev et al. Comparing cnn-based object detectors on two novel maritime datasets
CN115359366A (en) Remote sensing image target detection method based on parameter optimization
CN115861756A (en) Earth background small target identification method based on cascade combination network
CN114565824A (en) Single-stage rotating ship detection method based on full convolution network
CN116935332A (en) Fishing boat target detection and tracking method based on dynamic video
CN114463624A (en) Method and device for detecting illegal buildings applied to city management supervision
CN113486819A (en) Ship target detection method based on YOLOv4 algorithm
CN116363526B (en) MROCNet model construction and multisource remote sensing image change detection method and system
CN116630808A (en) Rotary ship detection method based on remote sensing image feature extraction
CN116503602A (en) Unstructured environment three-dimensional point cloud semantic segmentation method based on multi-level edge enhancement
Chen et al. SAR ship detection under complex background based on attention mechanism
Li et al. Research on ROI algorithm of ship image based on improved YOLO

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant