CN113989632A - Bridge detection method and device for remote sensing image, electronic equipment and storage medium - Google Patents
Bridge detection method and device for remote sensing image, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113989632A CN113989632A CN202111070328.7A CN202111070328A CN113989632A CN 113989632 A CN113989632 A CN 113989632A CN 202111070328 A CN202111070328 A CN 202111070328A CN 113989632 A CN113989632 A CN 113989632A
- Authority
- CN
- China
- Prior art keywords
- network
- bridge
- remote sensing
- sensing image
- convolution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30181—Earth observation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a bridge detection method and device for remote sensing images, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring a remote sensing image to be detected; inputting a remote sensing image to be detected into a bridge detector network model obtained by pre-training to obtain bridge position information in the remote sensing image; the bridge detector network model comprises an FPN network and an improved Faster R-CNN network; the improved Faster R-CNN network comprises an improved backhaul network, an RPN network and a RoiHead network; the improved backhaul network is formed by introducing a modulation deformable convolution module into a conventional backhaul network on the basis of the conventional backhaul network; the bridge detector network model is obtained by pre-training according to a training image set, wherein each training image in the training image set is provided with position marking information. The invention improves the bridge detection performance.
Description
Technical Field
The invention belongs to the technical field of signal and information processing, and particularly relates to a remote sensing image bridge detection method and device, electronic equipment and a storage medium.
Background
With the development of electronic and information technologies, the acquisition of large-scale airborne satellite-borne remote sensing data is easier, and the quality of remote sensing images is higher. The bridge is a key node in a transportation network, and the accurate position and the number of the bridge in the remote sensing image are obtained, so that the method has important significance for city monitoring, disaster assessment and military reconnaissance.
Aiming at the bridge detection problem, a large amount of research is carried out by scholars at home and abroad, and the existing achievements can be summarized into two types, namely a knowledge-driven detection method and a data-driven detection method. The knowledge-driven detection method relies on artificially set prior knowledge about a target, and generally the method firstly analyzes a scene, artificially defines the occurrence condition of a bridge and the characteristics of the bridge, and takes the situation as a judgment rule. And then, processing the remote sensing image by using technologies such as mathematical morphology and the like, extracting corresponding characteristics according to a judgment rule, and obtaining a final bridge target. The method has clear physical significance, high real-time performance and low requirement on data quantity. However, in practical application scenarios, the bridge has different form and size and different background environments, and the artificially defined prior knowledge is difficult to cover all practical situations. Data-driven detection methods typically first extract features extensively from a dataset. And then matching the extracted characteristics with the target class through an optimization algorithm, wherein the process is called a training process. The trained model has the capability of extracting relevant bridge features from various features and realizing bridge detection. The method does not need to artificially define the prior knowledge about the bridge, but whether the characteristic extraction is sufficient or not and whether the optimization algorithm is proper or not directly influences the expression of the model. A target detection method based on deep learning which is popular in recent years belongs to a data-driven detection method, and a typical bridge detector network model comprises a two-stage fast R-CNN network model and a single-stage YOLO algorithm family.
However, most of bridge detector network models based on deep learning rely on a conventional convolutional neural network to extract features, whereas the conventional convolutional neural network receptive field is a fixed square, and when the conventional convolutional neural network receptive field is applied to a bridge detection task, the problems of insufficient bridge feature extraction and irrelevant background information interference exist.
Disclosure of Invention
In order to solve the problems in the prior art, the invention provides a remote sensing image bridge detection method, a remote sensing image bridge detection device, electronic equipment and a storage medium. The technical problem to be solved by the invention is realized by the following technical scheme:
in a first aspect, an embodiment of the present invention provides a remote sensing image bridge detection method, including:
acquiring a remote sensing image to be detected;
inputting the remote sensing image to be detected into a bridge detector network model obtained by pre-training to obtain bridge position information of the remote sensing image;
wherein the bridge detector network model comprises an FPN network and an improved Faster R-CNN network; the improved Faster R-CNN network comprises an improved backhaul network, an RPN network and a RoiHead network; the improved Backbone network is formed by introducing a modulation deformable convolution module into a conventional Backbone network on the basis of the conventional Backbone network; the bridge detector network model is obtained by pre-training according to a training image set, wherein each training image in the training image set is provided with position marking information.
In one embodiment of the invention, in the bridge detector network model,
the improved backhaul network is used for extracting the characteristics of the input remote sensing image to be detected to obtain a plurality of characteristic information;
the FPN network is used for carrying out feature fusion on the feature information to obtain a multi-scale feature map;
the RPN is used for selecting a candidate region of the multi-scale feature map to obtain a candidate recommendation region;
and the RoiHead network is used for adjusting position information according to the multi-scale feature map and the candidate recommendation area and outputting the bridge position information of the remote sensing image.
In one embodiment of the invention, the improved backhaul network comprises a preprocessing convolution layer and 4 stage modules which are connected in sequence; each stage module comprises a plurality of block modules which are connected in sequence, the input end and the output end of each block module are connected, and each block module comprises a first convolution layer, a second convolution layer and a third convolution layer which are connected in sequence; the preprocessing convolution layer, the first convolution layer and the third convolution layer in all the stage modules are conventional convolution layers; in the first stage module, all of the second convolutional layers are conventional convolutional layers, and in the other three stage modules, all of the second convolutional layers are modulated deformable convolutional modules.
In one embodiment of the present invention, the convolution kernels of the preprocessed convolutional layers have a size of 7 × 7, and the convolution kernels of all of the first convolutional layers and the third convolutional layers have a size of 3 × 3.
In one embodiment of the invention, the size of the convolution kernel of the modulated deformable convolution module is 3 x 3.
In one embodiment of the present invention, the computation process of the modulation deformable convolution module is expressed as:
wherein, N is the total number of sampling points in the convolution process, x and f respectively represent an input characteristic diagram and an output characteristic diagram, k represents the central position of all the sampling points, and k + knN-th sample point, k, representing a conventional convolutionn∈(-1,1),(-1,0),...,(1,1)，ΔknAnd Δ mnRespectively representing the adaptive deviation and the adaptive modulation coefficient added by the modulation deformable convolution module at the nth sampling point, x (k + k)n+Δkn) Obtained by a bilinear interpolation method.
In a second aspect, an embodiment of the present invention provides a remote sensing image bridge detection apparatus, including:
the data acquisition module is used for acquiring a remote sensing image to be detected;
the data detection module is used for inputting the remote sensing image to be detected into a bridge detector network model obtained by pre-training to obtain bridge position information of the remote sensing image;
wherein the bridge detector network model comprises an FPN network and an improved Faster R-CNN network; the improved Faster R-CNN network comprises an improved backhaul network, an RPN network and a RoiHead network; the improved Backbone network is formed by introducing a modulation deformable convolution module into a conventional Backbone network on the basis of the conventional Backbone network; the bridge detector network model is obtained by pre-training according to a training image set, wherein each training image in the training image set is provided with position marking information.
In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete mutual communication through the communication bus;
the memory is used for storing a computer program;
the processor is used for realizing the steps of the remote sensing image bridge detection method provided by the embodiment of the invention when the program stored in the memory is executed.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the remote sensing image bridge detection method provided in the embodiment of the present invention are implemented.
The invention has the beneficial effects that:
according to the remote sensing image bridge detection method provided by the invention, the improved Faster R-CNN with the FPN network is used for bridge detection, and the improved Faster R-CNN is introduced with the modulation deformable convolution module, so that the receptive field of the improved Faster R-CNN network is more diverse, the receptive field shape can be self-adaptively fitted to various bridge shapes, the bridge characteristic extraction is more sufficient, the bridge detection performance is effectively improved, and the bridge detection method is more adaptive to the actual bridge detection application scene.
The present invention will be described in further detail with reference to the accompanying drawings and examples.
Drawings
FIG. 1 is a schematic flow chart of a bridge detection method using remote sensing images according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a bridge detector network model provided by an embodiment of the invention;
FIG. 3 is a schematic diagram of a specific structure of a bridge detector network model according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of each block module in an improved backhaul network according to an embodiment of the present invention;
FIG. 5 is a diagram illustrating a specific convolution process of a modulated deformable convolution module in each block module according to an embodiment of the present invention;
FIGS. 6a to 6e are schematic diagrams of bridge detection results of an original model and a model of the invention in a multi-target scene provided by an embodiment of the invention;
FIGS. 7a to 7e are schematic diagrams of bridge detection results of an original model and a model of the present invention in a dense target scene according to an embodiment of the present invention;
FIG. 8 is a schematic structural diagram of a bridge detection device using remote sensing images according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to specific examples, but the embodiments of the present invention are not limited thereto.
Example one
The main task of object detection is to find an object of a specified category in an input scene image and to obtain its precise location. The embodiment of the invention focuses on the bridge detection problem in the remote sensing image, introduces the modulation deformable convolution module, improves the existing data-driven detection method based on deep learning, and focuses on solving the problem that the main network receptive field of a bridge detector network model is fixed and various bridge characteristics cannot be fully extracted.
In a first aspect, an embodiment of the present invention provides a remote sensing image bridge detection method, including the following steps:
s101, obtaining a remote sensing image to be detected.
S102, inputting the remote sensing image to be detected into a bridge detector network model obtained through pre-training to obtain bridge position information of the remote sensing image.
Specifically, referring to fig. 2, when detecting a bridge in a remote sensing image according to an embodiment of the present invention, a Network model of a bridge detector is a Network structure including a Feature Pyramid Network (FPN for short) and an improved fast R-CNN Network; the improved Faster R-CNN Network comprises an improved backhaul Network, a regional selection Network (RPN for short) and a RoiHead Network; the improved Backbone network is formed by introducing a modulation deformable convolution module into a conventional Backbone network on the basis of the conventional Backbone network.
In the bridge detector network model, specifically: the improved backhaul network is used for extracting the characteristics of an input remote sensing image to be detected to obtain a plurality of characteristic information; the FPN network adopts a top-down path and is used for carrying out feature fusion on a plurality of feature information to obtain a multi-scale feature map; the RPN is used for selecting a candidate region for the multi-scale feature map to obtain a candidate recommended region; and the RoiHead network is used for adjusting the position information according to the multi-scale characteristic diagram and the candidate recommended region and outputting the bridge position information of the remote sensing image.
Referring to fig. 3, an embodiment of the present invention provides a structure of an optional improved backhaul network, where the improved backhaul network includes a preprocessing convolutional layer and 4 stage modules connected in sequence; each stage module comprises a plurality of block modules which are connected in sequence, the input end and the output end of each block module are connected, and the input end and the output end of each block module are subjected to addition operation through an adder; referring to fig. 4, each block module includes a first convolution layer, a second convolution layer, and a third convolution layer connected in sequence, where the first convolution layer, the second convolution layer, and the third convolution layer are respectively connected to an activation layer, for example, by using a Relu activation function; preprocessing the convolution layer, wherein the first convolution layer and the third convolution layer in all the stage modules are conventional convolution layers; in the first stage Module, all the second convolution layers are normal convolution layers, and in the other three stage modules, all the second convolution layers are Modulated Deformable convolution modules (md conv).
Preferably, the size of the convolution kernel of the preprocessed convolutional layer is 7 × 7, and the size of the convolution kernels of all the first convolutional layers and the third convolutional layers is 3 × 3.
Preferably, the size of the convolution kernel of the modulated deformable convolution module is 3 x 3.
The calculation process of the modulation deformable convolution module of the embodiment of the invention can be expressed as follows:
wherein, N is the total number of sampling points in the convolution process, x and f respectively represent an input characteristic diagram and an output characteristic diagram, k represents the central position of all the sampling points, and k + knN-th sample point, k, representing a conventional convolutionn∈(-1,1),(-1,0),...,(1,1)，ΔknAnd Δ mnRespectively representing the adaptive deviation and the adaptive modulation coefficient added by the modulation deformable convolution module at the nth sampling point. Wherein, x (k + k)n+Δkn) Can be obtained by using a bilinear interpolation method.
The convolution implementation process of the modulation deformable convolution module is illustrated in fig. 5, and fig. 5 shows a specific implementation process of the k × k modulation deformable convolution module when k is 3: inputting an input feature map with dimensions H multiplied by W multiplied by C into a modulation deformable convolution module, wherein H, W, C respectively represents the height, width and channel number of the input feature map, obtaining a three-dimensional tensor H multiplied by W multiplied by 3k through an additional conventional convolution branch, controlling the range of a modulation deformable convolution operation receptive field with (H, W) as the central position of a sampling point by the 3k variable values corresponding to any (H, W) position of the tensor, and 0< H < H < W. Wherein 2k variables are adaptive offsets (offsets) of k sampling points, and the remaining k variables are adaptive modulation coefficients (modulation scales) of k sampling points.
According to the embodiment of the invention, the modulation deformable convolution branch is introduced, the deviation and the modulation coefficient of each sampling point are acquired in a self-adaptive manner, the sensing field shape can be self-adaptively fitted to various bridge shapes, the bridge characteristics are more fully extracted, and the bridge detection performance is effectively improved.
Meanwhile, please refer to fig. 3 again, the specific network structures of the FPN network, the RPN network, and the RoiHead network are respectively schematically given, and here, the network structures are only schematically given, and the corresponding specific network structures can be respectively implemented by using the existing FPN network, RPN network, and RoiHead network, and the network structures of the FPN network, RPN network, and RoiHead network are not described in detail herein.
For the bridge detector network model, the embodiment of the invention performs training to determine according to the training image set, each training image in the training image set has position labeling information, and the labeling can be performed by using a rectangular frame, but is not limited to the rectangular frame. In the training process, a random Gradient Descent (SGD) optimizer is used for optimizing parameters of the bridge detector network model, and the bridge detector network model is trained to obtain a trained bridge detector network model. The loss function adopted in the training process can be a cross entropy loss function and a mean square error loss function.
And after training is completed, obtaining a trained bridge detector network model, inputting the remote sensing image to be detected into the trained bridge detector network model, so that the bridge target position information of the remote sensing image can be obtained, and bridge detection is realized.
In order to verify the effectiveness of the remote sensing image bridge detection method provided by the embodiment of the invention, the following experiment is used for explanation.
1. Conditions of the experiment
The method provided by the invention is verified by using the disclosed bridge remote sensing image set, the resolution of each bridge remote sensing image in the bridge remote sensing image set is 1-4 m, each bridge remote sensing image at least comprises one bridge target, and the bridge remote sensing image set comprises 2000 bridge remote sensing images and 3706 bridge instances. Carrying out division of a bridge remote sensing image training set and a bridge remote sensing image testing set on the bridge remote sensing image set: and randomly selecting 80% for training the bridge detector network model and 20% for testing the bridge detector network model. The method provided by the invention is verified by taking a Faster R-CNN network model with an FPN network as a basic framework. The experimental operating platform used NVIDIA GTX1660SUPER GPU and 16GB RAM for experimental validation based on python3.8 and pytorch 1.7.
2. Analysis of experimental content and results
The embodiment of the invention mainly solves the problem that the main network receptive field in the existing bridge detector network model based on deep learning is fixed and various bridge characteristics can not be effectively extracted. By comparing whether a modulation deformable convolution module is used in several types of backhaul networks or not, the training conditions of the bridge detector network models of all groups are ensured to be consistent, the detection performance of the bridge detector network models of all groups is analyzed quantitatively and qualitatively, and the effectiveness of the method provided by the invention is verified.
The first set of experiments: based on a Faster R-CNN network framework with an FPN network, specifically using ResNet as a backhaul network, respectively training a model using original ResNet as a conventional backhaul network and a model using ResNet introduced with mdconv as an improved backhaul network, testing all test images in a bridge remote sensing image test set (remote sensing images to be detected) by using the pre-trained models, and respectively using two indexes of Average Precision (AP for short) and Recall rate (Recall) to quantitatively analyze test results, wherein the test results are shown in Table 1.
TABLE 1 evaluation comparison of test results of the first set of experiments
| Model (model) | AP(％) | Recall(％) |
| Original model | 93.7 | 73.7 |
| Model of the invention | 95.7 | 74.4 |
From the experimental results in table 1, it can be seen that both the AP and the Recall indexes of the method provided in the embodiment of the present invention are superior to the original model, and particularly, the improvement range of the AP index is 2%, which indicates that in a bridge detector network model using a ResNet network as a backhaul network, the detection accuracy of a bridge can be effectively improved by using md conv, and the extraction capability of the bridge target feature reflecting the improved backhaul network is effectively improved.
The second set of experiments: based on a Faster R-CNN network framework with an FPN network, specifically using ResNeXt as a Backbone network, respectively training a model using original ResNeXt as a conventional Backbone network and a model using ResNeXt with md conv introduced as an improved Backbone network, testing all test images in a test image set by using a trained bridge detector network model, and respectively using AP and Recall indexes to perform quantitative analysis on test results, wherein the test results are shown in Table 2.
TABLE 2 evaluation comparison of test results of the second set of experiments
| Model (model) | AP(％) | Recall(％) |
| Original model | 94.4 | 74.5 |
| Model of the invention | 95.8 | 75.6 |
As can be seen from the experimental results in table 2, the method provided in the embodiment of the present invention can significantly improve the performance of the bridge detector network model in the bridge detector network model using ResNeXt as a backhaul. The AP index is improved by 1.4%, the Recall index is improved by 1.1%, which shows that the detection precision of the bridge detector network model is obviously improved, the missed detection condition of the bridge detector network model is less, and the extraction capability of the bridge target characteristics of the improved backhaul network is reflected laterally and effectively improved.
Referring to fig. 6a to 6e, fig. 6a to 6e respectively show a real bridge distribution and a corresponding bridge detection result in a multi-target scene, specifically, a marking condition of the real bridge distribution and the bridge position in the multi-target scene, a bridge detection result using an original model in a first set of experiments, a bridge detection result using the model of the present invention in the first set of experiments, a bridge detection result using the original model in a second set of experiments, and a bridge detection result using the model of the present invention in the second set of experiments are sequentially marked from left to right. The black rectangular boxes in fig. 6a are marked with a plurality of targets existing in the original scene, and the gray rectangular boxes in fig. 6b to 6e are marked with a plurality of targets detected by the detector network model, so that it can be seen that in the multi-target scene, the positions of the bounding boxes of the bridge targets corresponding to the bridge detection results of fig. 6c and 6e are more fitted with the real marking conditions by using the bridge detector network model provided by the invention in two groups of experiments. It should be noted that the bridge detection results obtained by using the original model in the second set of experiments are shown in fig. 6d, which shows that the missing detection condition occurs, but the model of the present invention can better detect all bridge targets in the multi-target scene.
Referring to fig. 7a to 7e, fig. 7a to 7e respectively show the real bridge distribution and the corresponding bridge detection results in the dense target scene, specifically, the real bridge distribution and position labeling conditions in the dense target scene, the bridge detection results using the original model in the first set of experiments, the bridge detection results using the model of the present invention in the first set of experiments, the bridge detection results using the original model in the second set of experiments, and the bridge detection results using the model of the present invention in the second set of experiments are sequentially shown from left to right. Like fig. 6a to 6e, the black rectangular boxes in fig. 7a mark a plurality of objects existing in the original dense object scene,
the gray rectangular boxes in fig. 7 b-7 e mark a plurality of targets detected by the detector network model, and it can be seen that in a dense multi-target scene, the bridge detector network model provided by the invention is used in two groups of experiments, and the model of the invention can better detect all bridge targets and the position regression is more accurate. It should also be noted that the bridge detection results obtained by using the original model in the first set of experiments are missing as shown in fig. 7b, but the model of the present invention can better detect all bridge targets in a dense target scene.
As can be seen from FIGS. 6a to 6e and FIGS. 7a to 7e, the bridge detector model provided by the present invention can better detect all bridge targets regardless of the multi-target bridge detection scenario or the dense multi-target bridge detection scenario, and is more suitable for the actual bridge detection application scenario.
In summary, the remote sensing image bridge detection method provided by the embodiment of the invention utilizes the improved Faster R-CNN with the FPN network to perform bridge detection, and the improved Faster R-CNN is introduced with the modulated deformable convolution module, so that the sensing fields of the improved Faster R-CNN network are more diverse, the shapes of the sensing fields can be self-adaptively fitted to various bridge shapes, the bridge characteristics are more fully extracted, the bridge detection performance is effectively improved, and the bridge detection method is more suitable for the actual bridge detection application scene.
In addition, the method provided by the embodiment of the invention does not need the prior knowledge about the target set by people, and can self-adaptively complete the bridge detection task; the bridge detection improvement method (introducing modulation deformable convolution) provided by the embodiment of the invention can be used for various detection models based on deep learning, including a two-stage detector network model and a single-stage detector network model, and has strong universality and easy embedding.
In a second aspect, an embodiment of the present invention provides a remote sensing image bridge detection apparatus, please refer to fig. 7, including:
and the data acquisition module 801 is used for acquiring the remote sensing image to be detected.
The data detection module 802 is configured to input the remote sensing image to be detected into a bridge detector network model obtained through pre-training, so as to obtain bridge position information of the remote sensing image;
the bridge detector network model comprises an FPN network and an improved Faster R-CNN network; the improved Faster R-CNN network comprises an improved backhaul network, an RPN network and a RoiHead network; the improved backhaul network is formed by introducing a modulation deformable convolution module into a conventional backhaul network on the basis of the conventional backhaul network; the bridge detector network model is obtained by pre-training according to a training image set, wherein each training image in the training image set is provided with position marking information.
Further, in the data detection module 802 according to the embodiment of the present invention, particularly in the bridge detector network model,
the improved backhaul network is used for extracting the characteristics of an input remote sensing image to be detected to obtain a plurality of characteristic information;
the FPN network is used for carrying out feature fusion on the feature information to obtain a multi-scale feature map;
the RPN is used for selecting a candidate region for the multi-scale feature map to obtain a candidate recommended region;
and the RoiHead network is used for adjusting the position information according to the multi-scale characteristic diagram and the candidate recommended region and outputting the bridge position information of the remote sensing image.
Further, in the data detection module 802 according to the embodiment of the present invention, the specifically improved backhaul network includes a preprocessing convolutional layer and 4 stage modules connected in sequence; each stage module comprises a plurality of block modules which are connected in sequence, the input end and the output end of each block module are connected, and each block module comprises a first convolution layer, a second convolution layer and a third convolution layer which are connected in sequence; preprocessing the convolution layer, wherein the first convolution layer and the third convolution layer in all the stage modules are conventional convolution layers; in the first stage module, all of the second convolutional layers are conventional convolutional layers, and in the other three stage modules, all of the second convolutional layers are modulated deformable convolutional modules.
Further, in the data detection module 802 according to the embodiment of the present invention, the size of the convolution kernel of the preprocessed convolutional layer is 7 × 7, and the sizes of the convolution kernels of all the first convolutional layer and the third convolutional layer are 3 × 3.
Further, in the data detection module 802 according to the embodiment of the present invention, the size of the convolution kernel of the modulation deformable convolution module is 3 × 3.
Further, in the data detection module 802 according to the embodiment of the present invention, the calculation process of the specific modulation deformable convolution module is represented as:
wherein, N is the total number of sampling points in the convolution process, x and f respectively represent an input characteristic diagram and an output characteristic diagram, k represents the central position of all the sampling points, and k + knRepresents a conventionSample point n of convolution, kn∈(-1,1),(-1,0),...,(1,1)，ΔknAnd Δ mnRespectively representing the adaptive deviation and the adaptive modulation coefficient added by the modulation deformable convolution module at the nth sampling point, x (k + k)n+Δkn) Obtained by a bilinear interpolation method.
In a third aspect, an embodiment of the present invention further provides an electronic device, please refer to fig. 8, which includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, where the processor 901, the communication interface 902, and the memory 903 complete communication with each other through the communication bus 904,
a memory 903 for storing computer programs;
the processor 901 is configured to implement the steps of the remote sensing image bridge detection method according to the first aspect when executing the program stored in the memory 903.
The electronic device may be: desktop computers, laptop computers, intelligent mobile terminals, servers, and the like. Without limitation, any electronic device that can implement the present invention is within the scope of the present invention.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
In a fourth aspect, corresponding to the remote sensing image bridge detection method provided in the first aspect, an embodiment of the present invention further provides a computer readable storage medium, where a computer program is stored in the computer readable storage medium, and when the computer program is executed by a processor, the steps of the remote sensing image bridge detection method provided in the embodiment of the present invention are implemented.
It should be noted that the apparatus, the electronic device and the storage medium according to the embodiments of the present invention are respectively an apparatus, an electronic device and a storage medium to which the remote sensing image bridge detection method is applied, and all embodiments of the remote sensing image bridge detection method are applicable to the apparatus, the electronic device and the storage medium, and can achieve the same or similar beneficial effects.
In the description of the present invention, it is to be understood that the terms "first", "second" and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples described in this specification can be combined and combined by those skilled in the art.
While the present application has been described in connection with various embodiments, other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from a review of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (9)
1. A bridge detection method based on remote sensing images is characterized by comprising the following steps:
acquiring a remote sensing image to be detected;
inputting the remote sensing image to be detected into a bridge detector network model obtained by pre-training to obtain bridge position information in the remote sensing image;
wherein the bridge detector network model comprises an FPN network and an improved Faster R-CNN network; the improved Faster R-CNN network comprises an improved backhaul network, an RPN network and a RoiHead network; the improved Backbone network is formed by introducing a modulation deformable convolution module into a conventional Backbone network on the basis of the conventional Backbone network; the bridge detector network model is obtained by pre-training according to a training image set, wherein each training image in the training image set is provided with position marking information.
2. The remote sensing image bridge inspection method of claim 1, wherein, in the bridge inspection network model,
the improved backhaul network is used for extracting the characteristics of the input remote sensing image to be detected to obtain a plurality of characteristic information;
the FPN network is used for carrying out feature fusion on the feature information to obtain a multi-scale feature map;
the RPN is used for selecting a candidate region based on the multi-scale feature map to obtain a candidate recommendation region;
and the RoiHead network is used for adjusting position information according to the multi-scale feature map and the candidate recommendation area and outputting the bridge position information of the remote sensing image.
3. The remote sensing image bridge detection method according to claim 1, wherein the improved backhaul network comprises a preprocessing convolutional layer and 4 stage modules which are connected in sequence; each stage module comprises a plurality of block modules which are connected in sequence, the input end and the output end of each block module are connected, and each block module comprises a first convolution layer, a second convolution layer and a third convolution layer which are connected in sequence; the preprocessing convolution layer, the first convolution layer and the third convolution layer in all the stage modules are conventional convolution layers; in the first stage module, all of the second convolutional layers are conventional convolutional layers, and in the other three stage modules, all of the second convolutional layers are modulated deformable convolutional modules.
4. The remote sensing image bridge detection method of claim 3, wherein the size of the convolution kernel of the preprocessed convolutional layer is 7 x 7; the convolution kernels of all the first convolution layers and the third convolution layers are 3 × 3 in size.
5. The remote sensing image bridge detection method of claim 3, wherein the size of the convolution kernel of the modulated deformable convolution module is 3 x 3.
6. The remote sensing image bridge detection method of claim 1, wherein the computation process of the modulation deformable convolution module is expressed as:
wherein, N is the total number of sampling points in the convolution process, x and f respectively represent an input characteristic diagram and an output characteristic diagram, k represents the central position of all the sampling points and represents the nth sampling point of the conventional convolution, and k represents the nth sampling point of the conventional convolutionn∈(-1,1),(-1,0),...,(1,1)，ΔknAnd Δ mnRespectively representing the adaptive deviation and the adaptive modulation coefficient added by the modulation deformable convolution module at the nth sampling point, x (k + k)n+Δkn) Obtained by a bilinear interpolation method.
7. A remote sensing image bridge detection device is characterized by comprising:
the data acquisition module is used for acquiring a remote sensing image to be detected;
the data detection module is used for inputting the remote sensing image to be detected into a bridge detector network model obtained by pre-training to obtain bridge position information of the remote sensing image;
wherein the bridge detector network model comprises an FPN network and an improved Faster R-CNN network; the improved Faster R-CNN network comprises an improved backhaul network, an RPN network and a RoiHead network; the improved Backbone network is formed by introducing a modulation deformable convolution module into a conventional Backbone network on the basis of the conventional Backbone network; the bridge detector network model is obtained by pre-training according to a training image set, wherein each training image in the training image set is provided with position marking information.
8. An electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus;
the memory is used for storing a computer program;
the processor is used for realizing the steps of the remote sensing image bridge detection method according to any one of claims 1 to 6 when executing the program stored in the memory.
9. A computer-readable storage medium, characterized in that,
the computer readable storage medium stores a computer program which when executed by a processor implements the steps of the remote sensing image bridge detection method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111070328.7A CN113989632A (en) | 2021-09-13 | 2021-09-13 | Bridge detection method and device for remote sensing image, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111070328.7A CN113989632A (en) | 2021-09-13 | 2021-09-13 | Bridge detection method and device for remote sensing image, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113989632A true CN113989632A (en) | 2022-01-28 |
Family
ID=79735719
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111070328.7A Withdrawn CN113989632A (en) | 2021-09-13 | 2021-09-13 | Bridge detection method and device for remote sensing image, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113989632A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115661666A (en) * | 2022-12-12 | 2023-01-31 | 航天宏图信息技术股份有限公司 | Bridge identification method and device in remote sensing image, electronic equipment and medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110188709A (en) * | 2019-06-03 | 2019-08-30 | 济南浪潮高新科技投资发展有限公司 | The detection method and detection system of oil drum in remote sensing image based on deep learning |
| CN111723660A (en) * | 2020-05-18 | 2020-09-29 | 天津大学 | Detection method for long ground target detection network |
| CN111861978A (en) * | 2020-05-29 | 2020-10-30 | 陕西师范大学 | Bridge crack example segmentation method based on Faster R-CNN |
| CN112733686A (en) * | 2020-12-31 | 2021-04-30 | 武汉兴图新科电子股份有限公司 | Target object identification method and device used in image of cloud federation |
| CN112861795A (en) * | 2021-03-12 | 2021-05-28 | 云知声智能科技股份有限公司 | Method and device for detecting salient target of remote sensing image based on multi-scale feature fusion |
| CN113159300A (en) * | 2021-05-15 | 2021-07-23 | 南京逸智网络空间技术创新研究院有限公司 | Image detection neural network model, training method thereof and image detection method |
-
2021
- 2021-09-13 CN CN202111070328.7A patent/CN113989632A/en not_active Withdrawn
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110188709A (en) * | 2019-06-03 | 2019-08-30 | 济南浪潮高新科技投资发展有限公司 | The detection method and detection system of oil drum in remote sensing image based on deep learning |
| CN111723660A (en) * | 2020-05-18 | 2020-09-29 | 天津大学 | Detection method for long ground target detection network |
| CN111861978A (en) * | 2020-05-29 | 2020-10-30 | 陕西师范大学 | Bridge crack example segmentation method based on Faster R-CNN |
| CN112733686A (en) * | 2020-12-31 | 2021-04-30 | 武汉兴图新科电子股份有限公司 | Target object identification method and device used in image of cloud federation |
| CN112861795A (en) * | 2021-03-12 | 2021-05-28 | 云知声智能科技股份有限公司 | Method and device for detecting salient target of remote sensing image based on multi-scale feature fusion |
| CN113159300A (en) * | 2021-05-15 | 2021-07-23 | 南京逸智网络空间技术创新研究院有限公司 | Image detection neural network model, training method thereof and image detection method |
Non-Patent Citations (3)
| Title |
|---|
| HE HUANG 等: "ROTATION AND SCALE-INVARIANT OBJECT DETECTOR FOR HIGH RESOLUTION OPTICAL REMOTE SENSING IMAGES", 2019 IEEE * |
| XIZHOU ZHU 等: "Deformable ConvNets v2: More Deformable, Better Results", 2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR) * |
| 田婷婷 等: "基于多尺度特征融合网络的遥感影像目标检测", 激光与光电子学进展 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115661666A (en) * | 2022-12-12 | 2023-01-31 | 航天宏图信息技术股份有限公司 | Bridge identification method and device in remote sensing image, electronic equipment and medium |
| CN115661666B (en) * | 2022-12-12 | 2023-04-07 | 航天宏图信息技术股份有限公司 | Bridge identification method and device in remote sensing image, electronic equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111339858B (en) | Oil and gas pipeline marker identification method based on neural network | |
| US20180182039A1 (en) | Method, system, apparatus, and storage medium for realizing antifraud in insurance claim based on consistency of multiple images | |
| US20190156486A1 (en) | Method and system of detecting image tampering, electronic device and storage medium | |
| CN107330731B (en) | Method and device for identifying click abnormity of advertisement space | |
| CN111398176B (en) | Water body water color abnormity remote sensing identification method and device based on pixel scale characteristics | |
| CN112085056B (en) | Target detection model generation method, device, equipment and storage medium | |
| CN113469088A (en) | SAR image ship target detection method and system in passive interference scene | |
| CN111144398A (en) | Target detection method, target detection device, computer equipment and storage medium | |
| CN112419268A (en) | Method, device, equipment and medium for detecting image defects of power transmission line | |
| CN112364974B (en) | YOLOv3 algorithm based on activation function improvement | |
| CN111179270A (en) | Image co-segmentation method and device based on attention mechanism | |
| CN113989632A (en) | Bridge detection method and device for remote sensing image, electronic equipment and storage medium | |
| CN116563841B (en) | Detection method and detection device for power distribution network equipment identification plate and electronic equipment | |
| CN116091874B (en) | Image verification method, training method, device, medium, equipment and program product | |
| CN109934151B (en) | Face detection method based on movidius computing chip and Yolo face | |
| CN116597246A (en) | Model training method, target detection method, electronic device and storage medium | |
| CN103903258B (en) | Method for detecting change of remote sensing image based on order statistic spectral clustering | |
| CN112308061B (en) | License plate character recognition method and device | |
| CN117036843A (en) | Target detection model training method, target detection method and device | |
| CN115063473A (en) | Object height detection method and device, computer equipment and storage medium | |
| CN114298229A (en) | Crop type determination method and device, electronic equipment and storage medium | |
| WO2020118901A1 (en) | Deep learning-based antenna downtilt angle measurement method | |
| CN110942179A (en) | Automatic driving route planning method and device and vehicle | |
| CN116740385B (en) | Equipment quality inspection method, device and system | |
| CN116630909B (en) | Unmanned intelligent monitoring system and method based on unmanned aerial vehicle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20220128 |