CN111178283A - Unmanned aerial vehicle image-based ground object identification and positioning method for established route - Google Patents

Unmanned aerial vehicle image-based ground object identification and positioning method for established route Download PDF

Info

Publication number
CN111178283A
CN111178283A CN201911407864.4A CN201911407864A CN111178283A CN 111178283 A CN111178283 A CN 111178283A CN 201911407864 A CN201911407864 A CN 201911407864A CN 111178283 A CN111178283 A CN 111178283A
Authority
CN
China
Prior art keywords
aerial vehicle
unmanned aerial
image
detection
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201911407864.4A
Other languages
Chinese (zh)
Other versions
CN111178283B (en
Inventor
张海军
孙明珊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Graduate School Harbin Institute of Technology
Original Assignee
Shenzhen Graduate School Harbin Institute of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Graduate School Harbin Institute of Technology filed Critical Shenzhen Graduate School Harbin Institute of Technology
Priority to CN201911407864.4A priority Critical patent/CN111178283B/en
Publication of CN111178283A publication Critical patent/CN111178283A/en
Application granted granted Critical
Publication of CN111178283B publication Critical patent/CN111178283B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/10Terrestrial scenes
    • G06V20/13Satellite images
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C11/00Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/23Clustering techniques

Landscapes

  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Remote Sensing (AREA)
  • Multimedia (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Astronomy & Astrophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to the field of unmanned aerial vehicle route inspection, and provides a method for identifying and positioning ground objects of a set route based on an unmanned aerial vehicle image, which comprises the following steps: constructing an unmanned aerial vehicle image detection segmentation data set; constructing a potential area search model; constructing a detection segmentation model based on a high-definition image; generating course positioning and safety early warning; and after the threat target positioning segmentation is completed, obtaining corresponding network parameters, inputting new unmanned aerial vehicle images into a detection segmenter to generate corresponding labeled images, drawing corresponding routes by utilizing unmanned aerial vehicle attitude information corresponding to each image, calculating the distance between the routes and the target, and giving an alarm to the target with the height within the range of H meters. By adopting the positioning method provided by the invention, the threat target is segmented by using the detection segmentation technology, the route is marked in the graph by using the pose information, the actual distance between the target object and the route is calculated, the safety early warning is carried out, and the positioning accuracy is high.

Description

Unmanned aerial vehicle image-based ground object identification and positioning method for established route
Technical Field
The invention belongs to the field of unmanned aerial vehicle route inspection, and particularly relates to a method for identifying and positioning ground objects of a set route based on an unmanned aerial vehicle image.
Background
In recent years, with the maturity of satellite positioning systems, the improvement of electronic and radio control technologies, and the appearance of fixed-wing drone structures, the application of the drone aerial photography technology in the industry enters a rapid development stage. The main requirements of industrial users are data acquisition, material transportation and the like, and the method is also applied to the fields of armed police, power inspection, pipeline inspection, national resources, medical treatment, disaster relief and the like. The application overcomes the defect that the traditional industry consumes a large amount of manpower and material resources, and simultaneously ensures the personal and property safety of operating personnel. The fixed-wing unmanned aerial vehicle has the characteristics of stable flight, long endurance time, accurate GPS positioning and accurate attitude calculation of an Inertial Measurement Unit (IMU), shows better performance than a plurality of rotors in tasks such as routing inspection, disaster relief and the like, and the most important link in the routing inspection task is detection, segmentation, positioning and ranging. For the detection segmentation task, efficient feature representation is a key step of the target detection problem.
However, the unmanned aerial vehicle image has the characteristics of high pixel, less characteristic details of a overlooking angle, large scale change of a target object, dense distribution of small target objects, complex ground surface background, large weather illumination change and the like. This makes such problems less well solved with the above-described object detection and instance segmentation methods. In addition, the lack of data with Mask information in the view of aviation becomes the learning bottleneck of Mask R-CNN in the field. The reason that the small target is lost in the current unmanned aerial vehicle image target detection and instance segmentation is analyzed, and two aspects can be summarized: firstly, a high-definition image is compressed before entering a detection model, so that the characteristics of a small object are lost by nearly 90% before the image enters the model; secondly, the basic network of the detection model mostly adopts the basic network of classification tasks, and the fact that the semantic information lacks local detail information and is not beneficial to positioning is emphasized.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to provide a ground object identification and positioning method of a given route based on an unmanned aerial vehicle image, which solves the technical problems of low positioning accuracy caused by more lost characteristics of small objects and less local detail information in the existing positioning method.
The invention provides a new identification and positioning method, which comprises the steps of firstly obtaining a potential area where a small target exists by using an area search algorithm, then detecting and segmenting an original image and a potential area picture together, wherein a high-definition feature extractor is adopted in feature extraction in the first stage in the detection and segmentation process, a high-quality suggestion frame is generated in a cascading and promoting mode, and a double-scoring mechanism of intersection ratio prediction branches is added in the second stage to promote mask attaching degree. In the step of carrying out the course positioning, different outer azimuth angles are defined, and the three-point collinear equation is utilized to carry out accurate positioning, so that the positioning accuracy is finally improved.
The purpose of the invention is realized by adopting the following technical scheme: a land feature identification and positioning method of a given route based on unmanned aerial vehicle images comprises the following steps: constructing an unmanned aerial vehicle image detection segmentation data set: marking the polygons for cars, trucks, excavators, house buildings, deposits, ground surface water damage and ground surface collapse which threaten the air route, and giving corresponding class marks; constructing a potential area search model: constructing a potential area search model by designing a high-definition image preprocessing algorithm; constructing a detection segmentation model based on a high-definition image: a detection segmentation model is constructed through an HBI-Mask detection segmentation network; generating course positioning and safety early warning: and after the threat target positioning segmentation is completed, obtaining corresponding network parameters, inputting new unmanned aerial vehicle images into a detection segmenter to generate corresponding labeled images, drawing corresponding routes by utilizing unmanned aerial vehicle attitude information corresponding to each image, calculating the distance between the routes and the target, and giving an alarm to the target with the height within the range of H meters.
Further, the construction of the unmanned aerial vehicle image detection segmentation data set comprises the following steps:
collecting pictures shot by the fixed-wing unmanned aerial vehicle, and settling the unmanned aerial vehicle pose information of each picture;
all pictures are marked by adopting VIA software to perform rectangular frame examples and polygonal examples, and the example types comprise: carts, trucks, building construction, water damage and collapse;
and obtaining M pictures through manual marking, cutting and pasting the examples at the later stage, and expanding the data set to N pictures, wherein N is 10 times of M.
Further, constructing the potential area search model comprises the following steps:
searching a potential area by utilizing semantic information of a target in the nature;
obtaining pixel positions of the house building in the image by using the house detector;
designing a self-adaptive clustering model, taking the house position coordinates obtained by each picture as input data, clustering in a single picture, and forming a house area by houses clustered into a cluster, wherein the largest external right rectangle of the cluster is a potential area;
and cutting the potential region, and correspondingly modifying the corresponding true value of the potential region.
Further, the construction of the detection segmentation model based on the high-definition image comprises the following steps:
performing feature extraction by adopting a parallel high-definition feature extraction network;
adopting a progressive regional suggestion network to improve the quality of the small object positive sample;
and a double-scoring mask production mechanism is adopted to improve mask attaching degree.
Further, generating the route location and the safety precaution comprises the following steps:
calculating the flying height, the pitch angle, the yaw angle, the roll angle, the longitude and the latitude of the unmanned aerial vehicle of each picture by using a dongle, and importing pile point longitude and latitude coordinates of a pile point route;
calculating the outer orientation angle of the unmanned aerial vehicle by using the pitch angle, the yaw angle, the roll angle and the longitude and latitude;
projecting the optical center and the stake point to the same local northeast coordinate system, wherein the pixel coordinate of the image point is calculated by using a three-point collinear method;
and connecting all the pile points in the graph to draw a course, and calculating and constructing the actual physical distance of the trolley and the pipeline which are detected and segmented in the detection and segmentation model based on the high-definition image.
Further, the height H is 30 meters.
The invention has the beneficial effects that: the invention adopts a target searching strategy from coarse to fine, simulates a mode of searching the target by human eyes, coarsely positions the potential range of a small target, and performs fine searching after focusing on the range; the integral detection model adopts a progressive detection frame, the number of positive samples is too small due to the fact that the intersection ratio of small object suggestion frames is lower than the common intersection ratio, and the higher intersection ratio generates less positive samples than a threshold value, and in order to relieve the imbalance of the positive samples and the negative samples caused by the reason, a cascaded network structure is adopted to gradually improve the intersection ratio threshold value, so that the quality of the positive samples is stably improved; the small object loss utilizes the base network connected in parallel stage by stage to obtain high-definition characteristics so as to avoid the excessive loss of the information of the small object and finally achieve the effect of reducing the loss rate of the small object. The unmanned aerial vehicle inspection system has the advantages that the inspection task of the unmanned aerial vehicle along a fixed line is realized, firstly, detection and segmentation of threats such as a trolley, a truck, an excavator and a house building are carried out on an image of an unmanned aerial vehicle which is subjected to line inspection aerial photography at a height of 300 meters, then, pixel coordinates of a fixed route are positioned, finally, the actual distance between the route and a threat target is calculated, and early warning processing is carried out on objects within the safety distance of a route area. The method greatly shortens the time for manually checking the threat target by human eyes, and in addition, the function of distance calculation is not possessed by human eyes, so that the method has great practical application value.
Drawings
FIG. 1 is a flow chart of a method for identifying and positioning land features of a given route based on images of an unmanned aerial vehicle according to the present invention;
FIG. 2 is a flow chart of a potential area search method provided by the present invention;
FIG. 3 is a diagram of a model framework for an HBI-Mask provided by the present invention;
FIG. 4 is a flow chart for solving the external azimuth angle provided by the present invention;
FIG. 5 is a diagram of the lane positioning and safety precaution effects provided by the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Fig. 1 shows a flowchart of the method for identifying and positioning land features of an established route based on images of an unmanned aerial vehicle, which is detailed as follows:
step S1: and constructing an unmanned aerial vehicle image detection segmentation data set. The data used for training in the invention adopts a JOUAV CW-20 unmanned plane, and the shooting camera adopts Nikon D810. Generally, line patrol aerial photography is carried out at a position 300 m higher than the ground, and images relate to scenes such as desert, hills, cities and towns. The method comprises the following steps of marking polygons for cars, trucks, excavators, house buildings, deposits, ground surface water damage and ground surface collapse which threaten the air route, and giving corresponding class marks, wherein the specific steps are as follows:
step S1-1: collecting pictures shot by the fixed-wing unmanned aerial vehicle, and settling the unmanned aerial vehicle pose information of each picture;
step S1-2: all pictures are marked by adopting VIA software to perform rectangular frame examples and polygonal examples, and the example types comprise: carts, trucks, building construction, water damage and collapse;
step S1-3: and (4) acquiring 4000 pictures through manual marking, cutting and pasting the examples at the later stage, and expanding the data set to 40000 pictures. In addition, there are also amplification modes with multiple combinations of cleavage and example scenarios.
Step S2: constructing a potential area search model: and constructing a potential area search model by designing a high-definition image preprocessing algorithm. By means of a latent area search model based on semantic information, the detection range of the small target is narrowed by relying on a given scene existing in the nature. Meanwhile, the method is also an effect of amplifying detection, so that the attention of detecting the segmentation model focuses on a potential area with a high possibility of having a small target, and the specific steps are as follows:
step S2-1: the method for searching the potential area utilizes semantic information that targets exist in nature, namely, vehicles mostly exist in a road or a house area, but the road background is single, and vehicle identification is easy. The background of the room area is complex, and the attention of the detection segmenter should be focused on the background;
step S2-2: obtaining pixel positions of the house building in the image by using the house detector;
step S2-3: and designing a self-adaptive clustering model, and clustering in a single picture by taking the house position coordinate obtained by each picture as input data. The houses grouped into a cluster form a house area, and the largest circumscribed positive rectangle of the cluster is the potential area.
Step S2-4: and cutting the potential region, and correspondingly modifying the corresponding true value of the potential region.
In the present embodiment, please refer to fig. 2, a potential area search model is designed. The method comprises the following specific steps:
firstly, the house is separated from the background through a house detection and separation model. And taking the house mask boundary points of all detection results, taking the average value as the central coordinate of each house, and taking the central point pixel coordinate of each house as the input data of the subsequent clustering algorithm.
Secondly, clustering the houses by adopting a clustering algorithm of self-adaptive cluster number, wherein the houses belonging to the same cluster form a house area. Because the number of the house areas in different pictures is different, the house area clustering cannot be performed on each picture by using a fixed cluster number. The specific flow of the algorithm is shown in the AdaCluster algorithm.
AdaCluster takes the pixel coordinate of the center point of the house in a single picture as input, predefines the step length for searching the optimal division level, and takes the clustering label of each pixel as an output result. The method comprises the steps of firstly calculating the similarity between every two pixel points, wherein the Euclidean distance is adopted, and the measurement of the house area mainly depends on the physical distance between houses. And calling a Joint method to obtain a transfer closure of the similarity matrix, and then calculating and dividing horizontal search step length by the transfer closure, wherein T is the transfer closure of the similarity matrix, and n is the number of samples in the data set.
Figure BDA0002349161490000051
The partition level is based on a specific similarity threshold, which ranges from 0 to 1. If the value is larger than the threshold value, the relationship between the two pixels is strong, the two pixels can be divided into one cluster, and if the value is smaller than the threshold value, the two pixels belong to different clusters. When the value takes 0, all samples constitute a class, and when it takes 1, it means that each sample is independently clustered. And obtaining a search step length, and carrying out sample division on all discrete equal difference division levels according to the 5 th step of the algorithm AdaCluster to obtain a group of division results, analyzing the group of division results, and obtaining the cluster number with the largest occurrence frequency as the final division number. The division number corresponds to a plurality of division levels, and the minimum value is the optimal division level.
Figure BDA0002349161490000052
Figure BDA0002349161490000061
The calculation of the transitive closure usually borrows a large top heap as an auxiliary structure to obtain the subscript i of a heap top element in T1,j1I consists of the column index of the row element in T with the top element of the heap. The specific flow can be described by an algorithm Joint.
Figure BDA0002349161490000062
And then, clustering the central points of each house obtained by the house detection model according to the number of the classes calculated in the previous step. This step is to obtain a room area, which is an area where small objects (vehicles) are gathered, by clustering the house distribution obtained in the previous step. The concrete effect can be seen in step four in fig. 2, where the points belonging to each cluster are represented by different colored points.
And finally, regarding the minimum value of the pixel coordinates in each cluster as the coordinates of the upper left corner of the potential area, and regarding the maximum value as the coordinates of the lower right corner of the potential area. In order to prevent vehicles around the room area from being lost, the coordinates of the point at the upper left corner are respectively subtracted by the room width and height, wherein the width and height are equal to the average width and height of all the rooms in the figure. Similarly, the coordinates of the lower right corner are correspondingly expanded. The expanded window is a potential area, and the potential area is cut out to be used as input data of the detection segmentation model.
For vehicles in non-room areas, even in the case of high-altitude shooting, the background is simple, so that the original image entering the detector can be detected, because the vehicle quantity characteristics are easily distinguished from the background. Therefore, the final detection and segmentation result is formed by fusing two parts, wherein one part is a cut picture subblock, the other part is a complete picture, the cut subarea is responsible for detecting small targets in a complex scene (a room area), and the complete picture is responsible for detecting small targets in a simple scene (a non-room area).
Step S3: constructing a detection segmentation model based on a high-definition image: and (5) detecting and segmenting the network through the HBI-Mask to construct a detection and segmentation model. And respectively constructing a high-definition feature extractor, a progressive regional suggestion generation network and a mask generator of a double-scoring mechanism. Aiming at the loss of small target features caused by convolution stacking, a parallel high-definition feature extraction network is adopted; analyzing the reason of imbalance of the small target area suggested positive samples, and designing a progressive area suggested network; and finally, adding a double-score mask generation mechanism based on intersection ratio prediction aiming at low mask attaching degree. These three measures are optimized from three steps of feature extraction, suggestion box generation and mask generation, respectively, see fig. 3. The method specifically comprises the following steps:
step S3-1: and (3) performing feature extraction by adopting a parallel high-definition feature extraction network: in order to prevent the feature extractor in the universal detection network from losing too many features related to small targets, a basic network capable of always keeping bottom-layer high-definition information is adopted as the feature extractor;
step S3-2: the quality of a small object positive sample is improved by adopting a progressive regional suggestion network: for the positive and negative sample division of the small object, the proposal frame with smaller cross-over ratio is tried to be started, and the result after regression is used as the input sample for judging the positive and negative samples under the threshold value of higher cross-over ratio, so that the sufficient positive samples can be ensured to be input into the learner for learning no matter under the condition of small cross-over ratio or large cross-over ratio;
step S3-3: the mask attaching degree is improved by adopting a double-scoring mask production mechanism: the merge ratio prediction branch is added as part of the mask evaluation score, where the ideal mask score is equal to the pixel-level merge ratio between the prediction mask and the true mask, referred to as the mask merge ratio.
Specifically, the steps are as follows:
first, an underlying network that can always maintain underlying high-definition information is employed as a feature extractor. This approach is more accurate than up-sampling recovery from higher feature levels, where the depth of the network from left to right is constantly deepened and the feature map scale from top to bottom is constantly reduced. And no four residual blocks form a stage, and a new-scale feature map is added every time the next stage is entered. The stage includes feature fusion operation among various branches.
Then, a network is generated using the progressive area suggestion box. Firstly, 9 anchor point frames with different sizes and aspect ratios are initialized from each position of the feature map, and the positions of the anchor point frames are often far from the real frame position of the small target, so that in order to guarantee the number of positive samples, the threshold value of IoU is set to be 0.5, the anchor point frame IoU greater than 0.5 is sampled as the positive sample, and the accuracy of the proposed frame generated by the RPN network is low at this moment. In order to further improve the accuracy, in the next stage, the suggested box generated in the last stage and having IoU larger than 0.5 is taken as the anchor candidate box of the next stage, and the threshold value is lifted to IoU to 0.6 at this time, because of the supervision effect of IoU in the last stage, the output predicted value is closer to the true value than the initialized anchor box, and the distribution gravity center moves to IoU higher after the processing of each stage, so that the problem of too small number of positive samples is not caused by the increase of the threshold value IoU.
Finally, a double-scoring mask generation mechanism is constructed. The mask score learning task is decomposed into a mask classification and IoU regression, the mask classification is considered that results can be obtained in the classification task of the position detection rectangular box, the IoU regression becomes a powerful guarantee for guaranteeing mask attaching degree, and more accurate scores contribute to better performance of the characterization model.
Step S4: generating course positioning and safety early warning: and after the threat target positioning segmentation is completed, obtaining corresponding network parameters, inputting the corresponding network parameters into the detection segmenter aiming at the new unmanned aerial vehicle image, and generating a corresponding annotation image. And drawing a corresponding air route by using the attitude information of the unmanned aerial vehicle corresponding to each image, calculating the distance between the air route and the target, and alarming the target within the range of 30 meters. The method comprises the following specific steps:
step S4-1: calculating the flying height, the pitch angle, the yaw angle, the roll angle, the longitude and the latitude of the unmanned aerial vehicle of each picture by using a dongle, and importing pile point longitude and latitude coordinates of a pile point route;
step S4-2: calculating the outer orientation angle of the unmanned aerial vehicle by using the pitch angle, the yaw angle, the roll angle and the longitude and latitude;
step S4-3: projecting the optical center and the stake point to the same local northeast coordinate system, and calculating the pixel coordinate of the image point by using a three-point collinear method;
step S4-4: all the pile points are connected in the graph and drawn into a route, and the actual physical distance between the separated trolley and the pipeline is calculated in the step S3.
Fig. 4 shows a flowchart for solving the outer azimuth angle, and in the present embodiment, the outer azimuth angle is calculated as follows: first, a rotation matrix from the geocentric coordinate system (e) to the secondary geocentric coordinate system (f) is calculated. The invention selects an ellipsoid tangent plane coordinate system of the starting point position of the petroleum pipeline as a ground auxiliary coordinate system, and the rotation matrix at the moment is as follows:
Figure BDA0002349161490000091
wherein L0 and B0 are longitude and latitude of the starting point, and are used as the origin of the auxiliary coordinate system.
And then calculating a rotation matrix from the navigation coordinate system (g) to the geocentric coordinate system (e), wherein the rotation relationship can be decomposed into that the rotation matrix rotates around the Z axis of the geocentric coordinate system for l degrees in a counterclockwise mode, then rotates around the Y axis after the rotation for 90+ b degrees in a clockwise mode, and l and b respectively represent the longitude and latitude of the unmanned aerial vehicle when the current picture is shot.
The rotation matrix is as follows:
Figure BDA0002349161490000092
and then calculating a rotation matrix from the IMU coordinate system (i) to the navigation coordinate system (g), wherein the PHR attitude angle of the unmanned aerial vehicle is used, and the calculation is shown in a formula 3-10, wherein p, h and r respectively correspond to a pitch angle, a yaw angle and a roll angle.
Figure BDA0002349161490000093
And finally, calculating a rotation matrix from the image space coordinate system (p) to the camera coordinate system (c), wherein the rotation matrix from the camera to the IMU coordinate system needs to be calculated before calculating the matrix, and the camera and the IMU coordinate system can be translated without rotation and can be directly skipped because the camera is vertically fixed on the belly. The image space coordinate system and the camera coordinate system differ only in the direction of the coordinate axes, so the rotation matrix consists of the following zero-one matrix:
Figure BDA0002349161490000094
the conversion relationship is represented by the multiplication of four rotation matrices, each arrow corresponding to one of the rotation matrices. The complete scheme can also be represented by the following formula:
Figure BDA0002349161490000095
Figure BDA0002349161490000096
the complete rotation relationship is the definition of the outer azimuth element, the subscript of each element in the formula only represents the number of rows and columns, and the outer azimuth is obtained by solving the inverse trigonometric function as follows:
Figure BDA0002349161490000101
the invention adopts a target searching strategy from coarse to fine, simulates a mode of searching the target by human eyes, coarsely positions the potential range of a small target, and performs fine searching after focusing on the range; the integral detection model adopts a progressive detection frame, the number of positive samples is too small due to the fact that the intersection ratio of small object suggestion frames is lower than the common intersection ratio, and the higher intersection ratio generates less positive samples than a threshold value, and in order to relieve the imbalance of the positive samples and the negative samples caused by the reason, a cascaded network structure is adopted to gradually improve the intersection ratio threshold value, so that the quality of the positive samples is stably improved; the small object loss utilizes the base network connected in parallel stage by stage to obtain high-definition characteristics so as to avoid the excessive loss of the information of the small object and finally achieve the effect of reducing the loss rate of the small object. The unmanned aerial vehicle inspection system has the advantages that the inspection task of the unmanned aerial vehicle along a fixed line is realized, firstly, detection and segmentation of threats such as a trolley, a truck, an excavator and a house building are carried out on an image of an unmanned aerial vehicle which is subjected to line inspection aerial photography at a height of 300 meters, then, pixel coordinates of a fixed route are positioned, finally, the actual distance between the route and a threat target is calculated, and early warning processing is carried out on objects within the safety distance of a route area. The method greatly shortens the time for manually checking the threat target by human eyes, and in addition, the function of distance calculation is not possessed by human eyes, so that the method has great practical application value.
The main contributions of the invention are the following three points: (1) the unmanned aerial vehicle detection segmentation data set is constructed, and a large number of pictures are shot through unmanned aerial vehicle routing inspection, wherein the pictures comprise attitude information such as the flight height, the pitch angle, the yaw angle, the roll angle and the like of the unmanned aerial vehicle; the examples are then marked out on the original picture with rectangular boxes and polygons, constituting the target detection data set. (2) The invention provides a potential region searching method for relieving the problem that small targets in a large-scale image are lost in a detection segmentation task. On the basis of the two-stage detection segmentation method, a high-definition feature extractor, a progressive region suggestion box generation network and the example segmentation of a double-scoring mechanism are provided, so that the problems of small target feature loss, sample imbalance and low mask fitting degree in the original method are effectively solved. (3) And calculating the pixel position of the pile point in the image by utilizing the attitude information of the unmanned aerial vehicle, the self-defined external azimuth angle conversion process and the three-point collinear equation, drawing the direction of the pipeline according to the pixel position, and calculating the physical distance between each detection target and the pipeline. FIG. 5 shows a course positioning and safety pre-warning effect diagram of the present invention, and the method for identifying and positioning the ground object of the established course based on the unmanned aerial vehicle image provided by the present invention has the advantages of accurate positioning and high accuracy.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.

Claims (6)

1. A method for recognizing and positioning ground objects of a given route based on an unmanned aerial vehicle image is characterized in that: the method comprises the following steps:
constructing an unmanned aerial vehicle image detection segmentation data set: marking the polygons for cars, trucks, excavators, house buildings, deposits, ground surface water damage and ground surface collapse which threaten the air route, and giving corresponding class marks;
constructing a potential area search model: constructing a potential area search model by designing a high-definition image preprocessing algorithm;
constructing a detection segmentation model based on a high-definition image: a detection segmentation model is constructed through an HBI-Mask detection segmentation network;
generating course positioning and safety early warning: and after the threat target positioning segmentation is completed, obtaining corresponding network parameters, inputting new unmanned aerial vehicle images into a detection segmenter to generate corresponding labeled images, drawing corresponding routes by utilizing unmanned aerial vehicle attitude information corresponding to each image, calculating the distance between the routes and the target, and giving an alarm to the target with the height within the range of H meters.
2. The method of claim 1, wherein: the unmanned aerial vehicle image detection segmentation data set construction method comprises the following steps:
collecting pictures shot by the fixed-wing unmanned aerial vehicle, and settling the unmanned aerial vehicle pose information of each picture;
all pictures are marked by adopting VIA software to perform rectangular frame examples and polygonal examples, and the example types comprise: carts, trucks, building construction, water damage and collapse;
and obtaining M pictures through manual marking, cutting and pasting the examples at the later stage, and expanding the data set to N pictures, wherein N is 10 times of M.
3. The method of claim 1, wherein: the construction of the potential area search model comprises the following steps:
searching a potential area by utilizing semantic information of a target in the nature;
obtaining pixel positions of the house building in the image by using the house detector;
designing a self-adaptive clustering model, taking the house position coordinates obtained by each picture as input data, clustering in a single picture, and forming a house area by houses clustered into a cluster, wherein the largest external right rectangle of the cluster is a potential area;
and cutting the potential region, and correspondingly modifying the corresponding true value of the potential region.
4. The method of claim 1, wherein: the method for constructing the detection segmentation model based on the high-definition image comprises the following steps:
performing feature extraction by adopting a parallel high-definition feature extraction network;
adopting a progressive regional suggestion network to improve the quality of the small object positive sample;
and a double-scoring mask production mechanism is adopted to improve mask attaching degree.
5. The method of claim 1, wherein: generating airline location and safety warnings includes the steps of:
calculating the flying height, the pitch angle, the yaw angle, the roll angle, the longitude and the latitude of the unmanned aerial vehicle of each picture by using a dongle, and importing pile point longitude and latitude coordinates of a pile point route;
calculating the outer orientation angle of the unmanned aerial vehicle by using the pitch angle, the yaw angle, the roll angle and the longitude and latitude;
projecting the optical center and the stake point to the same local northeast coordinate system, wherein the pixel coordinate of the image point is calculated by using a three-point collinear method;
and connecting all the pile points in the graph to draw a course, and calculating and constructing the actual physical distance of the trolley and the pipeline which are detected and segmented in the detection and segmentation model based on the high-definition image.
6. The method of claim 1, wherein: the height H is 30 meters.
CN201911407864.4A 2019-12-31 2019-12-31 Unmanned aerial vehicle image-based method for identifying and positioning ground feature of established route Active CN111178283B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911407864.4A CN111178283B (en) 2019-12-31 2019-12-31 Unmanned aerial vehicle image-based method for identifying and positioning ground feature of established route

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911407864.4A CN111178283B (en) 2019-12-31 2019-12-31 Unmanned aerial vehicle image-based method for identifying and positioning ground feature of established route

Publications (2)

Publication Number Publication Date
CN111178283A true CN111178283A (en) 2020-05-19
CN111178283B CN111178283B (en) 2023-07-25

Family

ID=70652331

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911407864.4A Active CN111178283B (en) 2019-12-31 2019-12-31 Unmanned aerial vehicle image-based method for identifying and positioning ground feature of established route

Country Status (1)

Country Link
CN (1) CN111178283B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111832443A (en) * 2020-06-28 2020-10-27 华中科技大学 Construction method and application of construction violation detection model
CN113296542A (en) * 2021-07-27 2021-08-24 成都睿铂科技有限责任公司 Aerial photography shooting point acquisition method and system
CN113344900A (en) * 2021-06-25 2021-09-03 北京市商汤科技开发有限公司 Airport runway intrusion detection method, airport runway intrusion detection device, storage medium and electronic equipment
CN113658350A (en) * 2021-08-26 2021-11-16 广东电网有限责任公司佛山供电局 Nest inspection operation management method and device, computer equipment and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108710863A (en) * 2018-05-24 2018-10-26 东北大学 Unmanned plane Scene Semantics dividing method based on deep learning and system
CN109447082A (en) * 2018-08-31 2019-03-08 武汉尺子科技有限公司 A kind of scene motion Target Segmentation method, system, storage medium and equipment
CN110472500A (en) * 2019-07-09 2019-11-19 北京理工大学 A kind of water surface sensation target fast algorithm of detecting based on high speed unmanned boat

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108710863A (en) * 2018-05-24 2018-10-26 东北大学 Unmanned plane Scene Semantics dividing method based on deep learning and system
CN109447082A (en) * 2018-08-31 2019-03-08 武汉尺子科技有限公司 A kind of scene motion Target Segmentation method, system, storage medium and equipment
CN110472500A (en) * 2019-07-09 2019-11-19 北京理工大学 A kind of water surface sensation target fast algorithm of detecting based on high speed unmanned boat

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111832443A (en) * 2020-06-28 2020-10-27 华中科技大学 Construction method and application of construction violation detection model
CN113344900A (en) * 2021-06-25 2021-09-03 北京市商汤科技开发有限公司 Airport runway intrusion detection method, airport runway intrusion detection device, storage medium and electronic equipment
CN113296542A (en) * 2021-07-27 2021-08-24 成都睿铂科技有限责任公司 Aerial photography shooting point acquisition method and system
CN113658350A (en) * 2021-08-26 2021-11-16 广东电网有限责任公司佛山供电局 Nest inspection operation management method and device, computer equipment and storage medium

Also Published As

Publication number Publication date
CN111178283B (en) 2023-07-25

Similar Documents

Publication Publication Date Title
Chen et al. Distribution line pole detection and counting based on YOLO using UAV inspection line video
CN111178283B (en) Unmanned aerial vehicle image-based method for identifying and positioning ground feature of established route
WO2021142902A1 (en) Danet-based unmanned aerial vehicle coastline floating garbage inspection system
CN103389103B (en) A kind of Characters of Geographical Environment map structuring based on data mining and air navigation aid
CN110021072B (en) Holographic mapping-oriented multi-platform point cloud intelligent processing method
CN105930819A (en) System for real-time identifying urban traffic lights based on single eye vision and GPS integrated navigation system
KR102507501B1 (en) Artificial Intelligence-based Water Quality Contaminant Monitoring System and Method
Chen et al. Valid: A comprehensive virtual aerial image dataset
CN112305559A (en) Power transmission line distance measuring method, device and system based on ground fixed-point laser radar scanning and electronic equipment
CN115077556A (en) Unmanned vehicle field operation path planning method based on multi-dimensional map
Li et al. Exploring multiple crowdsourced data to learn deep convolutional neural networks for road extraction
Dickenson et al. Rotated rectangles for symbolized building footprint extraction
Pi et al. Disaster impact information retrieval using deep learning object detection in crowdsourced drone footage
Shi et al. LiDAR localization at 100 FPS: a map-aided and template descriptor-based global method
Tan et al. An adaptive crack inspection method for building surface based on BIM, UAV and edge computing
CN112818964A (en) Unmanned aerial vehicle detection method based on FoveaBox anchor-free neural network
CN112818837B (en) Aerial photography vehicle weight recognition method based on attitude correction and difficult sample perception
CN115690610A (en) Unmanned aerial vehicle navigation method based on image matching
Liu et al. High-precision automatic detection method of illegal construction object images in complex scenes
CN115497052A (en) Saline soil identification monitoring method and system and electronic equipment
Chen et al. All-in-one YOLO architecture for safety hazard detection of environment along high-speed railway
Guo et al. DV3+ HED+: a DCNN-based framework to monitor temporary works and ESAs in railway construction project using VHR satellite images
Lafarge et al. Modeling urban landscapes from point clouds: a generic approach
Deng et al. Chimney and condensing tower detection based on FPN in high resolution remote sensing images
Zhu et al. Automatic utility pole inclination angle measurement using unmanned aerial vehicle and deep learning

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant