CN106097304A - A kind of unmanned plane real-time online ground drawing generating method - Google Patents
A kind of unmanned plane real-time online ground drawing generating method Download PDFInfo
- Publication number
- CN106097304A CN106097304A CN201610373105.0A CN201610373105A CN106097304A CN 106097304 A CN106097304 A CN 106097304A CN 201610373105 A CN201610373105 A CN 201610373105A CN 106097304 A CN106097304 A CN 106097304A
- Authority
- CN
- China
- Prior art keywords
- frame
- map
- unmanned plane
- key frame
- pixel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/05—Geographic models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30181—Earth observation
Abstract
The present invention proposes a kind of unmanned plane real-time online ground drawing generating method, unmanned plane realizes map real-time online rebuild, then pass, by high band wide data transmission link, the map rebuild back ground, be applied to unmanned plane and obtain the three-dimensional map of environment about in real time and realize the location to self.By the present invention, unmanned plane can be in the flight of unknown spatial domain, under the conditions of self-position is uncertain, by the detection of environmental information with compare, and extraction and the coupling to the characteristic information of environment, obtain the location of self and carry out the structure of three-dimensional map.Present invention can apply to battlefield demand (such as unmanned plane autonomous flight, attack)；The calamity emergency rescues such as fire/earthquake/flood；Unmanned plane security administration monitoring etc..This method is compared with the conventional method, either position excursion amount, angle drift amount or absolute error are all in above the average, this method obtains environment relatively multi information, and compared with the densest method, the experimental precision of this method still can reach use standard, and can directly run on CPU, real-time is good.
Description
Technical field
The present invention relates to Computer Image Processing and ground mapping field, be specially a kind of unmanned plane real-time online map raw
One-tenth method, by unmanned plane and the change of earth station's transferring content mode, it is achieved that unmanned plane real-time online map generates.
Background technology
The characteristic point on ground and boundary line (are such as utilized remote sensing, laser, ultrasonic by measurement means by traditional surveying and mapping technology
Ripple etc.) obtain figure and the positional information reflecting ground present situation.Traditional surveying and mapping technology is higher for sensor requirements, although its
Mapping precision is high, but high cost and constrain conventional measurement from information gathering to the time length obtaining a result required and be painted on some fields
Close the application of (such as higher to ageing requirement).
SLAM based on computer vision (Simultaneous Localization and Mapping), the most fixed
Position and map structuring, the theory important due to it and using value, be the most all the focus of scholar's research.But it is three-dimensional at present
The mode rebuild is that the image that unmanned aerial vehicle onboard camera photographs passes to ground computer by high definition figure, and ground computer is to shooting
To image carry out three-dimensional reconstruction.This method also exists the biggest deficiency:
1, image repeat in a large number, redundancy；
2, transmission speed is very big, causes transmission range little, and image is passed earth station back and is difficult to；
3, transmission signal is easily disturbed.
4, aircraft can not be made to do decision-making, automatically fly.
Summary of the invention
For solving the problem that prior art exists, the present invention proposes a kind of unmanned plane real-time online ground drawing generating method,
Unmanned plane realizes map real-time online rebuild, then pass, by high band wide data transmission link, the map rebuild back ground
Face.
The hardware of the present invention realizes being broadly divided into two large divisions, and Part I is a day dead end, and main device is by microminiature
Computer is placed on unmanned plane, and unmanned plane directly carries USB camera.At unmanned plane in flight course, entered by USB camera
Data are directly carried out by row image information collecting to micro-minicomputer, the micro-minicomputer image information to being collected
The image processing software that map reconstruction is micro-minicomputer by flight controller in flight course provides GPS, and by it
Pass to DDL figure to pass；Part II is ground surface end, and the information transmission of sky dead end and ground surface end is mainly passed by DDL figure and realizes.
During whole, micro-minicomputer and flight control system utilize serial ports to carry out information transmission.Mainly turn with USB
The GPS information flying control is passed to micro-minicomputer by UART.The information of flight control system and micro-minicomputer can be real-time pass through
DDL figure is passed to the DDL figure on ground and is passed.
Holding on high, the data message collected to be processed to be obtained from figure state and three-dimensional in real time by unmanned plane
The map of environment, is mainly realized by following steps:
Step 1: gather image
Unmanned aerial vehicle onboard collected by camera is to a series of images, and image passes to the micro-minicomputer of UAV system；
Step 2: the first two field picture that camera is obtained by the micro-minicomputer of UAV system carries out process and initialized
Map:
Step 2.1: go distortion to process the first two field picture, obtains the first two field picture after distortion；
Step 2.2: the first two field picture after going distortion is carried out degree of depth initialization: according to the shade of gray threshold value set,
Screen away in the first two field picture after distortion shade of gray more than the pixel of shade of gray threshold value, and give and being filtered out
The random depth value of pixel；
Step 2.3: the pixel back projection giving depth value in step 2.2 is returned three according to unmanned aerial vehicle onboard camera parameter
Dimension environment, the map after being initialized；
Step 2.4: the first two field picture after going distortion is set to key frame；
Step 3: the i-th two field picture obtained unmanned aerial vehicle onboard camera in real time is handled as follows, i=2,3,4 ...:
Step 3.1: go distortion to process the i-th two field picture, obtains the i-th two field picture after distortion；
Step 3.2: on the basis of current key frame, carries out the image alignment behaviour of the i-th two field picture after distortion and benchmark
Make, obtain the pose change to current key frame of i-th frame；
Step 3.3: according to the camera attitude that current key frame is corresponding, and the i-th frame that step 3.2 obtains is to current key
The pose change of frame, obtains camera attitude corresponding to the i-th frame and camera position under local coordinate；
Step 3.4: according to the shade of gray threshold value set, screens away shade of gray in the i-th two field picture after distortion big
In the pixel of shade of gray threshold value, and the camera attitude that the i-th frame of obtaining with step 3.3 according to camera parameter is corresponding, will sieve
Three-dimensional environment returns in the pixel back projection selected, and obtains filtered out pixel depth value；And by filtered out with deeply
The pixel of angle value adds in map；
Step 3.5: if the i-th frame of obtaining of step 3.2 changes more than the pose change set to the pose of current key frame
Threshold value, then replace current key frame as new key frame with the i-th frame.
Step 4: after the image procossing obtained in real time when the unmanned aerial vehicle onboard camera of default frame number completes, UAV system micro-
The map of generation is passed to ground surface end by the DDL figure biography of UAV system and is shown by minicomputer.
Further preferred version, described a kind of unmanned plane real-time online ground drawing generating method, it is characterised in that:
Extract and store the characteristic point in each key frame images；
If image alignment operation cannot realize in step 3.2, the most unsuccessfully rebuild:
Extract the characteristic point of the current frame image after going distortion, by each key frame of the characteristic point of present frame Yu storage
Characteristic point in image is mated, and finds the most key frame of successful match feature point number, if in this key frame successful
Join the number of characteristic point to account for the ratio of characteristic point sum in this key frame and be not more than 40%, then using present frame as the first frame, return
Return step 2；Otherwise using this key frame as on the basis of, carry out the image alignment behaviour of current frame image and benchmark after distortion
Make, obtain the present frame pose change to current key frame；
According to the camera attitude that benchmark is corresponding, and present frame is to the pose change of current key frame, obtains present frame pair
The camera attitude answered；
According to the shade of gray threshold value set, screen away shade of gray in the current frame image after distortion terraced more than gray scale
The pixel of degree threshold value, and according to the camera parameter camera attitude corresponding with present frame, the pixel back projection filtered out is returned
Three-dimensional environment, obtains filtered out pixel depth value；And the pixel with depth value filtered out is added map
In；Then continue to carry out according to step 3.
Further preferred version, described a kind of unmanned plane real-time online ground drawing generating method, it is characterised in that: use fast
Speed angular-point detection method extracts characteristic point.
Further preferred version, described a kind of unmanned plane real-time online ground drawing generating method, it is characterised in that: step
In 3.4, during the pixel with depth value filtered out is added map, if after a certain pixel back projection,
In three-dimensional neighborhood of a point corresponding in map, there are map three-dimensional point, then by after this pixel back projection in map right
Already present map three-dimensional point in the three-dimensional point answered, and three-dimensional point neighborhood is removed, and by after this pixel back projection on ground
Three-dimensional point corresponding in figure, adds in map with the weighted average point of the already present map three-dimensional point in three-dimensional point neighborhood.
Further preferred version, described a kind of unmanned plane real-time online ground drawing generating method, it is characterised in that: step
In 3.5, if the i-th frame of obtaining of step 3.2 changes more than the pose change threshold set to the pose of current key frame, and i-th
Frame is not less than 15 frames with the frame number difference of current key frame, then replace current key frame as new key frame with the i-th frame.
Further preferred version, described a kind of unmanned plane real-time online ground drawing generating method, it is characterised in that: step
Image alignment operation employing procedure below in 3.2:
First set i-th frame initial value to the pose change of current key frame；Pose according to the i-th frame to current key frame
Change, by the shade of gray that filters out in current key frame more than the pixel back projection of shade of gray threshold value to three-dimensional environment,
Project to, the i-th two field picture after distortion, obtain subpoint from three-dimensional environment again；And on the i-th two field picture after going distortion
Finding, the shade of gray filtered out in current key frame is more than the corresponding point of the pixel of shade of gray threshold value；Calculate subpoint
Shading value residual sum with corresponding point；Iteration changes the i-th frame and changes to the pose of current key frame, makes shading value residual sum
Little.
Further preferred version, described a kind of unmanned plane real-time online ground drawing generating method, it is characterised in that: use the
I-1 frame changes, to the pose of current key frame, the initial value that the pose as the i-th frame to current key frame changes.
Further preferred version, described a kind of unmanned plane real-time online ground drawing generating method, it is characterised in that: step
In 3.4, after obtaining filtered out pixel depth value；Use figure optimization method to camera corresponding to the i-th frame in local coordinate
Under position, and the pixel position with depth value filtered out is optimized, after optimizing with depth value
Pixel adds in map.
Further preferred version, described a kind of unmanned plane real-time online ground drawing generating method, it is characterised in that: will rebuild
After map be transformed under world coordinate system:
In unmanned plane real-time map process of reconstruction, inscribing when obtaining each frame by satellite positioning signal, unmanned plane exists
Trace information X under world coordinate systemn, what n represented is n-th frame；And in unmanned plane real-time map process of reconstruction, obtain each
Camera corresponding to frame position x under local coordinate systemn；Pass through majorized function
Obtain majorized function and take the transformation matrix δ that minima is corresponding, during wherein N is unmanned plane real-time map process of reconstruction
Totalframes, T (xn, δ) and represent the projection transform function being tied to world coordinate system from local coordinate, δ is for be tied to generation from local coordinate
The transformation matrix of boundary's coordinate system；According to the projection transform function corresponding for transformation matrix δ obtained, map reconstruction obtained is changed
Under world coordinate system.
Further preferred version, described a kind of unmanned plane real-time online ground drawing generating method, it is characterised in that: work as satellite
When framing signal frequency is less than frame frequency, moment t is gathered for each satellite positioning signaln, obtain unmanned plane at world coordinates
Trace information X under Xin；And with gathering moment tnCamera corresponding to the most each frame position interpolation under local coordinate system obtains
To gathering moment tnLower camera position x under local coordinate systemn；Pass through majorized function
Obtaining majorized function and take the transformation matrix δ that minima is corresponding, wherein N is that always gathering of satellite positioning signal is counted, T
(xn, δ) and represent the projection transform function being tied to world coordinate system from local coordinate, δ is for be tied to world coordinate system from local coordinate
Transformation matrix；According to the projection transform function corresponding for transformation matrix δ obtained, map reconstruction obtained is transformed into the world and sits
Under mark system.
Beneficial effect
The method that the present invention proposes is compared with many existing methods, and either position excursion amount, angle drift amount are the most absolutely
To error all in above the average, this method obtains environment relatively multi information, and compared with the densest method, the reality of this method
Testing precision and still can reach use standard, and can directly run on CPU, need not require GPU, real-time is good.
And, the present invention uses real-time online map generating mode on unmanned plane, image information transmission to be more easy to realize, subtract
The little storage of data volume, and the quantity of information of transmission, specifically: 1, repeat the most in a large number, the picture transfer of redundancy returns
Come；2, transmission quantity of information is little so that transfer rate is little, long transmission distance；3, map real-time reconstruction is carried out when unmanned plane during flying,
Aircraft can be allowed to make a policy, automatically fly, may be used for obstacle, path planning etc..
The additional aspect of the present invention and advantage will part be given in the following description, and part will become from the following description
Obtain substantially, or recognized by the practice of the present invention.
Accompanying drawing explanation
Above-mentioned and/or the additional aspect of the present invention and advantage are from combining the accompanying drawings below description to embodiment and will become
Substantially with easy to understand, wherein:
The projection of Fig. 1: trigonometric ratio determines pixel depth schematic diagram；
Fig. 2: through image alignment schemes schematic diagram；
Fig. 3: unsuccessfully rebuild schematic diagram；
Fig. 4: figure optimization method schematic diagram；
Fig. 5: matching satellite navigation data schematic diagram；
The frame original image that Fig. 6: camera obtains；
The three-dimensional point cloud atlas of Fig. 7: real-time reconstruction；
Fig. 8: use the map of the online real-time reconstruction of unmanned plane；
Fig. 9: spliced map；
Figure 10: unmanned plane online real-time map reconstructing system sky dead end schematic diagram；
Figure 11: unmanned plane online real-time map reconstructing system ground surface end schematic diagram.
Detailed description of the invention
Embodiments of the invention are described below in detail, and described embodiment is exemplary, it is intended to be used for explaining the present invention, and
It is not considered as limiting the invention.
The hardware of the present embodiment realizes being broadly divided into two large divisions, and Part I is held on high and carried out, as shown in Figure 10, the
Two parts are carried out in ground surface end, as shown in figure 11.
During whole, airborne micro-minicomputer and flight control system utilize USB-UART to carry out information transmission.Mainly
It is, with USB-UART, the GPS information flying control is passed to airborne micro-minicomputer.Flight control system and airborne micro-minicomputer
Information can be real-time pass to ground surface end by airborne DDL figure biography and show.
The specific embodiment step of the present embodiment is as follows:
Holding on high of task is to carry out the reconstruction of real-time map in flight course:
Step 1: gather image
Unmanned aerial vehicle onboard collected by camera is to a series of images, and is passed to unmanned by USB2.0 or USB3.0 by image
Airborne micro-minicomputer；Such transmission means than remote transmission fast a lot.
Step 2: the first two field picture that camera is obtained by the micro-minicomputer of UAV system carries out process and initialized
Map:
Step 2.1: go distortion to process the first two field picture by the nominal data obtained in advance, after obtaining distortion
First two field picture, for subsequent treatment.
Step 2.2: the first two field picture after going distortion is carried out degree of depth initialization: according to the shade of gray threshold value set,
Screen away in the first two field picture after distortion shade of gray more than the pixel of shade of gray threshold value, and give and being filtered out
The random depth value of pixel；Such random process can't affect reconstruction precision below, because by the process of tens frames
After, depth map can gradually level off to a model accurately.
Step 2.3: three-dimensional environment is returned in the pixel back projection giving depth value in step 2.2 according to camera parameter,
Map after initialization.
Step 2.4: the first two field picture after going distortion is set to key frame；Extract and store the feature in key frame
Point.Method for feature point detection has: 1. SIFT, 2. SURF, 3. fast-corner (Fast Corner Detection: by Gauss mistake
Filter, then Corner Detection) etc., owing to it is contemplated that be applied to real-time build environment three-dimensional map, program focuses on real-time, and front
Although two kinds of precision are high, effective, but its required time is the longest, is not suitable for the situation of real time execution, therefore this step
And subsequent step all uses fast-corner to detect characteristic point.
Step 3: the i-th two field picture of acquisition real-time to camera is handled as follows, i=2,3,4 ...:
Step 3.1: go distortion to process the i-th two field picture, obtains the i-th two field picture after distortion.
Step 3.2: after the i-th frame image data loads, system (i.e. follows the tracks of basis) on the basis of current key frame,
Carry out the image alignment operation of the i-th two field picture after distortion and benchmark, obtain the pose change to current key frame of i-th frame.
Owing to frame per second is higher, the difference (time, space) of consecutive frame is not very big, it is assumed herein that when two frames are the least
Between interval in, the shading value of same pixel does not changes (varying less, ignore).The most directly by comparing two width
The shading value of image, the pose obtaining two interframe by minimizing luminosity error changes, and i.e. completes image alignment and operates:
First set i-th frame initial value to the pose change of current key frame.
Pose change according to the i-th frame to current key frame, is more than ash by the shade of gray filtered out in current key frame
The pixel back projection of degree Grads threshold is to three-dimensional environment, then projects to the i-th two field picture after distortion from three-dimensional environment,
To subpoint；And find on the i-th two field picture after going distortion, the shade of gray filtered out in current key frame is more than gray scale ladder
The corresponding point of the pixel of degree threshold value；Calculate the shading value residual sum of subpoint and corresponding point；Iteration changes the i-th frame and closes to current
The pose change of key frame, makes shading value residual sum minimum.
Owing to frame per second is higher, the pose variable quantity that result in two interframe is that approximation is constant, therefore in a zonule
The i-th-1 frame can be used to change, to the pose of current key frame, the initial value that the pose as the i-th frame to current key frame changes.
The parameter matrix utilizing camera will be to by trigonometric ratio, calculate current in pixel back projection to three-dimensional environment
Frame (the i.e. i-th frame) and the attitudes vibration of current key frame, i.e. SE (3) converts: SE (3) is the matrix of 4 × 4, represents position
With attitudes vibration (in camera projection equation, being also called outer ginseng matrix):
This matrix is broadly divided into two large divisions, wherein from a00To a22For SO (3), represent attitude (angle in three dimensions
Degree) change, from T0To T2Locative change, i.e. (x, y, variable quantity z).SIM3 is added Scale parameter group by SE (3)
Becoming, SE (3) can be become SIM3 by SO (3) * s:
S represents Scale parameter, for affine transformation.
Step 3.3: according to the camera attitude that current key frame is corresponding, and the i-th frame that step 3.2 obtains is to current key
The pose change of frame, obtains camera attitude corresponding to the i-th frame and camera position under local coordinate.
Step 3.4: according to the shade of gray threshold value set, screens away shade of gray in the i-th two field picture after distortion big
In the pixel of shade of gray threshold value, and the camera attitude that the i-th frame of obtaining with step 3.3 according to camera parameter is corresponding, will sieve
Three-dimensional environment returns in the pixel back projection selected, and obtains filtered out pixel depth value；And by filtered out with deeply
The pixel of angle value adds in map.
Owing to before and after current whole system, dependency is strong, in order to reduce the shadow to final result of the error from sensor
Ring, correct the pose of each step, use here figure optimization method to camera corresponding to the i-th frame position under local coordinate, with
And the pixel position with depth value filtered out is optimized, the pixel with depth value after optimizing adds ground
In figure.Known figure optimization method formula is:
xkRepresent previously defined node (it can be appreciated that state), ZkRepresent limit (it can be appreciated that constraint), ekTable
Show that these nodes meet the state of constraint (without noise and ek=0), ΩkRepresent introduce information matrix, and constraint put
Reliability, if error is big, the confidence level of its correspondence is the least.After having defined variable, what we were to be done makes whole error letter exactly
Number is minimized, to reach the purpose of global optimum.Our three-dimensional position by point map and position of unmanned plane in the method
Appearance is defined as node, and the change of the projection relation from image to point map and the SE (3) between adjacent two frames is defined as limit, letter
Breath matrix comprises two aspects: point map by key frame observation frequency and the shade of gray at image midpoint.
In this external step 3.4, during the pixel with depth value filtered out is added map, if a certain
After pixel back projection, in three-dimensional neighborhood of a point (a certain little value of setting) corresponding in map, there are map three-dimensional
Point, then by three-dimensional point corresponding in map after this pixel back projection, and the already present map three in three-dimensional point neighborhood
Dimension point is removed, and by three-dimensional point corresponding in map after this pixel back projection, already present with in three-dimensional point neighborhood
The weighted average point of figure three-dimensional point adds in map.
Step 3.5: if the i-th frame of obtaining of step 3.2 changes more than the pose change set to the pose of current key frame
Threshold value, then replace current key frame as new key frame with the i-th frame.The present embodiment, in order to improve arithmetic speed, reduces storage
Data volume, if requiring here, the i-th frame of obtaining of step 3.2 changes to the pose change of current key frame more than the pose set
Threshold value, and the frame number difference of the i-th frame and current key frame is not less than 15 frames, then replace current key frame as new pass with the i-th frame
Key frame.
Setting up of key frame is owing to its pose having changes greatly relative to previous key frame, its detect three
Key frame before dimension environmental information is compared to has relatively big difference, therefore it is set to a scale, is used for extension globally
Scheme and detect whether follow-up frame has bigger pose to change.
During following the tracks of, if producing " frame losing " phenomenon, (possible reason has: camera moves too fast, causes present frame
Excessive with " gap " of current key frame, it is impossible to be tracked on current key frame, will if the most not carrying out process
Closely " contact ", all working before allowing loses meaning to cause neither one between front and back create two maps), so sentencing
If image alignment operation cannot realize in disconnected step 3.2, the most unsuccessfully rebuild:
Extract the characteristic point of the current frame image after going distortion, by each key frame of the characteristic point of present frame Yu storage
Characteristic point in image is mated, and finds the most key frame of successful match feature point number, if in this key frame successful
Join the number of characteristic point to account for the ratio of characteristic point sum in this key frame and be not more than 40%, then using present frame as the first frame, return
Return step 2；Otherwise using this key frame as on the basis of, carry out the image alignment behaviour of current frame image and benchmark after distortion
Make, obtain the present frame pose change to current key frame；
According to the camera attitude that benchmark is corresponding, and present frame is to the pose change of current key frame, obtains present frame pair
The camera attitude answered；
According to the shade of gray threshold value set, screen away shade of gray in the current frame image after distortion terraced more than gray scale
The pixel of degree threshold value, and according to the camera parameter camera attitude corresponding with present frame, the pixel back projection filtered out is returned
Three-dimensional environment, obtains filtered out pixel depth value；And the pixel with depth value filtered out is added map
In；Then continue to carry out according to step 3.
Owing to above-mentioned reconstruction three-dimensional environment out is based under local coordinate system, with true three-dimension environment in unification
Do not mate on yardstick, in order to preferably map is rebuild in application, so being fitted below based on satellite positioning signal, will weight
The three-dimensional environment building out matches in true three-dimension environment under unified yardstick.
In unmanned plane real-time map process of reconstruction, inscribing when obtaining each frame by satellite positioning signal, unmanned plane exists
Trace information X under world coordinate systemn, what n represented is n-th frame；And in unmanned plane real-time map process of reconstruction, obtain each
Camera corresponding to frame position x under local coordinate systemn；Pass through majorized function
Obtain majorized function and take the transformation matrix δ that minima is corresponding, during wherein N is unmanned plane real-time map process of reconstruction
Totalframes, T (xn, δ) and represent the projection transform function being tied to world coordinate system from local coordinate, δ is for be tied to generation from local coordinate
The transformation matrix of boundary's coordinate system；According to the projection transform function corresponding for transformation matrix δ obtained, map reconstruction obtained is changed
Under world coordinate system.
It addition, satellite positioning signal frequency is often below frame frequency, and the collection moment of satellite positioning signal adopts with image
Collection moment not close alignment, now, gathers moment t for each satellite positioning signaln, obtain unmanned plane at world coordinate system
Under trace information Xn；And with gathering moment tnCamera corresponding to the most each frame position interpolation under local coordinate system obtains
Gather moment tnLower camera position x under local coordinate systemn；Pass through majorized function
Obtaining majorized function and take the transformation matrix δ that minima is corresponding, wherein N is that always gathering of satellite positioning signal is counted, T
(xn, δ) and represent the projection transform function being tied to world coordinate system from local coordinate, δ is for be tied to world coordinate system from local coordinate
Transformation matrix；According to the projection transform function corresponding for transformation matrix δ obtained, map reconstruction obtained is transformed into the world and sits
Under mark system.
Step 4: after the image procossing obtained in real time when the unmanned aerial vehicle onboard camera of default frame number completes, UAV system micro-
The map of generation is passed to ground surface end by the DDL figure biography of UAV system and is shown by minicomputer.For example, it is possible to be set in airborne
Micro-minicomputer has often processed the 10th frame, the 20th frame ... wait after presetting two field picture, is passed through by the map generated in real time
The DDL figure biography of UAV system is passed to ground surface end and is shown.
The result of reconstruction is given below, including in real time and the test result of off-line data, and with the regarding of currently some main flows
Feel that SLAM method is made comparisons, evaluate systematic function.The most all of test all mint17 (based on Ubuntu14.04,
Carry out on 64bit), 8G RAM, without GPU, dominant frequency 3.50GHz.
Table 1:RGB-D benchmark results compares
Table 2: definitely tracking error test result compares (unit cm)
| This patent method | [3] | [4] | [5] | [6] | |
| fr2-xyz | 1.46 | 3.78 | 24.29 | 1.21 | 2.7 |
| Sim | 0.36 | 2.31 | - | 0.14 | - |
| Sim-desk | 0.05 | 1.54 | - | 0.26 | - |
| fr2-desk | 4.53 | 13.51 | X | 1.78 | 9.6 |
Note: [1] method is: Dense RGB-D Odometry；[2] method is: PTAM (keypoint-based)；[3]
Method is: semi-densemono-VO；[4] method is: keypoint-based mono-SLAM；[5] method is: Direct
RGB-D SLAM；[6] method is: keypoint-based RGB-D SLAM；Represent the data not finding the method, X table
Show and follow the trail of unsuccessfully (Tracking Lost)；By the data made comparisons from Computer Vision Group；Can by comparing
To find: the method for this patent either position excursion amount, angle drift amount or absolute error, all in above the average, are compared
Relatively other method, our method obtains environment relatively multi information, and simultaneously with the densest method, our experimental precision still can
Enough reach use standard, and can directly run on CPU, GPU need not be required.
In addition this method demonstrates real-time map structure on unmanned plane, as shown in Figure 8 and Figure 9.Unmanned plane in native system
On be equipped with a high-definition camera and micro-minicomputer, utilizing micro-minicomputer to run this method can calculate in real time
To sparse some cloud of the earth, utilize the aircraft pose resolved, some cloud, characteristic point to be spliced in real time by the image of shooting and obtain two-dimensionally
Figure.It is able to demonstrate that the inventive method is capable of real-time reconstruction map by this experiment.
Although above it has been shown and described that embodiments of the invention, it is to be understood that above-described embodiment is example
Property, it is impossible to be interpreted as limitation of the present invention, those of ordinary skill in the art is without departing from the principle of the present invention and objective
In the case of above-described embodiment can be changed within the scope of the invention, revise, replace and modification.
Claims (10)
1. a unmanned plane real-time online ground drawing generating method, it is characterised in that: comprise the following steps:
Step 1: gather image
Unmanned aerial vehicle onboard collected by camera is to a series of images, and image passes to the micro-minicomputer of UAV system；
Step 2: the first two field picture that camera is obtained by the micro-minicomputer of UAV system carry out process obtain initialize map:
Step 2.1: go distortion to process the first two field picture, obtains the first two field picture after distortion；
Step 2.2: the first two field picture after going distortion is carried out degree of depth initialization: according to the shade of gray threshold value set, screening
Go out in the first two field picture after distortion shade of gray more than the pixel of shade of gray threshold value, and give filtered out pixel
The random depth value of point；
Step 2.3: three-dimensional ring is returned in the pixel back projection giving depth value in step 2.2 according to unmanned aerial vehicle onboard camera parameter
Border, the map after being initialized；
Step 2.4: the first two field picture after going distortion is set to key frame；
Step 3: the i-th two field picture obtained unmanned aerial vehicle onboard camera in real time is handled as follows, i=2,3,4 ...:
Step 3.1: go distortion to process the i-th two field picture, obtains the i-th two field picture after distortion；
Step 3.2: on the basis of current key frame, carries out the image alignment operation of the i-th two field picture after distortion and benchmark,
Change to the i-th frame to the pose of current key frame；
Step 3.3: according to the camera attitude that current key frame is corresponding, and the i-th frame that step 3.2 obtains is to current key frame
Pose changes, and obtains camera attitude corresponding to the i-th frame and camera position under local coordinate；
Step 3.4: according to the shade of gray threshold value set, screens away shade of gray in the i-th two field picture after distortion and is more than ash
The pixel of degree Grads threshold, and the camera attitude that the i-th frame of obtaining with step 3.3 according to camera parameter is corresponding, will filter out
Pixel back projection return three-dimensional environment, obtain filtered out pixel depth value；And by filtered out with depth value
Pixel add in map；
Step 3.5: if the i-th frame of obtaining of step 3.2 changes more than the pose change threshold set to the pose of current key frame,
Then replace current key frame as new key frame with the i-th frame；
Step 4: after the image procossing obtained in real time when the unmanned aerial vehicle onboard camera of default frame number completes, the microminiature of UAV system
The map of generation is passed to ground surface end by the DDL figure biography of UAV system and is shown by computer.
A kind of unmanned plane real-time online ground drawing generating method, it is characterised in that:
Extract and store the characteristic point in each key frame images；
If image alignment operation cannot realize in step 3.2, the most unsuccessfully rebuild:
Extract the characteristic point of the current frame image after going distortion, by each key frame images of the characteristic point of present frame Yu storage
In characteristic point mate, find the most key frame of successful match feature point number, if successful match is special in this key frame
Levy number a little to account for the ratio of the sum of characteristic point in this key frame and be not more than 40%, then using present frame as the first frame, return step
Rapid 2；Otherwise using this key frame as on the basis of, carry out the image alignment operation of current frame image and benchmark after distortion,
Change to present frame to the pose of current key frame；
According to the camera attitude that benchmark is corresponding, and present frame is to the pose change of current key frame, obtains present frame corresponding
Camera attitude；
According to the shade of gray threshold value set, screen away shade of gray in the current frame image after distortion and be more than shade of gray threshold
The pixel of value, and according to the camera parameter camera attitude corresponding with present frame, three-dimensional is returned by the pixel back projection filtered out
Environment, obtains filtered out pixel depth value；And the pixel with depth value filtered out is added in map；And
Rear continuation is carried out according to step 3.
A kind of unmanned plane real-time online ground drawing generating method, it is characterised in that: use Fast Corner
Detection method extracts characteristic point.
A kind of unmanned plane real-time online ground drawing generating method, it is characterised in that: in step 3.4, will
During the pixel with depth value filtered out adds map, if after a certain pixel back projection, right in map
In the three-dimensional neighborhood of a point answered, there are map three-dimensional point, then by three-dimensional corresponding in map after this pixel back projection
Already present map three-dimensional point in point, and three-dimensional point neighborhood is removed, and by after this pixel back projection in map corresponding
Three-dimensional point, add in map with the weighted average point of the already present map three-dimensional point in three-dimensional point neighborhood.
A kind of unmanned plane real-time online ground drawing generating method, it is characterised in that: in step 3.5, if
The i-th frame that step 3.2 obtains changes more than the pose change threshold set to the pose of current key frame, and the i-th frame is with current
The frame number difference of key frame is not less than 15 frames, then replace current key frame as new key frame with the i-th frame.
A kind of unmanned plane real-time online ground drawing generating method, it is characterised in that: in step 3.2
Image alignment operation employing procedure below:
First set i-th frame initial value to the pose change of current key frame；Pose according to the i-th frame to current key frame becomes
Change, by the shade of gray that filters out in current key frame more than the pixel back projection of shade of gray threshold value to three-dimensional environment, then
Project to, the i-th two field picture after distortion, obtain subpoint from three-dimensional environment；And look on the i-th two field picture after going distortion
Arriving, the shade of gray filtered out in current key frame is more than the corresponding point of the pixel of shade of gray threshold value；Calculate subpoint with
The shading value residual sum of corresponding point；Iteration changes the pose change to current key frame of i-th frame, makes shading value residual sum minimum.
A kind of unmanned plane real-time online ground drawing generating method, it is characterised in that: use the i-th-1 frame
Pose to current key frame changes the initial value that the pose as the i-th frame to current key frame changes.
A kind of unmanned plane real-time online ground drawing generating method, it is characterised in that: in step 3.4,
After the pixel depth value filtered out；Use figure optimization method to camera corresponding to the i-th frame position under local coordinate,
And the pixel position with depth value filtered out is optimized, the pixel with depth value after optimizing adds
In map.
9. according to a kind of unmanned plane real-time online ground drawing generating method described in claim 1 or 8, it is characterised in that: after rebuilding
Map be transformed under world coordinate system:
In unmanned plane real-time map process of reconstruction, inscribing when obtaining each frame by satellite positioning signal, unmanned plane is in the world
Trace information X under coordinate systemn, what n represented is n-th frame；And in unmanned plane real-time map process of reconstruction, obtain each frame pair
The camera answered position x under local coordinate systemn；Pass through majorized function
Obtain majorized function and take the transformation matrix δ that minima is corresponding, the total frame during wherein N is unmanned plane real-time map process of reconstruction
Number, T (xn, δ) and represent the projection transform function being tied to world coordinate system from local coordinate, δ sits for being tied to the world from local coordinate
The transformation matrix of mark system；According to the projection transform function corresponding for transformation matrix δ obtained, map reconstruction obtained is transformed into generation
Under boundary's coordinate system.
10. according to a kind of unmanned plane real-time online ground drawing generating method described in claim 1 or 8, it is characterised in that: work as satellite
When position signal frequency is less than frame frequency, moment t is gathered for each satellite positioning signaln, obtain unmanned plane at world coordinate system
Under trace information Xn；And with gathering moment tnCamera corresponding to the most each frame position interpolation under local coordinate system obtains
Gather moment tnLower camera position x under local coordinate systemn；Pass through majorized function
Obtaining majorized function and take the transformation matrix δ that minima is corresponding, wherein N is that always gathering of satellite positioning signal is counted, T (xn,
δ) expression is tied to the projection transform function of world coordinate system from local coordinate, and δ is the change being tied to world coordinate system from local coordinate
Change matrix；According to the projection transform function corresponding for transformation matrix δ obtained, map reconstruction obtained is transformed into world coordinate system
Under.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610373105.0A CN106097304B (en) | 2016-05-31 | 2016-05-31 | A kind of unmanned plane real-time online ground drawing generating method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610373105.0A CN106097304B (en) | 2016-05-31 | 2016-05-31 | A kind of unmanned plane real-time online ground drawing generating method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106097304A true CN106097304A (en) | 2016-11-09 |
| CN106097304B CN106097304B (en) | 2019-04-23 |
Family
ID=57229505
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610373105.0A Active CN106097304B (en) | 2016-05-31 | 2016-05-31 | A kind of unmanned plane real-time online ground drawing generating method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106097304B (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108389477A (en) * | 2018-03-05 | 2018-08-10 | 广州星唯信息科技有限公司 | A kind of correction guidance method for driving training field training |
| CN108496134A (en) * | 2017-05-31 | 2018-09-04 | 深圳市大疆创新科技有限公司 | Unmanned plane makes a return voyage paths planning method and device |
| CN108513648A (en) * | 2017-06-19 | 2018-09-07 | 深圳市大疆创新科技有限公司 | Map constructing method, map structuring system, unmanned vehicle and control terminal |
| CN108648270A (en) * | 2018-05-12 | 2018-10-12 | 西北工业大学 | Unmanned plane real-time three-dimensional scene reconstruction method based on EG-SLAM |
| CN108812599A (en) * | 2018-06-13 | 2018-11-16 | 仲恺农业工程学院 | A kind of unmanned plane plant protection monitoring system and method for manual manipulation |
| CN108846084A (en) * | 2018-06-11 | 2018-11-20 | 成都山河空间信息技术有限公司 | A kind of generation system and method for live-action map |
| WO2019100214A1 (en) * | 2017-11-21 | 2019-05-31 | 深圳市大疆创新科技有限公司 | Method, device, and unmanned aerial vehicle for generating output image |
| CN110047142A (en) * | 2019-03-19 | 2019-07-23 | 中国科学院深圳先进技术研究院 | No-manned plane three-dimensional map constructing method, device, computer equipment and storage medium |
| CN110285822A (en) * | 2019-07-01 | 2019-09-27 | 东莞理工学院 | Unmanned plane builds the fusion application system and method for nomography Yu unmanned vehicle navigation algorithm |
| CN110648398A (en) * | 2019-08-07 | 2020-01-03 | 武汉九州位讯科技有限公司 | Real-time ortho image generation method and system based on unmanned aerial vehicle aerial data |
| CN110779496A (en) * | 2018-07-30 | 2020-02-11 | 阿里巴巴集团控股有限公司 | Three-dimensional map construction system, method, device and storage medium |
| CN110858414A (en) * | 2018-08-13 | 2020-03-03 | 北京嘀嘀无限科技发展有限公司 | Image processing method and device, readable storage medium and augmented reality system |
| WO2020198963A1 (en) * | 2019-03-29 | 2020-10-08 | 深圳市大疆创新科技有限公司 | Data processing method and apparatus related to photographing device, and image processing device |
| CN111798373A (en) * | 2020-06-11 | 2020-10-20 | 西安视野慧图智能科技有限公司 | Rapid unmanned aerial vehicle image stitching method based on local plane hypothesis and six-degree-of-freedom pose optimization |
| CN113190012A (en) * | 2021-05-10 | 2021-07-30 | 山东大学 | Robot task autonomous planning method and system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103247075A (en) * | 2013-05-13 | 2013-08-14 | 北京工业大学 | Variational mechanism-based indoor scene three-dimensional reconstruction method |
| CN103822635A (en) * | 2014-03-05 | 2014-05-28 | 北京航空航天大学 | Visual information based real-time calculation method of spatial position of flying unmanned aircraft |
| US20140225985A1 (en) * | 2012-10-17 | 2014-08-14 | DotProduct LLC | Handheld portable optical scanner and method of using |
| CN104200523A (en) * | 2014-09-11 | 2014-12-10 | 中国科学院自动化研究所 | Large-scale scene three-dimensional reconstruction method for fusion of additional information |
| CN104537709A (en) * | 2014-12-15 | 2015-04-22 | 西北工业大学 | Real-time three-dimensional reconstruction key frame determination method based on position and orientation changes |
| CN105571588A (en) * | 2016-03-10 | 2016-05-11 | 赛度科技（北京）有限责任公司 | Method for building three-dimensional aerial airway map of unmanned aerial vehicle and displaying airway of three-dimensional aerial airway map |
-
2016
- 2016-05-31 CN CN201610373105.0A patent/CN106097304B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140225985A1 (en) * | 2012-10-17 | 2014-08-14 | DotProduct LLC | Handheld portable optical scanner and method of using |
| CN103247075A (en) * | 2013-05-13 | 2013-08-14 | 北京工业大学 | Variational mechanism-based indoor scene three-dimensional reconstruction method |
| CN103822635A (en) * | 2014-03-05 | 2014-05-28 | 北京航空航天大学 | Visual information based real-time calculation method of spatial position of flying unmanned aircraft |
| CN104200523A (en) * | 2014-09-11 | 2014-12-10 | 中国科学院自动化研究所 | Large-scale scene three-dimensional reconstruction method for fusion of additional information |
| CN104537709A (en) * | 2014-12-15 | 2015-04-22 | 西北工业大学 | Real-time three-dimensional reconstruction key frame determination method based on position and orientation changes |
| CN105571588A (en) * | 2016-03-10 | 2016-05-11 | 赛度科技（北京）有限责任公司 | Method for building three-dimensional aerial airway map of unmanned aerial vehicle and displaying airway of three-dimensional aerial airway map |
Non-Patent Citations (5)
| Title |
|---|
| DOMINIK NEUMANN 等: "Real-time RGB-D Mapping and 3-D Modeling on the GPU using the Random Ball Cover Data Structure", 《2011 IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION WORKSHOPS》 * |
| FELIX ENDRES 等: "3-D Mapping With an RGB-D Camera", 《IEEE TRANSACTIONS ON ROBOTICS》 * |
| XUEPING ZHU 等: "Model of Collaborative UAV Swarm Toward Coordination and Control Mechanisms Study", 《ICCS 2015 INTERNATIONAL CONFERENCE ON COMPUTATIONAL SCIENCE》 * |
| 张臻炜 等: "一种基于计算机视觉的无人机实时三维重建方法", 《机械与电子》 * |
| 黄金鑫 等: "一种改进的未知环境无人机三维地图实时创建方法", 《机械与电子》 * |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108496134A (en) * | 2017-05-31 | 2018-09-04 | 深圳市大疆创新科技有限公司 | Unmanned plane makes a return voyage paths planning method and device |
| WO2018232576A1 (en) * | 2017-06-19 | 2018-12-27 | 深圳市大疆创新科技有限公司 | Map building method, map building system, unmanned aerial vehicle, and control terminal |
| CN108513648A (en) * | 2017-06-19 | 2018-09-07 | 深圳市大疆创新科技有限公司 | Map constructing method, map structuring system, unmanned vehicle and control terminal |
| CN110073403A (en) * | 2017-11-21 | 2019-07-30 | 深圳市大疆创新科技有限公司 | Image output generation method, equipment and unmanned plane |
| WO2019100214A1 (en) * | 2017-11-21 | 2019-05-31 | 深圳市大疆创新科技有限公司 | Method, device, and unmanned aerial vehicle for generating output image |
| CN108389477A (en) * | 2018-03-05 | 2018-08-10 | 广州星唯信息科技有限公司 | A kind of correction guidance method for driving training field training |
| CN108648270A (en) * | 2018-05-12 | 2018-10-12 | 西北工业大学 | Unmanned plane real-time three-dimensional scene reconstruction method based on EG-SLAM |
| CN108648270B (en) * | 2018-05-12 | 2022-04-19 | 西北工业大学 | Unmanned aerial vehicle real-time three-dimensional scene reconstruction method capable of realizing real-time synchronous positioning and map construction |
| CN108846084A (en) * | 2018-06-11 | 2018-11-20 | 成都山河空间信息技术有限公司 | A kind of generation system and method for live-action map |
| CN108812599A (en) * | 2018-06-13 | 2018-11-16 | 仲恺农业工程学院 | A kind of unmanned plane plant protection monitoring system and method for manual manipulation |
| CN108812599B (en) * | 2018-06-13 | 2021-08-27 | 仲恺农业工程学院 | Unmanned aerial vehicle plant protection monitoring system and method for manual control |
| CN110779496B (en) * | 2018-07-30 | 2022-03-22 | 斑马智行网络(香港)有限公司 | Three-dimensional map construction system, method, device and storage medium |
| CN110779496A (en) * | 2018-07-30 | 2020-02-11 | 阿里巴巴集团控股有限公司 | Three-dimensional map construction system, method, device and storage medium |
| CN110858414A (en) * | 2018-08-13 | 2020-03-03 | 北京嘀嘀无限科技发展有限公司 | Image processing method and device, readable storage medium and augmented reality system |
| CN110047142A (en) * | 2019-03-19 | 2019-07-23 | 中国科学院深圳先进技术研究院 | No-manned plane three-dimensional map constructing method, device, computer equipment and storage medium |
| WO2020198963A1 (en) * | 2019-03-29 | 2020-10-08 | 深圳市大疆创新科技有限公司 | Data processing method and apparatus related to photographing device, and image processing device |
| CN110285822A (en) * | 2019-07-01 | 2019-09-27 | 东莞理工学院 | Unmanned plane builds the fusion application system and method for nomography Yu unmanned vehicle navigation algorithm |
| CN110648398B (en) * | 2019-08-07 | 2020-09-11 | 武汉九州位讯科技有限公司 | Real-time ortho image generation method and system based on unmanned aerial vehicle aerial data |
| CN110648398A (en) * | 2019-08-07 | 2020-01-03 | 武汉九州位讯科技有限公司 | Real-time ortho image generation method and system based on unmanned aerial vehicle aerial data |
| CN111798373A (en) * | 2020-06-11 | 2020-10-20 | 西安视野慧图智能科技有限公司 | Rapid unmanned aerial vehicle image stitching method based on local plane hypothesis and six-degree-of-freedom pose optimization |
| CN113190012A (en) * | 2021-05-10 | 2021-07-30 | 山东大学 | Robot task autonomous planning method and system |
| CN113190012B (en) * | 2021-05-10 | 2022-08-12 | 山东大学 | Robot task autonomous planning method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106097304B (en) | 2019-04-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106097304A (en) | A kind of unmanned plane real-time online ground drawing generating method | |
| CN105678754B (en) | A kind of unmanned plane real-time map method for reconstructing | |
| CN105865454B (en) | A kind of Navigation of Pilotless Aircraft method generated based on real-time online map | |
| CN103954283A (en) | Scene matching/visual odometry-based inertial integrated navigation method | |
| CN102435188B (en) | Monocular vision/inertia autonomous navigation method for indoor environment | |
| CN106595659A (en) | Map merging method of unmanned aerial vehicle visual SLAM under city complex environment | |
| CN106940704A (en) | A kind of localization method and device based on grating map | |
| CN108648270A (en) | Unmanned plane real-time three-dimensional scene reconstruction method based on EG-SLAM | |
| Lin et al. | A decentralized framework for simultaneous calibration, localization and mapping with multiple LiDARs | |
| CN110726406A (en) | Improved nonlinear optimization monocular inertial navigation SLAM method | |
| Sanfourche et al. | Perception for UAV: Vision-Based Navigation and Environment Modeling. | |
| CN113625774B (en) | Local map matching and end-to-end ranging multi-unmanned aerial vehicle co-location system and method | |
| CN108520559A (en) | A method of the unmanned plane location navigation based on binocular vision | |
| CN102607532B (en) | Quick low-level image matching method by utilizing flight control data | |
| Zhang et al. | Aerial orthoimage generation for UAV remote sensing | |
| Ye et al. | Monocular direct sparse localization in a prior 3d surfel map | |
| CN111536970B (en) | Infrared inertial integrated navigation method for low-visibility large-scale scene | |
| Lin et al. | R $^ 3$ LIVE++: A Robust, Real-time, Radiance reconstruction package with a tightly-coupled LiDAR-Inertial-Visual state Estimator | |
| Liu | A robust and efficient lidar-inertial-visual fused simultaneous localization and mapping system with loop closure | |
| CN112556719A (en) | Visual inertial odometer implementation method based on CNN-EKF | |
| Warren et al. | High altitude stereo visual odometry | |
| Jung et al. | Photometric visual-inertial navigation with uncertainty-aware ensembles | |
| Daftry et al. | Semi-dense visual odometry for monocular navigation in cluttered environment | |
| Zheng et al. | Dual LIDAR online calibration and mapping and perception system | |
| CN114419259B (en) | Visual positioning method and system based on physical model imaging simulation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |