CN110021041A - Unmanned scene progressive mesh structural remodeling method based on binocular camera - Google Patents
Unmanned scene progressive mesh structural remodeling method based on binocular camera Download PDFInfo
- Publication number
- CN110021041A CN110021041A CN201910156872.XA CN201910156872A CN110021041A CN 110021041 A CN110021041 A CN 110021041A CN 201910156872 A CN201910156872 A CN 201910156872A CN 110021041 A CN110021041 A CN 110021041A
- Authority
- CN
- China
- Prior art keywords
- scene
- grid
- visual signature
- camera
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/579—Depth or shape recovery from multiple images from motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20228—Disparity calculation for image-based rendering
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The unmanned scene progressive mesh structural remodeling method based on binocular camera that the invention discloses a kind of.Its disparity map is calculated after pre-processing to the binocular video frame of input, and corresponding depth map is calculated based on this, and initial scene gridding structure is then constructed by triangulation and grid subdivision stage；Network smoothing processing is carried out after eliminating grid sting phenomenon；The motion information between present frame and previous frame is calculated using the location information that satellite navigation acquires, it resolves to obtain the new patch grids that global scene gridding structure is added in conjunction with the scene grid structural remodeling result of present frame, completes the work of scene grid structures to form eventually by part again triangulation methodology.The present invention innovatively utilizes the vision difference between two field pictures to find new vision region, and global scene reconstructed results are updated incrementally with gridding structure type, to obtain better performances and under multiclass environment with the method for reconstructing three-dimensional scene of higher robustness.
Description
Technical field
The invention belongs to 3 D scene rebuilding technical fields, and in particular to a kind of unmanned scene based on binocular camera
Progressive mesh structural remodeling method
Background technique
Modern Shi Jin, automatic driving vehicle, the unmanned carrier such as automatic flight unmanned plane has obtained significant progress, right
For these unmanned carriers, self-navigation and path planning in traveling or flight course are very crucial technology rings
Section.For nearly all self-navigation and Path Planning Technique, various types of scene structure maps are all essential
, traditional scene rebuilding generally uses the offline scenario structural remodeling scheme based on all kinds of high-precision hardware, such as: automatic
Driving field, researcher often pass through one and are equipped with a kind of high-precision RTK (carrier wave phase for positioning when high-precision real
Position differential technique) the manned vehicle of equipment and multi-thread beam laser radar carries out the acquisition and reconstruction of scene structure data.This
There are two drawbacks for kind acquisition method: 1) scene structure constructed by is made of dense laser point cloud, is substantially scatterplot structure,
And the scene structure in real scene is made of continuous line or face, scatterplot structure can lose important scene continuous structure letter
Breath；2) hardware cost needed for this scene structure reconstruction model is very high, is not particularly suited for all research institutions and development teams.
For the drawbacks described above that the offline scenario structural remodeling scheme based on all kinds of high-precision hardware is included, and with
The development of SLAM (synchronous to position and build figure) technology, has had some advanced SLAM technologies that can directly estimate by state now
The mode of meter constructs dense scene map, but this kind of technology still remains and some is difficult to the drawbacks of being solved perfectly at present: 1) institute
The scene structure of building is made of the dense point cloud with colouring information, is substantially scatterplot structure, and the field in real scene
Scape structure is made of continuous line or face, and scatterplot structure can lose important scene continuous structure information；2) more due to scene
Various in sample and scene to be difficult to expect emergency case, even state-of-the-art SLAM technology also can not be in all types of rings
Outstanding state is kept to resolve effect in border, so the dense scene map for fully relying on the building of SLAM technology tends not to reach
Satisfactory precision.
To sum up, problem to be solved of the present invention is:
1. the discontinuous problem of scene structure reconstructed results: dense map constructed by many classical scenario structural remodeling schemes
It is a kind of spot style " pseudo- dense " map, this map obviously can not directly support the self-navigation of all kinds of unmanned carriers
With the tasks such as path planning；
2. the excessively high problem of scene structure reconstructed cost: in the scene based on high-precision RTK equipment Yu multi-thread beam laser radar
In structural remodeling scheme, research and development team generally requires the very expensive hardware device of procurement price, such as high-precision satellite
Navigation equipment and 64 line laser radars etc., for most of research and development team, this hardware cost will be will cause significantly
Financial burden, for company, this hardware device scheme problem excessively high due to own cost, it is clear that be unable to meet volume
Production demand.
3. scene structure reconstructed results precision deficiency problem: many classical scenario structural remodeling schemes are based especially on each
The scene structure reconstruction model of type games information solving technique, the scene structure result constructed are often not achieved satisfactory
Precision, it is because the algorithm of single type often can not be in love in institute after all that the reason of causing such case, which has a lot,
Satisfactory processing result can be obtained under condition, the solution institute that this is also primarily based upon algorithm types technology is intrinsic extensively
One of disadvantage.
Summary of the invention
In order to solve the problems in background technique, present invention combination stereovision technique, Triangulation Technique, grid subdivision
Technology and lattice optimization techniques etc., the increment type scene of high stability can be kept under outdoor free environments by having developed a whole set of
Gridding structural remodeling method, and by the way that experimental results demonstrate the validity of the system.
The technical solution adopted by the present invention includes the following steps:
1) the single frames binocular video frame in the scene acquired by vehicle-mounted binocular camera is inputted, to the single frames binocular vision of input
Frequency frame carries out scene grid structural remodeling, therefrom obtains network structure feature.
The present invention is applied to unmanned or other indoor or outdoors scenes with clear visual texture feature, nobody drives
It sails scene and specifically includes the common road scene such as urban road, country road, highway.
The step 1) specifically:
1.1) visual signature in the two images in the single frames binocular video frame acquired by binocular camera is extracted respectively
Point；Herein can be there are many visual signature point Choice, such as FAST characteristic point, ORB characteristic point, BRIEF characteristic point etc.,
Can be according to varying environment feature, situations such as different system frame per second requirements, different hardware equipment, is chosen.In different type environment
In, the best feature vertex type of effect is likely to be different；FAST characteristic point, ORB characteristic point or the BRIEF acquired
Characteristic point is visual signature point.
1.2) visual signature point matching stage: the point of visual signature obtained in step 1.1) is matched based on violence
(Brute-Force Matcher) method is matched to obtain visual signature point pair, calculates the parallax of every a pair of of visual signature point
Value；
1.3) camera coordinates the visual signature point depth estimation stage: are established using left mesh camera initial position as coordinate origin
System, exists according to the visual signature point of the left mesh image of parallax value computation vision characteristic point centering of visual signature point pair in step 1.2)
Three-dimensional position in camera coordinates system, specific calculating process are as follows:
1.3.1 the depth value of each visual signature point) is calculated, to obtain the sparse depth figure of left mesh image:
Wherein, d is parallax value, and b is binocular camera baseline length, and f is camera focus, and z is depth value；
1.3.2 three-dimensional position of the visual signature point of left mesh image in camera coordinates system) is calculated:
Pw=zK-1Puv
Wherein, K is the Intrinsic Matrix of binocular camera, PuvThe homogeneous seat fastened for visual signature point in image pixel coordinates
Mark, image pixel coordinates system are using the image upper left corner as the two-dimensional coordinate system of origin, PwIt is visual signature point in camera coordinates system
In coordinate, u is visual signature point corresponding position in transverse coordinate axis in image pixel coordinates system, and v is visual signature point
The corresponding position in longitudinal coordinate axle in image coordinate system, X are visual signature point corresponding position in x-axis in camera coordinates system
It sets, Y is that corresponding position, Z are visual signature point z-axis in camera coordinates system to visual signature point in y-axis in camera coordinates system
Upper corresponding position；
1.4) the triangulation stage: carrying out subdivision to visual signature point set inside using Delaunay Triangulation method,
Triangle gridding structure is obtained, wherein the visual signature point set is the set of visual signature point on left mesh image；
1.5) the grid subdivision stage: according to step 1.4) triangle gridding structure generated and the depth of each grid vertex
Angle value carries out depth interpolation to the triangle gridding structure of present frame, describes sub- iteration by Census and finds integrated triangular net lattice knot
Visual signature point in structure on the left mesh image of depth error maximum preceding 5% as visual signature point to be updated, and according to
Hamming distance matches visual signature point to be updated again, visual signature point set is added in visual signature point to be updated, again
Delaunay Triangulation is carried out inside visual signature point set, thus the scene grid structure after being segmented.
2) it is found in scene grid structure using network structure feature and the position of grid " sting " phenomenon occurs, and eliminated
Grid " sting " phenomenon recycles approximate Laplce's smoothing method to promote the smoothness of overall scenario network structure.
" sting " phenomenon refers to the several nets of single or only a few in the local location appearance of scene gridding structure
Lattice vertex depth compares the excessive phenomenon of difference for trellis depth around, and this phenomenon is to lead to overall scenario gridding knot
One of the main reason of structure estimation inaccuracy.
The step 2) specifically:
2.1) grid " sting " removes the stage: each grid vertex in scene grid structure is handled to traversal formula, when
The depth of grid vertex be more than or less than all of its neighbor grid vertex depth, and with the mean depth on all of its neighbor vertex it
Between absolute value of the difference when being greater than threshold value, with the depth of the mean depth on all of its neighbor vertex substitution grid vertex；
2.2) network smoothing stage: with a kind of approximate Laplce's smoothing method to the net in scene grid structure
Lattice vertex is smoothed one by one, and the calculation of approximate Laplce's smoothing method is as follows:
Wherein, ZcFor the depth value of grid vertex to be processed, ZcIt is averaged for all of its neighbor vertex of grid vertex to be processed
Depth value, α are the damping parameter manually set, PwFor position of the grid vertex to be processed in camera coordinates system, PoIt is to be processed
Position of the grid vertex after optimization in camera coordinates system.
3) using the location information of Vehicular satellite navigation equipment acquisition scene, by location information resolving previous frame and currently
Motion information between frame carries out new vision region detection to obtain the transformation matrix that description interframe continuously moves later；Newly
Visual zone detection are as follows: virtual image is constructed according to the scene grid structure of transformation matrix and previous frame, utilizes virtual image
The new vision region of previous frame compared with the vision difference between present frame determines present frame.
The detection method in new vision region is to detect in present frame in a manner of based on virtual image in the step 3)
New vision region, it is specific as follows:
It constructs and obtains after each pixel of left mesh image in previous frame single frames binocular video frame is handled as follows
Virtual image:
PpFor the homogeneous coordinates that the pixel coordinate of left mesh image of the pixel in previous frame single frames binocular video frame is fastened,
zpDepth value for pixel at the previous frame moment, PcFor left mesh image of the pixel in present frame single frames binocular video frame
The homogeneous coordinates that pixel coordinate is fastened, zcDepth value for pixel at the present frame moment, T are the corresponding fortune of inter motion information
Dynamic matrix, K are the Intrinsic Matrix of binocular camera；
Image repair is carried out to virtual image using Navier-Stokes equation, then compares the virtual image after repairing
Vision difference in the left mesh image of present frame on each tri patch region, obtains vision difference value, the tool of vision difference value
Body calculation is as follows:
Wherein, n is sum of all pixels in tri patch region, gpiFor gray value of the ith pixel position in virtual image,
gciFor in the gray value of the left mesh image of present frame, d is vision difference value；
It finally chooses vision difference value and is higher than composition face of the tri patch of average visual difference value as new vision region
Piece completes the detection in new vision region；The average visual difference value is the virtual image and the left mesh image of present frame after repairing
In the average value of all vision difference values that obtains more afterwards of vision difference on each tri patch region.
4) it relative to the cumulative motion information of the camera position of initial frame and has been constructed using the camera position of present frame
Scene dough sheet in overall scenario gridding structure determines local scene to be updated in conjunction with the new vision region in step 3)
Gridding locations of structures.
The scene grid that the overall scenario gridding structure constructed constructs for all frames before present frame
Structure；The scene grid structure that overall scenario gridding structure constructs for all frames before present frame and present frame.
The step 4) is specific as follows: using following calculation method by three in the overall scenario gridding structure constructed
Edged surface piece projects on present frame:
Wherein, PwFor position of the vertex in the world coordinate system using first frame as origin of tri patch, z is depth, T
For present frame camera position relative to initial frame camera position cumulative motion information, K be binocular camera intrinsic parameter square
Battle array, PuvThe homogeneous coordinates fastened for the vertex of the tri patch projected on present frame in pixel coordinate；
In the overall scenario gridding structure constructed, by or part weight Chong Die with the new vision region in present frame
It is folded, and tri patch position of the space length between new vision region corresponding to present frame within 5 meters is considered as
Local scene gridding locations of structures to be updated.
5) scene grid structure increment formula joining method: by new vision region and local scene gridding knot to be updated
After structure position is associated, the splicing of overall scenario gridding structure is carried out by Delaunay Triangulation method.
Scene grid structure increment formula joining method in the step 5) specifically: by the new vision region of present frame
In all tri patch carry out connected region search, all connected regions are sorted out, by each connected region and to be updated
The position that projects on the left mesh image of present frame of local scene gridding locations of structures concentrate after carry out Delaunay tri- again
Angle subdivision updates overall scenario gridding structure with this.
6) iteration step 1-5, until vehicle-mounted binocular camera obtain all single frames binocular video frames all everywhere
Reason, it is final to obtain the overall scenario gridding structural remodeling result that can satisfy unpiloted use demand.
The beneficial effects of the present invention are:
One, the present invention innovatively has found new vision region using the vision difference between two field pictures, and with gridding knot
Configuration formula incrementally updates global scene reconstructed results, to obtain better performances and have higher robust under multiclass environment
The method for reconstructing three-dimensional scene of property.
Two, the present invention significantly optimizes following three classes defect possessed by classical scenario structural remodeling scheme:
1. the discontinuous problem of scene structure reconstructed results: scene structure constructed by the present invention is gridding structure, this
Gridding structure be it is stringent continuous, there is no any structural hole theory, can satisfy all kinds of unmanned carriers carry out it is automatic
For the demand of scene structure map when the tasks such as navigation, path planning.
2. the excessively high problem of scene structure reconstructed cost: the present invention does not depend on expensive multi-thread beam laser radar completely and sets
It is standby, hardware cost needed for significantly reducing scene structure reconstructing system.
3. scene structure reconstructed results precision deficiency problem: the present invention solves scene structure reconstructed results essence from two angles
It spends insufficient problem: 1) improving the precision that scene depth resolves by Mesh Subdivision Technique and lattice optimization techniques；2) pass through satellite
Navigation equipment obtains inter motion information more more accurate than the inter motion information obtained by motion information solving technique.
Detailed description of the invention
Fig. 1 is general frame figure of the invention；
Fig. 2 illustrates contextual data acquisition module；
Fig. 3 illustrates single frames scene grid structure and resolves module；
Fig. 4 is mapping graph of the single frames network on single-frame images；
Fig. 5 illustrates single frames scene grid structure optimization module；
Fig. 6 illustrates the scene grid structure optimization effect reflected with depth map；(a) optimization to be reflected with depth map
Preceding scene gridding structure；It (b) is the scene grid structure after the optimization that is reflected with depth map；
Fig. 7 illustrates scene grid structure increment formula update module；
Fig. 8 illustrates the virtual image of generation；It (a) is the virtual image before image repair processing；(b) at image repair
Virtual image after reason；
Fig. 9 illustrates new vision area detection result；
Figure 10 illustrates the local corresponding diagram in new vision region；(a) (c) is depth value inaccuracy or vision in previous frame
The incomplete part of information；(b) (d) is new vision area detection result；
Figure 11 illustrates scene grid structural remodeling result；It (a) is grid configuration；(b) form is rendered for RGB.
Specific embodiment
As shown in Figure 1, the invention mainly comprises following four technology modules, each technology modules are made with reference to the accompanying drawing
It is further described:
One, contextual data acquisition module (as shown in Figure 2)
First stage, the video frame acquisition phase based on binocular camera:
It is continuous needing to construct using equipment such as vehicle-mounted binocular camera, UAV system binocular camera or hand-held binocular cameras
Mobile exploration is carried out in the scene of gridding structure, acquires the continuous binocular video frame of scene, it is worth mentioning at this point that, it is used here
Binocular camera be by strictly demarcating, for binocular camera equipment preferable for quality, the step should be
It completes in assembling stage, for general for quality or self assembly binocular camera equipment, needs by additional stringent
Staking-out work since this work is not core of the invention content, therefore is not unfolded to be described in detail.At this stage, it needs to guarantee
Binocular camera keeps horizontal in collection process, and keeps frame rate stable, just can guarantee acquired scene visual information in this way
Availability.
Second stage, the motion information acquisition stage based on satellite navigation:
Using all kinds of satellite navigations, synchronous recording world coordinates during recording scene continuous binocular video frame
Location information in system, and resolved from location information and obtain inter motion information.It should be noted that since frame number is different or
Some other error for being difficult to completely eliminate, the position that the scene visual information and satellite navigation that binocular camera is recorded are recorded
Information can not accomplish proper fully synchronized, and the solution taken of the present invention is here: assuming that mobile vehicle exists
The movement of (< 0.1s) is uniform motion in the very short time, is then believed according to corresponding scene visual information time stamp with position
The difference ceased between timestamp carries out at the uniform velocity position compensation.It is demonstrated experimentally that this solution is effective.
Two, single frames scene grid structure resolves module (as shown in Figure 3)
First stage, visual signature point extract the stage:
In three-dimensional reconstruction field, the basis that scene structure is rebuild is the resolving of scene depth, in the present invention, scene
Depth is obtained by Stereo Vision, stringent this requires having between the two images in single frames binocular video frame
This point may be implemented there are many technical solution, in the present invention, since structure is by grid in accurately pixel matching relationship
Form presented, it is desirable that grid vertex can more efficiently embody the significant structure feature of scene, therefore use
Visual signature point, which is extracted, realizes that the accurate pixel between two images matches with matched mode.Herein can there are many view
Feel characteristic point Choice, such as FAST characteristic point, ORB characteristic point, BRIEF characteristic point etc., can according to varying environment feature,
Situations such as different system frame per second requirements, different hardware equipment, is flexibly chosen, it is demonstrated experimentally that effect is most in different type environment
Good feature vertex type is likely to be different.
Second stage, visual signature point matching stage:
Since the basic thought of the scene depth estimation in the present invention is stereoscopic vision thought, so having extracted single frames
After visual signature point in binocular video frame in two images, it is also necessary to be carried out to the visual signature point in two images accurate
Matching.This process can be realized that rebuilding not due to scene structure of the invention is on-line reorganization by kinds of schemes, and
It is by reconstructing completion offline, so do not have strict requirements for the real-time of matching scheme, therefore present invention employs sudden and violent
Power matches the mode of (Brute-Force Matcher), according to selected visual signature vertex type, directly calculating two images
Between each visual signature point between Euclidean distance or Hamming distance, then carry out two images between visual signature point
With work, it is demonstrated experimentally that this matching strategy is effective.
Phase III, the visual signature point depth estimation stage:
After completing the visual signature point between the two images in single frames binocular video frame and precisely matching, so that it may
According to pixel difference in horizontal direction between matching visual signature point pair, the parallax of corresponding position is obtained.In the present invention, with binocular
Magazine left camera be scene structure perception basis, therefore pass through this phase process, available left mesh image it is sparse
Disparity map, the position with parallax value information are exactly the position of visual signature point on left mesh image.When having known parallax value and double
After mesh camera parameter, according to binocular camera modular concept, so that it may obtain the accurate depth value of corresponding points, calculation is as follows:
Wherein, d is parallax value, and b is binocular camera baseline length, and f is camera focus, and z is depth results；
So far, the sparse depth figure of left mesh image has just been obtained.
After the sparse depth figure for having obtained left mesh image, also need to know the pixel with depth value in three dimensions
Exact position, can be calculated here by pinhole camera modular concept, calculation is as follows:
Pw=zK-1Puv
Wherein, K is the Intrinsic Matrix of binocular camera, PuvThe homogeneous seat fastened for visual signature point in image pixel coordinates
Mark, PwFor the coordinate of visual signature point in three dimensions.
So far, the exact position of each visual signature point in three dimensions in left mesh image has just been obtained.
Fourth stage, the triangulation stage:
What the present invention to be rebuild is the scene grid structure for having Complete Continuity, only sparse scene characteristic point
Accurate three-dimensional is insufficient for this point if position.Herein, have been obtained that represent scene in left mesh image significant
The position of the visual signature point of feature in three dimensions, need by a kind of triangulation scheme by these scatterplots by point with
Continuous grid is connected and composed between point.Delaunay Triangulation scheme is used in the present invention to be realized, this is chosen
The main reason for kind triangulation scheme is that Delaunay Triangulation scheme maximizes minimum angle, be may be constructed closest to rule
The triangulation network then changed, this is very helpful for real embodiment scene structure.
5th stage, the grid subdivision stage:
The scene grid structure obtained by Delaunay Triangulation is the initial scene grid structure of comparison, essence
Degree is not enough to that reality is supported to use, needs to make grid reach higher by the further subdivided meshes of grid subdivision scheme
Precision.It determines the grid vertex that existing grid is newly added, is the chief motivation of mesh refinement scheme.In the third rank of this module
Duan Zhong has obtained the sparse disparities figure of left mesh image, and in the fourth stage of this certain block, it is corresponding initial to have obtained left mesh image
Scene grid structure, it is basic based on these data, so that it may which that the thick of left mesh image is obtained by triangle interior interpolation method
Close disparity map, according to the dense disparity map of left mesh image, so that it may which any location of pixels in left mesh image is corresponded into right mesh
In image, by Census description (a kind of description calculating side based on neighborhood of pixel points statistics for measuring pairing pixel
Formula) between Hamming distance, judge the error degree of corresponding parallax.After obtaining the dense error degree figure of left mesh image,
Image uniform is divided into several parts, several highest location of pixels of error degree is taken to add in each part as new
Enter the position of the grid vertex of initial displaying grid, but these put the exact position in corresponding three-dimensional space not yet,
For these points, since whole system is to be to maintain binocular phase based on binocular camera system, and when recording scene visual information
Machine level, so only needing to describe the correspondence in the son search highest right mesh image of matching degree according to Census on a horizontal
Then position calculates parallax, so that it may obtain the depth value of these points, can be obtained according to depth value and camera Intrinsic Matrix
To these exact positions of point in three dimensions.It, will be original after the new grid vertex position that grid is added has been determined
Delaunay Triangulation result empties, and re-starting Delaunay Triangulation further according to all grid vertexes can be obtained carefully
Scene grid structure after point.According to available accuracy needs, successive ignition, which carries out above-mentioned grid subdivision process, be can be obtained completely
The scene grid structural remodeling result as shown in Figure 4 of sufficient required precision.
Three, single frames scene grid structure optimization module (as shown in Figure 5)
First stage, grid sting remove the stage:
After having obtained single frames scene grid structural remodeling result, due in the grid subdivision stage, only iteratively
Grid vertex is added into network, not according to stringent logic Delete Mesh vertex, so scene grid structure
Error is still remained, most notable one error is exactly " sting " error.This error takes the form of in the certain of grid
There is the very high triangle gridding vertex of the projecting degree of only a few in part suddenly, and this vertex constitutes the view similar to " sting "
Feel effect, this arises primarily at matching error during resolving depth based on stereoscopic vision, obtains according to matching result
To parallax it is excessive or too small be likely to cause " sting " phenomenon.In the present invention, the project plan comparison for removing grid sting is straight
It connects, judges whether a grid vertex is the key that cause " sting " phenomenon vertex, mainly calculates the depth of the grid vertex
Absolute value of the difference between the mean depth on all of its neighbor vertex, and compare depth and all of its neighbor vertex of the grid vertex
Depth, when the depth absolute value of the difference is greater than some threshold value for setting based on practical experience, and the depth of the grid vertex
More than or less than all of its neighbor vertex depth when, being considered as the point is the key that cause " sting " phenomenon vertex, and with the point
All of its neighbor vertex mean depth replace the point original depth, thus can solve single " sting " phenomenon.Experiment
It proves, after the process is acted on entire net region, so that it may solve all " stings " in scene grid structure
Phenomenon.
Second stage, network smoothing stage:
In traditional network Smooth scheme, through being put down frequently with Laplce's Smooth scheme to network
Sliding, this mode can significantly improve the smoothness of network, while can also change the position of several grid vertexes inside grid
It sets.But in the present invention, the grid vertex in order not to be changed position changes in the position that image pixel coordinates are fastened
Becoming, a kind of approximation Laplce's Smooth scheme is used, and this Smooth scheme is still to be handled by grid vertex,
It is handled by following formula:
Wherein, ZcFor the depth value of grid vertex to be processed, ZnIt is averaged for all of its neighbor vertex of grid vertex to be processed
Depth value, α are the damping parameter manually set, PwFor the position of grid vertex to be processed in three dimensions, PoFor net to be processed
Position of the lattice vertex after optimization in three dimensions.
It is demonstrated experimentally that after the process is acted on entire net region, so that it may significantly improve scene grid knot
The smoothness of structure is allowed to more close to real scene structure, be illustrated in figure 6 excellent with the scene grid structure of depth map reflection
Change effect, Fig. 6 (a) and Fig. 6 (b) are respectively the scene grid structure before and after the optimization reflected with depth map.
Four, scene grid structure increment formula update module (as shown in Figure 7)
First stage, scene grid topology update detection-phase:
After by first time single frames scene grid structure optimization resume module, single frames scene grid has just been obtained
All three-dimensional triangulation dough sheets for including in reconstructed results as a result, and be uniformly stored in scene grid structural library by structural remodeling.
But this is only the scene grid structural remodeling of single frames as a result, being unsatisfactory for reality use needs, in the present invention, with one
The form of kind increment type expands the gridding structure of scene, first after new frame binocular image input system, system meeting frame by frame
The corresponding scene grid structure of new binocular image frame is constructed according to above three module, then judges the left side in new binocular image
Which region belongs to new vision region on mesh image, and includes with the tri patch that is related to as needing to be added using new vision region
New tri patch in original scene grid structural library.In the present invention, new tri patch is completed using virtual image scheme
Choose work.The corresponding scene grid structure of previous frame binocular image is obtained in system, present frame binocular image is corresponding
Scene grid structure and two frame binocular images between motion information, according to principle of computer vision, it is available on
Each point on left mesh image in one frame binocular image is incident upon in current binocular image after motion information effect
Left mesh image on position, for a single point on the left mesh image in previous frame binocular image specific calculation such as
Under:
If the homogeneous coordinates that pixel coordinate of this in the left mesh image in previous frame binocular image is fastened are Pp, this point
It is z in the depth value at previous frame momentp, pixel coordinate of this in the left mesh image in present frame binocular image is fastened neat
Secondary coordinate is Pc, depth value of this at the present frame moment is zc, the corresponding kinematic matrix of inter motion information be T, camera it is interior
Parameter matrix is K, is calculated by following formula:
When all pixels with depth value acted on the process in the left mesh image in previous frame binocular image
Later, so that it may obtain a secondary virtual image generated according to depth value and inter motion information, so far, which is band
The image in cavity needs to repair the image by interpolation method, and what is taken in the present invention is based on Navier-Stokes equation
Image repair scheme, as shown in Figure 8 it is demonstrated experimentally that the recovery scenario has reached satisfactory journey for the reparation result of image
Degree, Fig. 8 (a) are the virtual image before image repair processing, and Fig. 8 (b) is image repair treated virtual image.It is being repaired
After virtual image after multiple, the gray value of the image and the gray value of the left mesh image in present frame binocular image are compared
Right, statistical average gray value error chooses gray value error and is higher than the tri patch of average gray value error as new vision area
The composition dough sheet (white face panel region as shown in Figure 9 illustrates new vision area detection result) in domain, scene net to be added
It formats in structural library, the scene grid topology update after being is prepared.
Figure 10 illustrates the local corresponding diagram in new vision region, wherein (a) (c) be previous frame in depth value inaccuracy or
The incomplete part of visual information；(b) (d) is (a) (c) corresponding new vision area detection result.
Second stage, the scene grid topology update stage:
In order to which the scene grid structure for generating increment type is gridding structure continuous and without redundancy dough sheet, institute
After the tri patch in the representative new vision region for having obtained handling on last stage, it is also necessary to selected original scene grid
Which scene tri patch in structural library is that needs are optimised.In the present invention, the side based on interframe mapping is still taken
Case completes this work.Process is embodied are as follows: (what is set in a particular embodiment is ten by several frames before present frame
Frame) tri patch in corresponding scene grid structural library is mapped to present frame according to motion information, by in present frame
The overlapping of new vision region or partly overlapping tri patch, which are considered as, answers optimised tri patch；Again by new vision area in present frame
All tri patch in domain carry out connected region search, sort out all connected regions；Then according to these triangles to be optimized
It is corresponding with all tri patch in present frame in new vision region that dough sheet is mapped to the image coordinate vertex after present frame
Image coordinate vertex concentrates in together, and by Delaunay Triangulation technology, on the basis of each connected region, carries out again
Triangulation work；Finally, delete these tri patch to be optimized in scene gridding structural library, and by new Delaunay tri-
Scene grid structural library is added in divided region corresponding tri patch in angle.The operation is carried out frame by frame, just can obtain global scope
On continuous scene grid structural remodeling result.As Figure 11 illustrates scene grid structural remodeling as a result, wherein Figure 11 (a)
For grid configuration, for effect of visualization, structure more than certain depth is replaced with scatterplot, and Figure 11 (b) is that RGB renders form.
Finally it is pointed out that above embodiments are only the more representational examples of the present invention.Obviously, technology of the invention
Scheme is not limited to above-described embodiment, and acceptable there are many deformations.Those skilled in the art can not depart from the present invention
The invention state of mind under, various modifications or variation are made for above-described embodiment, thus protection scope of the present invention not by
Above-described embodiment is limited, and should be the maximum magnitude for meeting the inventive features that claims are mentioned.
Claims (6)
1. a kind of unmanned scene progressive mesh structural remodeling method based on binocular camera, which is characterized in that including
Following steps:
1) the single frames binocular video frame in the scene acquired by vehicle-mounted binocular camera is inputted, to the single frames binocular video frame of input
Carry out scene grid structural remodeling；
2) it is found in scene grid structure using network structure feature and the position of grid " sting " phenomenon occurs, and eliminate grid
" sting " phenomenon recycles approximate Laplce's smoothing method to promote the smoothness of overall scenario network structure；
3) using the location information of Vehicular satellite navigation equipment acquisition scene, by location information resolve previous frame and present frame it
Between motion information carry out new vision region detection later to obtain the transformation matrix that continuously moves of description interframe；New vision
Region detection are as follows: according to the scene grid structure of transformation matrix and previous frame construct virtual image, using virtual image with work as
Vision difference between previous frame determines that present frame compares the new vision region of previous frame；
4) using the camera position of present frame relative to the cumulative motion information of the camera position of initial frame and the entirety constructed
Scene dough sheet in scene grid structure determines local scene grid to be updated in conjunction with the new vision region in step 3)
Change locations of structures；
5) scene grid structure increment formula joining method: by new vision region and local scene gridding structure bit to be updated
It sets after being associated, passes through the splicing that Delaunay Triangulation method carries out overall scenario gridding structure；
6) iteration step 1-5, until all single frames binocular video frames that vehicle-mounted binocular camera obtains are processed, most
Overall scenario gridding structural remodeling result is obtained eventually.
2. the unmanned scene progressive mesh structural remodeling method according to claim 1 based on binocular camera,
It is characterized by:
The step 1) specifically:
1.1) the visual signature point in the two images in the single frames binocular video frame acquired by binocular camera is extracted respectively；
1.2) visual signature point matching stage: the point of visual signature obtained in step 1.1) is based on violence and matches (Brute-
Force Matcher) method matched to obtain visual signature point pair, calculate the parallax value of every a pair of of visual signature point；
1.3) camera coordinates system, root the visual signature point depth estimation stage: are established using left mesh camera initial position as coordinate origin
According to the visual signature point of the left mesh image of parallax value computation vision characteristic point centering of visual signature point pair in step 1.2) in camera
Three-dimensional position in coordinate system, specific calculating process are as follows:
1.3.1 the depth value of each visual signature point) is calculated, to obtain the sparse depth figure of left mesh image:
Wherein, d is parallax value, and b is binocular camera baseline length, and f is camera focus, and z is depth value；
1.3.2 three-dimensional position of the visual signature point of left mesh image in camera coordinates system) is calculated:
Pw=zK-1Puv
Wherein, K is the Intrinsic Matrix of binocular camera, PuvFor the homogeneous coordinates that visual signature point is fastened in image pixel coordinates,
PwFor coordinate of the visual signature point in camera coordinates system, u is visual signature point transverse coordinate axis in image pixel coordinates system
Upper corresponding position, v are that corresponding position, X are visual signature point to visual signature point in longitudinal coordinate axle in image coordinate system
The corresponding position in x-axis in camera coordinates system, Y are that corresponding position, Z are visual signature point in y-axis in camera coordinates system
Visual signature point corresponding position in z-axis in camera coordinates system；
1.4) it the triangulation stage: using Delaunay Triangulation method to subdivision is carried out inside visual signature point set, obtains
Triangle gridding structure, wherein the visual signature point set is the set of visual signature point on left mesh image；
1.5) the grid subdivision stage: according to step 1.4) triangle gridding structure generated and the depth value of each grid vertex
Depth interpolation is carried out to the triangle gridding structure of present frame, sub- iteration is described by Census and is found in integrated triangular net lattice structure
Visual signature point on the left mesh image of depth error maximum preceding 5% is as visual signature point to be updated, and according to Hamming
Distance matches visual signature point to be updated again, and visual signature point set is added in visual signature point to be updated, is being regarded again
Feel and carry out Delaunay Triangulation inside feature point set, thus the scene grid structure after being segmented.
3. the unmanned scene progressive mesh structural remodeling method according to claim 1 based on binocular camera,
It is characterized by:
The step 2) specifically:
2.1) grid " sting " removes the stage: handling to traversal formula each grid vertex in scene grid structure, works as grid
The depth on vertex is more than or less than the depth of all of its neighbor grid vertex, and between the mean depth on all of its neighbor vertex
When absolute value of the difference is greater than threshold value, with the depth of the mean depth substitution grid vertex on all of its neighbor vertex；
2.2) network smoothing stage: with a kind of approximate Laplce's smoothing method to the grid top in scene grid structure
Point is smoothed one by one, and the calculation of approximate Laplce's smoothing method is as follows:
Wherein, ZcFor the depth value of grid vertex to be processed, ZnFor the mean depth on all of its neighbor vertex of grid vertex to be processed
Value, α is the damping parameter manually set, PwFor position of the grid vertex to be processed in camera coordinates system, PoFor grid to be processed
Position of the vertex after optimization in camera coordinates system.
4. the unmanned scene progressive mesh structural remodeling method according to claim 1 based on binocular camera,
It is characterized by: the detection method in new vision region is to detect present frame in a manner of based on virtual image in the step 3)
In new vision region, it is specific as follows:
Building obtains virtual after each pixel of left mesh image in previous frame single frames binocular video frame is handled as follows
Image:
PpFor the homogeneous coordinates that the pixel coordinate of left mesh image of the pixel in previous frame single frames binocular video frame is fastened, zpFor
Depth value of the pixel at the previous frame moment, PcFor the pixel of left mesh image of the pixel in present frame single frames binocular video frame
Homogeneous coordinates on coordinate system, zcDepth value for pixel at the present frame moment, T are the corresponding movement square of inter motion information
Battle array, K are the Intrinsic Matrix of binocular camera；
Image repair is carried out to virtual image using Navier-Stokes equation, then compares the virtual image after repairing and works as
Vision difference in the left mesh image of previous frame on each tri patch region obtains vision difference value, the specific meter of vision difference value
Calculation mode is as follows:
Wherein, n is sum of all pixels in tri patch region, gpiFor gray value of the ith pixel position in virtual image, gciFor
In the gray value of the left mesh image of present frame, d is vision difference value；
It finally chooses vision difference value and is higher than composition dough sheet of the tri patch of average visual difference value as new vision region, it is complete
At the detection in new vision region.
5. the unmanned scene progressive mesh structural remodeling method according to claim 1 based on binocular camera,
It is characterized in that,
The step 4) is specific as follows: using following calculation method by the triangular facet in the overall scenario gridding structure constructed
Piece projects on present frame:
Wherein, PwFor position of the vertex in the world coordinate system using first frame as origin of tri patch, z is depth, and T is to work as
Cumulative motion information of the camera position of previous frame relative to the camera position of initial frame, K are the Intrinsic Matrix of binocular camera, Puv
The homogeneous coordinates fastened for the vertex of the tri patch projected on present frame in pixel coordinate；
, will be Chong Die with the new vision region in present frame or partly overlap in the overall scenario gridding structure constructed, and
Tri patch position of the space length within 5 meters between new vision region corresponding to present frame is considered as to be updated
Local scene gridding locations of structures.
6. the unmanned scene progressive mesh structural remodeling method according to claim 1 based on binocular camera,
It is characterized by:
Scene grid structure increment formula joining method in the step 5) specifically: will be in the new vision region of present frame
All tri patch carry out connected region search, all connected regions are sorted out, by each connected region and office to be updated
The position that portion's scene grid locations of structures projects on the left mesh image of present frame carries out Delaunay triangle again after concentrating and cuts open
Point, overall scenario gridding structure is updated with this.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910156872.XA CN110021041B (en) | 2019-03-01 | 2019-03-01 | Unmanned scene incremental gridding structure reconstruction method based on binocular camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910156872.XA CN110021041B (en) | 2019-03-01 | 2019-03-01 | Unmanned scene incremental gridding structure reconstruction method based on binocular camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110021041A true CN110021041A (en) | 2019-07-16 |
| CN110021041B CN110021041B (en) | 2021-02-12 |
Family
ID=67189161
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910156872.XA Active CN110021041B (en) | 2019-03-01 | 2019-03-01 | Unmanned scene incremental gridding structure reconstruction method based on binocular camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110021041B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113989428A (en) * | 2021-09-03 | 2022-01-28 | 北京科技大学 | Metallurgical reservoir area global three-dimensional reconstruction method and device based on depth vision |
| CN115100621A (en) * | 2022-08-25 | 2022-09-23 | 北京中科慧眼科技有限公司 | Ground scene detection method and system based on deep learning network |
| WO2024152797A1 (en) * | 2023-01-16 | 2024-07-25 | 北京字跳网络技术有限公司 | Video supplementation method and apparatus, medium and electronic device |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1235185A2 (en) * | 2001-02-21 | 2002-08-28 | Boly Media Communications Inc. | Method of compressing digital images |
| CN101067868A (en) * | 2007-05-25 | 2007-11-07 | 同济大学 | System and method for converting disordered point cloud to triangular net based on adaptive flatness |
| CN101581575A (en) * | 2009-06-19 | 2009-11-18 | 南昌航空大学 | Three-dimensional rebuilding method based on laser and camera data fusion |
| CN101866497A (en) * | 2010-06-18 | 2010-10-20 | 北京交通大学 | Binocular stereo vision based intelligent three-dimensional human face rebuilding method and system |
| CN102074052A (en) * | 2011-01-20 | 2011-05-25 | 山东理工大学 | Sampling point topological neighbor-based method for reconstructing surface topology of scattered point cloud |
| CN102496184A (en) * | 2011-12-12 | 2012-06-13 | 南京大学 | Increment three-dimensional reconstruction method based on bayes and facial model |
| US9256496B1 (en) * | 2008-12-15 | 2016-02-09 | Open Invention Network, Llc | System and method for hybrid kernel—and user-space incremental and full checkpointing |
| CN106780735A (en) * | 2016-12-29 | 2017-05-31 | 深圳先进技术研究院 | A kind of semantic map constructing method, device and a kind of robot |
| CN107610228A (en) * | 2017-07-05 | 2018-01-19 | 山东理工大学 | Curved surface increment topology rebuilding method based on massive point cloud |
| CN108876909A (en) * | 2018-06-08 | 2018-11-23 | 桂林电子科技大学 | A kind of three-dimensional rebuilding method based on more image mosaics |
-
2019
- 2019-03-01 CN CN201910156872.XA patent/CN110021041B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1235185A2 (en) * | 2001-02-21 | 2002-08-28 | Boly Media Communications Inc. | Method of compressing digital images |
| CN101067868A (en) * | 2007-05-25 | 2007-11-07 | 同济大学 | System and method for converting disordered point cloud to triangular net based on adaptive flatness |
| US9256496B1 (en) * | 2008-12-15 | 2016-02-09 | Open Invention Network, Llc | System and method for hybrid kernel—and user-space incremental and full checkpointing |
| CN101581575A (en) * | 2009-06-19 | 2009-11-18 | 南昌航空大学 | Three-dimensional rebuilding method based on laser and camera data fusion |
| CN101866497A (en) * | 2010-06-18 | 2010-10-20 | 北京交通大学 | Binocular stereo vision based intelligent three-dimensional human face rebuilding method and system |
| CN102074052A (en) * | 2011-01-20 | 2011-05-25 | 山东理工大学 | Sampling point topological neighbor-based method for reconstructing surface topology of scattered point cloud |
| CN102496184A (en) * | 2011-12-12 | 2012-06-13 | 南京大学 | Increment three-dimensional reconstruction method based on bayes and facial model |
| CN106780735A (en) * | 2016-12-29 | 2017-05-31 | 深圳先进技术研究院 | A kind of semantic map constructing method, device and a kind of robot |
| CN107610228A (en) * | 2017-07-05 | 2018-01-19 | 山东理工大学 | Curved surface increment topology rebuilding method based on massive point cloud |
| CN108876909A (en) * | 2018-06-08 | 2018-11-23 | 桂林电子科技大学 | A kind of three-dimensional rebuilding method based on more image mosaics |
Non-Patent Citations (3)
| Title |
|---|
| ANDREA ROMANONI等: "Efficient moving point handling for incremental 3D manifold reconstruction", 《ARXIV》 * |
| 张广羚: "面向未知三维场景重建系统的设计与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 章国锋等: "基于单目视觉的同时定位与地图构建方法综述", 《计算机辅助设计与图形学学报》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113989428A (en) * | 2021-09-03 | 2022-01-28 | 北京科技大学 | Metallurgical reservoir area global three-dimensional reconstruction method and device based on depth vision |
| CN115100621A (en) * | 2022-08-25 | 2022-09-23 | 北京中科慧眼科技有限公司 | Ground scene detection method and system based on deep learning network |
| WO2024152797A1 (en) * | 2023-01-16 | 2024-07-25 | 北京字跳网络技术有限公司 | Video supplementation method and apparatus, medium and electronic device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110021041B (en) | 2021-02-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111968129B (en) | Instant positioning and map construction system and method with semantic perception | |
| CN105096386B (en) | A wide range of complicated urban environment geometry map automatic generation method | |
| CN104537709B (en) | It is a kind of that method is determined based on the real-time three-dimensional reconstruction key frame that pose changes | |
| CN107025685B (en) | Airborne building roof point cloud modeling method under topology perception | |
| CN108520554B (en) | Binocular three-dimensional dense mapping method based on ORB-SLAM2 | |
| CN110223383A (en) | A kind of plant three-dimensional reconstruction method and system based on depth map repairing | |
| CN105160702A (en) | Stereoscopic image dense matching method and system based on LiDAR point cloud assistance | |
| CN110021041A (en) | Unmanned scene progressive mesh structural remodeling method based on binocular camera | |
| JP2000113193A (en) | Synthesizing method for multi-viewpoint three- dimensional data and recording medium | |
| CN101383054A (en) | Hybrid three-dimensional reconstructing method based on image and scanning data | |
| JP2023002757A (en) | Method, device, and electronic apparatus for creating high precision map | |
| CN113393577B (en) | Oblique photography terrain reconstruction method | |
| CN117456136A (en) | Digital twin scene intelligent generation method based on multi-mode visual recognition | |
| Ceriani et al. | Pose interpolation slam for large maps using moving 3d sensors | |
| CN111915517A (en) | Global positioning method for RGB-D camera in indoor illumination adverse environment | |
| CN113031597A (en) | Autonomous obstacle avoidance method based on deep learning and stereoscopic vision | |
| Shivakumar et al. | Real time dense depth estimation by fusing stereo with sparse depth measurements | |
| Wu et al. | A new stereo dense matching benchmark dataset for deep learning | |
| CN118314300A (en) | Engineering measurement accurate positioning and three-dimensional modeling method and system | |
| CN117974899B (en) | Three-dimensional scene display method and system based on digital twinning | |
| CN112991436B (en) | Monocular vision SLAM method based on object size prior information | |
| Li et al. | BA-LIOM: tightly coupled laser-inertial odometry and mapping with bundle adjustment | |
| Chen et al. | Densefusion: Large-scale online dense pointcloud and dsm mapping for uavs | |
| CN104090945A (en) | Geographic space entity constructing method and system | |
| CN103955959A (en) | Full-automatic texture mapping method based on vehicle-mounted laser measurement system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |