CN106910242A - The method and system of indoor full scene three-dimensional reconstruction are carried out based on depth camera - Google Patents
The method and system of indoor full scene three-dimensional reconstruction are carried out based on depth camera Download PDFInfo
- Publication number
- CN106910242A CN106910242A CN201710051366.5A CN201710051366A CN106910242A CN 106910242 A CN106910242 A CN 106910242A CN 201710051366 A CN201710051366 A CN 201710051366A CN 106910242 A CN106910242 A CN 106910242A
- Authority
- CN
- China
- Prior art keywords
- depth image
- depth
- carried out
- segmentation
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 230000004927 fusion Effects 0.000 claims abstract description 84
- 230000011218 segmentation Effects 0.000 claims abstract description 62
- 238000001914 filtration Methods 0.000 claims abstract description 51
- 238000005457 optimization Methods 0.000 claims abstract description 43
- 238000001514 detection method Methods 0.000 claims abstract description 35
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 34
- 230000002146 bilateral effect Effects 0.000 claims abstract description 28
- 230000006978 adaptation Effects 0.000 claims abstract description 27
- 230000000007 visual effect Effects 0.000 claims abstract description 16
- 230000009466 transformation Effects 0.000 claims description 18
- 230000008569 process Effects 0.000 claims description 13
- 238000000605 extraction Methods 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 9
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 6
- 238000013519 translation Methods 0.000 claims description 6
- 230000008859 change Effects 0.000 claims description 4
- 230000017105 transposition Effects 0.000 claims description 3
- 238000004215 lattice model Methods 0.000 claims 1
- 230000006870 function Effects 0.000 description 35
- 238000010586 diagram Methods 0.000 description 15
- 230000006399 behavior Effects 0.000 description 10
- 230000000694 effects Effects 0.000 description 9
- 239000004576 sand Substances 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 230000003190 augmentative effect Effects 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000007499 fusion processing Methods 0.000 description 2
- 230000005484 gravity Effects 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 239000000155 melt Substances 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 239000000178 monomer Substances 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
- G06T2207/20028—Bilateral filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
Abstract
The present invention discloses a kind of method and system that indoor full scene three-dimensional reconstruction is carried out based on consumer level depth camera.Wherein, the method includes that obtaining depth image carries out self adaptation bilateral filtering；Visual odometry estimation is carried out using filtered depth image, view-based access control model content does automatic segmentation to image sequence, closed loop detection is done between section and section, and carry out global optimization；Volume data fusion is weighted according to the camera trace information after optimization, so as to rebuild indoor full scene threedimensional model.The embodiment of the present invention realizes guarantor side, the denoising to depth map by self adaptation bilateral filtering algorithm, view-based access control model content automatic segmentation algorithm can effectively reduce the accumulated error during visual odometry is estimated and improve registration accuracy, weighted body data anastomosing algorithm is additionally used, the geometric detail of body surface can be effectively kept.Thus, the technical problem for how improving reconstruction accuracy under indoor scene is solved such that it is able to obtain indoor scene model that is complete, accurate, becoming more meticulous.
Description
Technical field
The present invention relates to technical field of computer vision, in particular it relates to one kind carries out room based on consumer level depth camera
The method and system of interior full scene three-dimensional reconstruction.
Background technology
It is one of challenging research topic in computer vision that indoor scene high-precision three-dimensional is rebuild, and is related to calculate
The theory and technology of the multiple fields such as machine vision, computer graphics, pattern-recognition, optimization.Realize that Three-dimensional Gravity has various
Approach, conventional method is the knot that scene or body surface are obtained using the distance measuring sensors such as laser, radar or structured light technique
Structure information carries out three-dimensional reconstruction, but these instruments are mostly expensive and not portable, so application scenario is limited.With meter
The development of calculation machine vision technique, researchers to begin one's study and carry out three-dimensional reconstruction using the method for pure vision, wherein emerging
A large amount of beneficial research work.
After consumer level depth camera Microsoft Kinect are released, people directly can be compared just using depth data
Indoor scene three-dimensional reconstruction is carried out promptly.The KinectFusion algorithms that Newcombe et al. is proposed obtain figure using Kinect
The depth information of each point as in, by iterative approximation nearest neighbor point (Iterative Closest Point, ICP) algorithm by three
Coordinate of the dimension point under present frame camera coordinates system is alignd to estimate present frame camera with the coordinate in world model
Attitude, then volume data is carried out by curved surface implicit function (Truncated Signed Distance Function, TSDF) iteration
Fusion, obtains dense threedimensional model.Although Kinect obtains depth not influenceed by illumination condition and texture-rich degree,
Its depth data range only has 0.5-4m, and grid model position and size be fixed, so being only applicable to local, quiet
The indoor scene of state.
Indoor scene three-dimensional reconstruction is carried out based on consumer level depth camera, following problem is generally there are：(1) consumer level
Depth camera obtain depth image resolution ratio is small, noise causes that greatly body surface details is difficult to keep, and depth value scope
It is limited to cannot be directly used to full scene three-dimensional reconstruction；(2) accumulated error that camera Attitude estimation is produced can cause mistake, distortion
Threedimensional model；(3) consumer level depth camera is typically all that hand-held shoots, and the motion state of camera is more random, acquisition
The quality of data is alternated betwwen good and bad, and effect is rebuild in influence.
Kintinuous algorithms are proposed in order to carry out complete indoor scene three-dimensional reconstruction, Whelan et al., it is right
KinectFusion's further expands.The algorithm is solved greatly using the mode that ShiftingTSDFVolume recycles video memory
The problem of grid model video memory consumption during scene rebuilding, and closed loop detection is carried out by the key frame that DBoW finds matching, finally
Pose figure and model are done and is optimized, so as to obtain large scene threedimensional model.Choi et al. proposes Elastic Fragment think ofs
Think, first RGBD data flows are segmented every 50 frames, individually do visual odometry to every section and estimate, from point cloud number intersegmental two-by-two
Finding matching according to the sub- FPFH of middle extraction geometric description carries out closed loop detection, is re-introduced into line processes constraints to testing result
The closed loop of mistake is optimized, removed, finally volume data fusion is carried out using the odometer information after optimization.By segment processing
Detect that realizing indoor full scene rebuilds with closed loop, but do not account for retaining the local geometric details of object, and it is this
The method of fixed segments is carrying out when true indoor scene is rebuild not robust.It is general that Zeng et al. proposes 3D Match description
Read, RGBD data flows first are fixed into segment processing for the algorithm and reconstruction obtains partial model, from the 3D models that each is segmented
The upper input for extracting key point as 3D convolutional networks (ConvNet), the characteristic vector obtained with the e-learning is used as another
The input of matrixing network (Metric network), by similarity-rough set output matching result.Due to depth network have it is non-
Normal obvious feature learning advantage, geometrical registration is made of 3D Match with respect to other description can improve reconstruction precision.But
This method needs first to carry out partial 3 d reconstruction, geometrical registration is done using deep learning network, then export global three-dimensional mould
Type, and network training needs substantial amounts of data, and whole reconstruction flow is less efficient.
In terms of reconstruction accuracy is improved, Angela et al. proposes VSBR algorithms, and its main thought is to utilize light and shade
Recover shape (Shape from Shading, SFS) Technique on T SDF data to be merged again after carrying out hierarchy optimization, to solve
Excess smoothness causes the problem of body surface loss in detail during TSDF data fusions, so as to obtain more fine three-dimensional structure mould
Type.But this method is only rebuild relatively effectively to the monomer under perfect light source, and indoor scene is carried because light source changes greatly precision
Rise unobvious.
In view of this, it is special to propose the present invention.
The content of the invention
In order to solve above mentioned problem of the prior art, it has been how solution improves reconstruction accuracy under indoor scene
Technical problem, there is provided a kind of method and system that indoor full scene three-dimensional reconstruction is carried out based on consumer level depth camera.
To achieve these goals, on the one hand, following technical scheme is provided：
A kind of method that indoor full scene three-dimensional reconstruction is carried out based on consumer level depth camera, the method can include：
Obtain depth image；
Self adaptation bilateral filtering is carried out to the depth image；
The segment fusion and registration process of view-based access control model content are carried out to filtered depth image；
According to result, volume data fusion is weighted, so as to rebuild indoor full scene threedimensional model.
Preferably, it is described self adaptation bilateral filtering is carried out to the depth image to specifically include：
Self adaptation bilateral filtering is carried out according to following formula：
Wherein, the u and ukAny pixel and its field pixel on the depth image are represented respectively；The Z
(u) and Z (uk) the correspondence u and u is represented respectivelykDepth value；It is describedRepresent corresponding depth after filtering
Value；The W is represented in fieldOn normalization factor；The wsWith the wcIt is illustrated respectively in spatial domain and codomain filtering
Gaussian kernel function.
Preferably, the gaussian kernel function filtered in spatial domain and codomain determines according to following formula：
Wherein, the δsWith the δcIt is respectively the variance of spatial domain and codomain gaussian kernel function；
Wherein, the δsWith the δcDetermined according to following formula：
Wherein, the f represents the focal length of the depth camera, the KsWith the KcRepresent constant.
Preferably, the segment fusion and registration process that view-based access control model content is carried out to filtered depth image is specific
Including：View-based access control model content is segmented to range image sequence, and each segmentation is carried out between segment fusion, and the segmentation
Closed loop detection is carried out, the result to closed loop detection does global optimization.
Preferably, the view-based access control model content is segmented to range image sequence, and each segmentation is carried out piecemeal and melts
Close, and closed loop detection is carried out between the segmentation, doing global optimization to the result that closed loop is detected specifically includes：
View-based access control model content detection automatic segmentation method is segmented to range image sequence, by similar depth image
Hold and divide in a segmentation, and segment fusion is carried out to each segmentation, determine the transformation relation between the depth image, and root
Closed loop detection is done between section and section according to the transformation relation, to realize global optimization.
Preferably, the view-based access control model content detection automatic segmentation method is segmented to range image sequence, will be similar
Depth image content point be segmented at one, and segment fusion is carried out to each segmentation, determine between the depth image
Transformation relation, and closed loop detection is done between section and section according to the transformation relation, to realize global optimization, specifically include：
Using Kintinuous frameworks, visual odometry estimation is carried out, obtain the camera pose letter under every frame depth image
Breath；
According to the camera posture information, will be by described per the corresponding cloud data back projection of frame depth image to initial seat
Under mark system, similarity-rough set is carried out with the depth image of initial frame with the depth image obtained after projection, and when similarity is less than
During similarity threshold, camera pose is initialized, be segmented；
PFFH geometric descriptions in each segmentation cloud data is extracted, and rough registration is carried out between every two sections, and
Smart registration is carried out using GICP algorithms, the matching relationship between section and section is obtained；
Using the posture information and the described section of matching relationship and section between of each segmentation, structure is schemed and uses G2O frames
Frame carries out figure optimization, the camera trace information after being optimized, so as to realize the global optimization.
Preferably, it is described according to the camera posture information, will be by described anti-per the corresponding cloud data of frame depth image
Project under initial coordinate system, similarity-rough set is carried out with the depth image of initial frame with the depth image obtained after projection, and
When similarity is less than similarity threshold, camera pose is initialized, be segmented, specifically included：
Step 1：Calculate the similarity of every frame depth image and the first frame depth image；
Step 2：Judge the similarity whether less than similarity threshold；
Step 3：If so, being then segmented to the range image sequence；
Step 4：Using next frame depth image as next segmentation start frame depth image, and repeat step 1 and
Step 2, until having processed all frame depth images.
Preferably, the step 1 is specifically included：
According to projection relation and the depth value of any frame depth image, and each on the depth image is calculated using following formula
The first space three-dimensional point corresponding to pixel：
P=π-1(up,Z(up))
Wherein, the upIt is any pixel on the depth image；Z (the up) and the p represent the u respectivelypIt is right
The depth value and the first space three-dimensional point answered；The π represents the projection relation；
The first space three-dimensional point rotation translation is transformed under world coordinate system according to following formula, obtains second space three
Dimension point：
Q=Tip
Wherein, the TiRepresent the i-th frame depth map correspondence space three-dimensional point to the rotation translation matrix under world coordinate system；
The p represents the first space three-dimensional point, and the q represents the second space three-dimensional point；The i takes positive integer；
According to following formula by the second space three-dimensional point back projection to two dimensional image plane, the depth map after being projected
Picture：
Wherein, the uqIt is the pixel after the corresponding projections of the q on depth image；The fx, the fy, the cxWith
The cyRepresent the internal reference of depth camera；The xq、yq、zqRepresent the coordinate of the q；The transposition of the T representing matrixs；
The valid pixel number on the depth image after the start frame depth image and any frame projection is calculated respectively, and
By both ratios as similarity.
Preferably, it is described according to result, volume data fusion is weighted, so as to rebuild indoor full scene three-dimensional mould
Type is specifically included：According to the result, the depth image of each frame is merged using unblind distance function grid model, and
Three dimensions is represented using voxel grid, so as to obtain indoor full scene threedimensional model.
Preferably, according to the result, the depth map of each frame is merged using unblind distance function grid model
Picture, and three dimensions is represented using voxel grid, so as to obtain indoor full scene threedimensional model, specifically include：
Based on noise behavior and interest regional model, the unblind is carried out using Volumetric method frameworks
Distance function data weighting is merged；
Mesh model extractions are carried out using Marching cubes algorithms, so as to obtain the indoor full scene three-dimensional mould
Type.
Preferably, the unblind distance function determines according to following formula：
fi(v)=[K-1zi(u)[uT,1]T]z-[vi]z
Wherein, fiV () represents unblind distance function, namely grid to the distance on object model surface, positive negative indication
The grid is blocked side or in visible side on surface, and zero crossing is exactly the point on surface；The K represents the phase
The Intrinsic Matrix of machine；The u represents pixel；The ziU () represents the corresponding depth values of the pixel u；The viRepresent body
Element.
Preferably, the data weighting fusion is carried out according to following formula：
Wherein, the v represents voxel；The fi(v) and the wiV () represents the corresponding unblind of the voxel v respectively
Distance function and its weight function；The n takes positive integer；The F (v) to represent and block symbol corresponding to the voxel v after merging
Number distance function value；The W (v) represents the weight of the unblind distance function value corresponding to voxel v after merging；
Wherein, the weight function can determine according to following formula：
Wherein, the diRepresent the radius in interest region；The δsIt is the noise variance in depth data；The w is normal
Number.
To achieve these goals, on the other hand, additionally provide one kind carries out interior completely based on consumer level depth camera
The system of scene three-dimensional reconstruction, the system includes：
Acquisition module, for obtaining depth image；
Filtration module, for carrying out self adaptation bilateral filtering to the depth image；
Segment fusion and registration module, for filtered depth image is carried out view-based access control model content segment fusion and
Registration process；
Volume data Fusion Module, for according to result, being weighted volume data fusion, so as to rebuild indoor complete field
Scape threedimensional model.
Preferably, the filtration module specifically for：
Self adaptation bilateral filtering is carried out according to following formula：
Wherein, the u and ukAny pixel and its field pixel on the depth image are represented respectively；The Z
(u) and Z (uk) the correspondence u and u is represented respectivelykDepth value；It is describedRepresent corresponding depth after filtering
Value；The W is represented in fieldOn normalization factor；The wsWith the wcIt is illustrated respectively in spatial domain and codomain filtering
Gaussian kernel function.
Preferably, the segment fusion specifically can be used for registration module：View-based access control model content is to range image sequence
It is segmented, and each segmentation is carried out carrying out closed loop detection between segment fusion, and the segmentation, the result to closed loop detection is done
Global optimization.
Preferably, the segment fusion also specifically can be used for registration module：
View-based access control model content detection automatic segmentation method is segmented to range image sequence, by similar depth image
Hold in point being segmented at one, segment fusion is carried out to each segmentation, determine the transformation relation between the depth image, and according to
The transformation relation does closed loop detection between section and section, to realize global optimization.
Preferably, the segment fusion is specifically included with registration module：
Camera posture information acquiring unit, for using Kintinuous frameworks, carries out visual odometry estimation, obtains every
Camera posture information under frame depth image；
Segmenting unit, for according to the camera posture information, will be by described per the corresponding cloud data of frame depth image
Back projection carries out similarity-rough set with the depth image obtained after projection under initial coordinate system with the depth image of initial frame,
And when similarity is less than similarity threshold, camera pose is initialized, it is segmented；
Registration unit, for extracting PFFH geometric descriptions in each segmentation cloud data, and enters between every two sections
Row rough registration, and smart registration is carried out using GICP algorithms, obtain the matching relationship between section and section；
Optimization unit, for posture information and the described section of matching relationship and section between using each segmentation, builds
Scheme and figure optimization is carried out using G2O frameworks, the camera trace information after being optimized, so as to realize the global optimization.
Preferably, the segmenting unit is specifically included：
Computing unit, the similarity for calculating every frame depth image and the first frame depth image；
Judging unit, for judging the similarity whether less than similarity threshold；
Segmentation subelement, for when the similarity is less than similarity threshold, being divided the range image sequence
Section；
Processing unit, for using next frame depth image as the start frame depth image of next segmentation, and repeating
Computing unit and judging unit, until having processed all frame depth images.
Preferably, the volume data Fusion Module specifically for：According to the result, using unblind apart from letter
Number grid model merges the depth image of each frame, and represents three dimensions using voxel grid, so as to obtain indoor complete field
Scape threedimensional model.
Preferably, the volume data Fusion Module is specifically included：
Weighted Fusion unit, for based on noise behavior and interest region, being carried out using Volumetric method frameworks
The unblind distance function data weighting fusion；
Extraction unit, for carrying out Mesh model extractions using Marching cubes algorithms, so as to obtain the interior
Full scene threedimensional model.
The embodiment of the present invention provides a kind of method that indoor full scene three-dimensional reconstruction is carried out based on consumer level depth camera
And system.Wherein, the method includes obtaining depth image；Self adaptation bilateral filtering is carried out to depth image；To filtered depth
Degree image carries out the segment fusion and registration process of view-based access control model content；According to result, volume data fusion is weighted, from
And rebuild indoor full scene threedimensional model.The embodiment of the present invention is melted by carrying out the piecemeal of view-based access control model content to depth image
Close and registration, the accumulated error that can be effectively reduced during visual odometry is estimated simultaneously improves registration accuracy, also using weighted body number
According to blending algorithm, this can effectively keep the geometric detail of body surface, thus, solve and how to improve three under indoor scene
Tie up the technical problem of reconstruction precision such that it is able to obtain indoor scene model that is complete, accurate, becoming more meticulous.
Brief description of the drawings
Fig. 1 is the side that indoor full scene three-dimensional reconstruction is carried out based on consumer level depth camera according to the embodiment of the present invention
The schematic flow sheet of method；
Fig. 2 a are the corresponding coloured image of depth image according to the embodiment of the present invention；
Fig. 2 b are the point cloud schematic diagram obtained from depth image according to the embodiment of the present invention；
Fig. 2 c are that the point cloud schematic diagram that bilateral filtering is obtained is carried out to depth image according to the embodiment of the present invention；
Fig. 2 d are that the point cloud schematic diagram that self adaptation bilateral filtering is obtained is carried out to depth image according to the embodiment of the present invention
Fig. 3 is the schematic flow sheet according to the fusion of the view-based access control model content section of the embodiment of the present invention, registration；
Fig. 4 is the weighted body data fusion process schematic diagram according to the embodiment of the present invention；
Fig. 5 a are the three-dimensional reconstruction result schematic diagram with non-weighted body data anastomosing algorithm；
Fig. 5 b are the local detail schematic diagram of threedimensional model in Fig. 5 a；
Fig. 5 c are that the three-dimensional reconstruction result that the weighted body data anastomosing algorithm proposed according to the embodiment of the present invention is obtained is illustrated
Figure；
Fig. 5 d are the local detail schematic diagram of threedimensional model in Fig. 5 c；
Fig. 6 is to use what the embodiment of the present invention was proposed on 3D Scene Data data sets according to the embodiment of the present invention
Method carries out the effect diagram of three-dimensional reconstruction；
Fig. 7 is that the present invention is used on Augmented ICL-NUIM Dataset data sets according to the embodiment of the present invention
The method that embodiment is proposed carries out the effect diagram of three-dimensional reconstruction；
Fig. 8 is the indoor scene gathered according to the utilization Microsoft Kinect for Windows of the embodiment of the present invention
Data carry out the effect diagram of three-dimensional reconstruction；
Fig. 9 is to be based on what consumer level depth camera carried out indoor full scene three-dimensional reconstruction according to the embodiment of the present invention
The structural representation of system.
Specific embodiment
The preferred embodiment of the present invention described with reference to the accompanying drawings.It will be apparent to a skilled person that this
A little implementation methods are used only for explaining know-why of the invention, it is not intended that limit the scope of the invention.
The embodiment of the present invention provides a kind of method that indoor full scene three-dimensional reconstruction is carried out based on consumer level depth camera.
As shown in figure 1, the method includes：
S100：Obtain depth image.
Specifically, this step can include：Depth map is obtained using based on the consumer level depth camera of structure light principle
Picture.
Wherein, based on structure light principle consumer level depth camera (Microsoft Kinect for Windows and
Xtion, abbreviation depth camera), it is, by emitting structural light, to receive reflective information to obtain the depth data of depth image.
In actual applications, it is possible to use handheld consumer level depth camera Microsoft Kinect for Windows
Gather true indoor scene data.
Depth data can be calculated according to following formula：
Wherein, f represents the focal length of consumer level depth camera；B represents baseline；D represents parallax.
S110：Self adaptation bilateral filtering is carried out to depth image.
This step is entered using the noise behavior of the consumer level depth camera based on structure light principle to the depth image for obtaining
Row self adaptation bilateral filtering.
Wherein, self adaptation bilateral filtering algorithm refers to all to be filtered in the spatial domain and codomain of depth image.
In actual applications, the bilateral filter of self adaptation can be set according to the noise behavior of depth camera and its inner parameter
The parameter of ripple algorithm, so can effectively remove noise and retain marginal information.
Local derviation is asked on parallax D to depth Z, there is following relation：
The noise of depth data occurs mainly with quantizing process, as can be seen from the above equation the variance and depth value of depth noise
Quadratic power is directly proportional, that is to say, that depth value is bigger, and noise is also bigger.In order to effectively remove the noise in depth image, this hair
Bright embodiment defines filtering algorithm based on this noise behavior.
Specifically, above-mentioned self adaptation bilateral filtering can be carried out according to following formula：
Wherein, u and ukAny pixel and its field pixel on depth image are represented respectively；Z (u) and Z (uk) represent respectively
Correspondence u and ukDepth value；Represent corresponding depth value after filtering；W is represented in fieldOn normalization factor；ws
And wcIt is illustrated respectively in the gaussian kernel function of spatial domain and codomain filtering.
In the above-described embodiments, wsAnd wcCan be determined according to following formula：
Wherein, δsAnd δcIt is respectively the variance of spatial domain and codomain gaussian kernel function.
δsAnd δcRelevant with depth value size, its value is not fixed.
Specifically, in the above-described embodiments, δsAnd δcCan be determined according to following formula：
Wherein, f represents the focal length of depth camera, KsAnd KcConstant is represented, its specific value has with the parameter of depth camera
Close.
Fig. 2 a-d schematically illustrate the effect comparison schematic diagram of different filtering algorithms.Wherein, Fig. 2 a show depth
The corresponding coloured image of image.Fig. 2 b show the point cloud obtained from depth image.Fig. 2 c is shown depth image is carried out it is double
The point cloud that side filtering is obtained.Fig. 2 d show carries out the point cloud that self adaptation bilateral filtering is obtained to depth image.
The embodiment of the present invention is by using self adaptation bilateral filtering method, it is possible to achieve the guarantor side of depth map, denoising.
S120：The segment fusion and registration process of view-based access control model content are carried out to depth image.
This step view-based access control model content is segmented to range image sequence, and carries out segment fusion to each segmentation, and
Closed loop detection is carried out between segmentation, the result to closed loop detection does global optimization.Wherein, range image sequence is depth image data
Stream.
Preferably, this step can include：Determine the transformation relation between depth image, view-based access control model content automatic segmentation
Method range image sequence is segmented, during similar depth image content point is segmented at one, to each be segmented into
Row segment fusion, determines the transformation relation between depth image, and does closed loop detection between section and section according to transformation relation, and
Realize global optimization.
Further, this step can include：
S121：Using Kintinuous frameworks, visual odometry estimation is carried out, obtain the phase seat in the plane under every frame depth image
Appearance information.
S122：According to camera posture information, will be by the corresponding cloud data back projection of every frame depth image to initial coordinate
Under system, similarity-rough set is carried out with the depth image obtained after projection and the depth image of initial frame, and when similarity is less than phase
During like degree threshold value, camera pose is initialized, be segmented.
S123：PFFH geometric descriptions in each segmentation cloud data is extracted, and rough registration is carried out between every two sections,
And smart registration is carried out using GICP algorithms, obtain the matching relationship between section and section.
This step is to doing closed loop detection between section and section.
S124：Using the matching relationship between the posture information and section and section of each segmentation, structure is schemed and uses G2O frames
Frame carries out figure optimization, the camera trace information after being optimized, so as to realize global optimization.
This step applies (Simultaneous Localization and Calibration, SLAC) mould in optimization
Formula improves non-rigid distortion, introduces the closed loop matching of line processes constraint deletion errors.
Above-mentioned steps S122 can also be specifically included：
S1221：Calculate the similarity of every frame depth image and the first frame depth image.
S1222：Whether the similarity is judged less than similarity threshold, if so, then performing step S1223；Otherwise, step is performed
Rapid S1224.
S1223：Range image sequence is segmented.
This step view-based access control model content carries out segment processing to range image sequence.So can both efficiently solve vision
Odometer estimates the accumulated error problem for producing, again can be by similar content mergence together, so as to improve registration accuracy.
S1224：Range image sequence is not segmented.
S1225：Using next frame depth image as the start frame depth image of next segmentation, and step is repeated
S1221 and step S1222, until having processed all frame depth images.
In the above-described embodiments, the step of calculating the similarity per frame depth image and the first frame depth image specifically can be with
Including：
S12211：According to projection relation and the depth value of any frame depth image, and calculated on depth image using following formula
The first space three-dimensional point corresponding to each pixel：
P=π-1(up,Z(up))
Wherein, upIt is any pixel on depth image；Z(up) and p represent u respectivelypCorresponding depth value and the first space
Three-dimensional point；π represents projection relation, i.e., per the corresponding cloud data back projection of frame depth image to the 2D-3D under initial coordinate system
The Transformation Relation of Projection.
S12212：The rotation translation of the first space three-dimensional point is transformed under world coordinate system according to following formula, obtains second empty
Between three-dimensional point：
Q=Tip
Wherein, TiThe i-th frame depth map correspondence space three-dimensional point is represented to the rotation translation matrix under world coordinate system, it can
Obtained with by visual odometry estimation；I takes positive integer；P represents the first space three-dimensional point, and q represents second space three-dimensional point, p
Coordinate with q is respectively：
P=(xp,yp,zp), q=(xq,yq,zq)。
S12213：According to following formula by second space three-dimensional point back projection to two dimensional image plane, the depth after being projected
Image：
Wherein, uqIt is the pixel after the corresponding projections of q on depth image；fx、fy、cxAnd cyRepresent the internal reference of depth camera；
xq、yq、zqRepresent the coordinate of q；The transposition of T representing matrixs.
S12214：The valid pixel on the depth image after start frame depth image and any frame projection is calculated respectively
Number, and by both ratios as similarity.
For example, similarity is calculated according to following formula：
Wherein, n0And niThe valid pixel on the depth image after start frame depth image and any frame projection is represented respectively
Number；ρ represents similarity.
Fig. 3 schematically illustrates the fusion of view-based access control model content section, the schematic flow sheet of registration.
The embodiment of the present invention use view-based access control model content automatic segmentation algorithm, can effectively reduce visual odometry estimate in
Accumulated error, improves registration accuracy.
S130：According to result, volume data fusion is weighted, so as to rebuild indoor full scene threedimensional model.
Specifically, this step can include：Segment fusion and registration process result according to view-based access control model content, using cut
Disconnected symbolic measurement (TSDF) grid model merges the depth image of each frame, and represents three dimensions using voxel grid,
So as to obtain indoor full scene threedimensional model.
This step can further include：
S131：Based on noise behavior and interest region, unblind distance is carried out using Volumetric method frameworks
Function data Weighted Fusion.
S132：Mesh model extractions are carried out using Marching cubes algorithms.
In actual applications, each frame can be merged using TSDF grid models according to the estimated result of visual odometry
Depth image represents three dimensions using resolution ratio for the voxel grid of m, i.e., each three dimensions is divided into m blocks, each net
Lattice v stores two values：Unblind distance function fi(v) and its weight wi(v)。
Wherein it is possible to determine unblind distance function according to following formula：
fi(v)=[K-1zi(u)[uT,1]T]z-[vi]z
Wherein, fiV () represents unblind distance function, namely grid to the distance on object model surface, positive negative indication
The grid is blocked side or in visible side on surface, and zero crossing is exactly the point on surface；K represents the internal reference of camera
Matrix number；U represents pixel；ziU () represents the corresponding depth values of pixel u；viRepresent voxel.Wherein, the camera can be depth phase
Machine or depth camera.
Wherein it is possible to carry out data weighting fusion according to following formula：
Wherein, fi(v) and wiV () represents the corresponding unblind distance function (TSDF) of voxel v and its weights letter respectively
Number；N takes positive integer；F (v) represents the unblind distance function value corresponding to voxel v after merging；W (v) represents voxel after fusion
The weight of the unblind distance function value corresponding to v.
In the above-described embodiments, weight function can determine according to the noise behavior of depth data and interest region,
Its value is unfixed.In order to keep the geometric detail of body surface, by the small region of noise and the weights of area-of-interest
Set greatly, the weights in the big region of noise or region of loseing interest in are set small.
Specifically, weight function can determine according to following formula：
Wherein, diThe radius in interest region is represented, radius is smaller to represent interested, and weights are bigger；δsIn being depth data
Noise variance, its value is consistent with the variance of self adaptation bilateral filtering algorithm spatial domain kernel function；W is constant, it is preferable that its
Can be 1 or 0 with value.
Fig. 4 schematically illustrates weighted body data fusion process schematic diagram.
The embodiment of the present invention can effectively keep the geometric detail of body surface using weighted body data anastomosing algorithm, can
Indoor scene model that is complete, accurate, becoming more meticulous is obtained, with good robustness and autgmentability.
Fig. 5 a are schematically illustrated with the three-dimensional reconstruction result of non-weighted body data anastomosing algorithm；Fig. 5 b are exemplarily
Show the local detail of threedimensional model in Fig. 5 a；Fig. 5 c schematically illustrate the weighted body proposed using the embodiment of the present invention
The three-dimensional reconstruction result that data anastomosing algorithm is obtained；Fig. 5 d schematically illustrate the local detail of threedimensional model in Fig. 5 c.
Fig. 6 is schematically illustrated the method proposed using the embodiment of the present invention on 3D Scene Data data sets and entered
The effect diagram of row three-dimensional reconstruction；Fig. 7 is schematically illustrated on Augmented ICL-NUIM Dataset data sets
The method proposed using the embodiment of the present invention carries out the effect diagram of three-dimensional reconstruction；Fig. 8 schematically illustrates utilization
The indoor scene data of Microsoft Kinect for Windows collections carry out the effect diagram of three-dimensional reconstruction.
It should be noted that herein although the embodiment of the present invention is described with said sequence, those skilled in the art's energy
It is enough to understand, may also take on different from description order herein to implement the present invention, these simple changes should also be included in this
Within the protection domain of invention.
Based on embodiment of the method identical technology design, the embodiment of the present invention also provides a kind of based on consumer level depth phase
The system that machine carries out indoor full scene three-dimensional reconstruction, as shown in figure 9, the system 90 includes：Acquisition module 92, filtration module
94th, segment fusion and registration module 96 and volume data Fusion Module 98.Wherein, acquisition module 92 is used to obtain depth image.Filter
Ripple module 94 is used to carry out self adaptation bilateral filtering to depth image.Segment fusion is used for filtered depth with registration module 96
Degree image carries out the segment fusion and registration process of view-based access control model content.Volume data Fusion Module 98 is used for according to result,
Volume data fusion is weighted, so as to rebuild indoor full scene threedimensional model.
By using above-mentioned technical proposal, the accumulation that can be effectively reduced during visual odometry is estimated is missed the embodiment of the present invention
Difference, and registration accuracy is improved, the geometric detail of body surface can be effectively kept, room that is complete, accurate, becoming more meticulous can be obtained
Interior model of place.
In certain embodiments, filtration module specifically for：Self adaptation bilateral filtering is carried out according to following formula：
Wherein, u and ukAny pixel and its field pixel on depth image are represented respectively；Z (u) and Z (uk) represent respectively
Correspondence u and ukDepth value；Represent corresponding depth value after filtering；W is represented in fieldOn normalization factor；ws
And wcIt is illustrated respectively in the gaussian kernel function of spatial domain and codomain filtering.
In certain embodiments, segment fusion specifically can be used for registration module：View-based access control model content is to depth image
Sequence is segmented, and each segmentation is carried out to carry out closed loop detection between segment fusion, and segmentation, and the result to closed loop detection is done
Global optimization.
In further embodiments, segment fusion also specifically can be used for registration module：Determine between depth image
Transformation relation, view-based access control model content detection automatic segmentation method is segmented to range image sequence, by similar depth image
During content point is segmented at one, and segment fusion is carried out to each segmentation, the transformation relation between depth image is determined, according to change
Change relation and closed loop detection is done between section and section, and realize global optimization.
In some preferred embodiments, segment fusion can specifically include with registration module：Camera posture information is obtained
Unit, segmenting unit, registration unit and optimization unit.Wherein, camera posture information acquiring unit is used to use Kintinuous
Framework, carries out visual odometry estimation, obtains the camera posture information under every frame depth image.Segmenting unit is used for according to camera
Posture information, will be by the corresponding cloud data back projection of every frame depth image to initial coordinate system under, with the depth obtained after projection
Degree image carries out similarity-rough set with the depth image of initial frame, and when similarity is less than similarity threshold, initializes camera
Pose, is segmented.Registration unit be used for extract it is each segmentation cloud data in PFFH geometric descriptions, and every two sections it
Between carry out rough registration, and smart registration is carried out using GICP algorithms, obtain the matching relationship between section and section.Optimization unit is used for
Using the matching relationship between the posture information and section and section of each segmentation, structure is schemed and carries out figure optimization using G2O frameworks,
Camera trace information after being optimized, so as to realize global optimization.
Wherein, above-mentioned segmenting unit can specifically include：Computing unit, judging unit, segmentation subelement and processing unit.
Wherein, computing unit is used to calculate the similarity of every frame depth image and the first frame depth image.Judging unit is used to judge phase
Whether similarity threshold is less than like degree.Segmentation subelement is used for when similarity is less than similarity threshold, to range image sequence
It is segmented.Processing unit is used to, using next frame depth image as the start frame depth image of next segmentation, and repeat
Computing unit and judging unit, until having processed all frame depth images.
In certain embodiments, volume data Fusion Module specifically can be used for according to result, using unblind away from
The depth image of each frame is merged from function grid model, and three dimensions is represented using voxel grid, it is indoor complete so as to obtain
Whole scene threedimensional model.
In certain embodiments, volume data Fusion Module can specifically include Weighted Fusion unit and extraction unit.Wherein,
Weighted Fusion unit is used to, based on noise behavior and interest region, unblind is carried out using Volumetric method frameworks
Distance function data weighting is merged.Extraction unit is used to carry out Mesh model extractions using Marching cubes algorithms, so that
Obtain indoor full scene threedimensional model.
The present invention is described in detail with a preferred embodiment below.
The system for carrying out indoor full scene three-dimensional reconstruction based on consumer level depth camera includes acquisition module, filtering mould
Block, segment fusion and registration module and volume data Fusion Module.Wherein：
Acquisition module is used to carry out depth image collection to indoor scene using depth camera.
Filtration module is used to do the depth image for obtaining the treatment of self adaptation bilateral filtering.
The acquisition module is the equivalent of above-mentioned acquisition module.In actual applications, it is possible to use handheld consumer level
Depth camera Microsoft Kinect for Windows gather true indoor scene data.Then, the depth to collecting
Image carries out self adaptation bilateral filtering, and the noise behavior and its inner parameter according to depth camera are bilateral to set self adaptation automatically
Parameter in filtering method, therefore, the embodiment of the present invention can effectively remove noise and retain marginal information.
Segment fusion does automatic segmentation for view-based access control model content with registration module to data flow, and each segmentation carries out piecemeal
Fusion, closed loop detection is carried out between segmentation, and the result to closed loop detection does global optimization.
The segment fusion and registration module carry out the automatic Partitioning fusion of view-based access control model content, registering.
In a preferred embodiment, segment fusion is specifically included with registration module：Posture information acquisition module, point
Root module, rough registration module, smart registration module and optimization module.Wherein, posture information acquisition module is used to use
Kintinuous frameworks, carry out visual odometry estimation, obtain the camera posture information under every frame depth image.Segmentation module is used
Under will be by the corresponding cloud data back projection of every frame depth image to initial coordinate system according to camera posture information, after projection
The depth image of depth image and initial frame carry out similarity-rough set, initialize camera if similarity is less than similarity threshold
Pose, and carry out new segmentation.The PFFH geometric descriptions that rough registration module is used to extract in each segmentation cloud data are sub, and
Every two it is intersegmental between carry out rough registration；Smart registration module is used to carry out smart registration using GICP algorithms, to obtain between section and section
Matching relationship.Optimization module is used for the posture information using each section and the matching relationship between section and section, builds figure simultaneously
Figure optimization is carried out using G2O frameworks.
Preferably, above-mentioned optimization module is further used for using SLAC (Simultaneous Localization and
Calibration) pattern is to optimize non-rigid distortion, and the closed loop matching of deletion error is constrained using line processes.
Above-mentioned segment fusion carries out segment processing with registration module view-based access control model content to RGBD data flows, both can be effective
Ground solves the problems, such as visual odometry estimate generation accumulated error, again can by similar content mergence together, such that it is able to
Improve registration accuracy.
Volume data Fusion Module is used to be weighted volume data fusion according to the camera trace information after optimization, obtains scene
Threedimensional model.
The volume data Fusion Module defines unblind distance according to the noise behavior and area-of-interest of depth camera
The weight function of function realizes the holding of the geometric detail of body surface.
The experiment in the system of indoor full scene three-dimensional reconstruction is carried out based on consumer level depth camera to show：Based on consumption
The high-precision three-dimensional method for reconstructing of level depth camera, can obtain indoor scene model that is complete, accurate, becoming more meticulous, system tool
There is good robustness and autgmentability.
The above-mentioned system embodiment for carrying out indoor full scene three-dimensional reconstruction based on consumer level depth camera can be used for holding
Row carries out the embodiment of the method for indoor full scene three-dimensional reconstruction based on consumer level depth camera, its know-why, is solved
Technical problem and the technique effect of generation are similar, can refer to mutually；For convenience and simplicity of description, saved between each embodiment
Description identical part is omited.
It should be noted that above-described embodiment provide indoor full scene Three-dimensional Gravity is carried out based on consumer level depth camera
The system and method built when indoor full scene three-dimensional reconstruction is carried out, only with above-mentioned each functional module, unit or step draw
Point it is illustrated, for example, acquisition module in foregoing can also be as acquisition module, in actual applications, can basis
Need and above-mentioned functions distribution is completed by different functional modules, unit or step, mould that will be in the embodiment of the present invention
Block, unit or step are decomposed or combined again, for example, it is pre- acquisition module or collection and filtration module can be merged into data
Processing module.
So far, combined preferred embodiment shown in the drawings describes technical scheme, but, this area
Technical staff is it is easily understood that protection scope of the present invention is expressly not limited to these specific embodiments.Without departing from this
On the premise of the principle of invention, those skilled in the art can make equivalent change or replacement to correlation technique feature, these
Technical scheme after changing or replacing it is fallen within protection scope of the present invention.
Claims (20)
1. a kind of method that indoor full scene three-dimensional reconstruction is carried out based on consumer level depth camera, it is characterised in that the side
Method includes：
Obtain depth image；
Self adaptation bilateral filtering is carried out to the depth image；
The segment fusion and registration process of view-based access control model content are carried out to filtered depth image；
According to result, volume data fusion is weighted, so as to rebuild indoor full scene threedimensional model.
2. method according to claim 1, it is characterised in that described that self adaptation bilateral filtering is carried out to the depth image
Specifically include：
Self adaptation bilateral filtering is carried out according to following formula：
Wherein, the u and ukAny pixel and its field pixel on the depth image are represented respectively；The Z (u) and
Z (the uk) the correspondence u and u is represented respectivelykDepth value；It is describedRepresent corresponding depth value after filtering；It is described
W is represented in fieldOn normalization factor；The wsWith the wcIt is illustrated respectively in the Gauss of spatial domain and codomain filtering
Kernel function.
3. method according to claim 2, it is characterised in that the gaussian kernel function root filtered in spatial domain and codomain
Determine according to following formula：
Wherein, the δsWith the δcIt is respectively the variance of spatial domain and codomain gaussian kernel function；
Wherein, the δ and the δ determine according to following formula：
Wherein, the f represents the focal length of the depth camera, the KsWith the KcRepresent constant.
4. method according to claim 1, it is characterised in that described to be carried out in view-based access control model to filtered depth image
The segment fusion and registration process of appearance are specifically included：View-based access control model content is segmented to range image sequence, and to each point
Closed loop detection is carried out between Duan Jinhang segment fusions, and the segmentation, the result to closed loop detection does global optimization.
5. method according to claim 4, it is characterised in that the view-based access control model content is divided range image sequence
Section, and each segmentation is carried out carrying out closed loop detection between segment fusion, and the segmentation, the result to closed loop detection does global excellent
Change is specifically included：
View-based access control model content detection automatic segmentation method is segmented to range image sequence, by similar depth image content point
In being segmented at one, and segment fusion is carried out to each segmentation, determine the transformation relation between the depth image, and according to institute
State transformation relation and closed loop detection is done between section and section, to realize global optimization.
6. method according to claim 5, it is characterised in that the view-based access control model content detection automatic segmentation method is to depth
Degree image sequence is segmented, and similar depth image content is divided in being segmented at one, and is carried out piecemeal to each segmentation and melted
Close, determine the transformation relation between the depth image, and do closed loop detection between section and section according to the transformation relation, with
Global optimization is realized, is specifically included：
Using Kintinuous frameworks, visual odometry estimation is carried out, obtain the camera posture information under every frame depth image；
According to the camera posture information, will be by described per the corresponding cloud data back projection of frame depth image to initial coordinate system
Under, similarity-rough set is carried out with the depth image obtained after projection and the depth image of initial frame, and when similarity is less than similar
During degree threshold value, camera pose is initialized, be segmented；
PFFH geometric descriptions in each segmentation cloud data is extracted, and rough registration is carried out between every two sections, and used
GICP algorithms carry out smart registration, obtain the matching relationship between section and section；
Using the posture information and the described section of matching relationship and section between of each segmentation, structure is schemed and is entered using G2O frameworks
Row figure optimization, the camera trace information after being optimized, so as to realize the global optimization.
7. method according to claim 6, it is characterised in that described according to the camera posture information, will be by described every
The corresponding cloud data back projection of frame depth image under initial coordinate system, with the depth image obtained after projection and initial frame
Depth image carries out similarity-rough set, and when similarity is less than similarity threshold, initializes camera pose, is segmented, and has
Body includes：
Step 1：Calculate the similarity of every frame depth image and the first frame depth image；
Step 2：Judge the similarity whether less than similarity threshold；
Step 3：If so, being then segmented to the range image sequence；
Step 4：Using next frame depth image as the start frame depth image of next segmentation, and step 1 and step are repeated
2, until having processed all frame depth images.
8. method according to claim 7, it is characterised in that the step 1 is specifically included：
According to projection relation and the depth value of any frame depth image, and each pixel on the depth image is calculated using following formula
The first corresponding space three-dimensional point：
P=π-1(up,Z(up))
Wherein, the upIt is any pixel on the depth image；Z (the up) and the p represent the u respectivelypIt is corresponding
Depth value and the first space three-dimensional point；The π represents the projection relation；
The first space three-dimensional point rotation translation is transformed under world coordinate system according to following formula, obtains second space three-dimensional
Point：
Q=Tip
Wherein, the TiRepresent the i-th frame depth map correspondence space three-dimensional point to the rotation translation matrix under world coordinate system；The p
The first space three-dimensional point is represented, the q represents the second space three-dimensional point；The i takes positive integer；
According to following formula by the second space three-dimensional point back projection to two dimensional image plane, the depth image after being projected：
Wherein, the uqIt is the pixel after the corresponding projections of the q on depth image；The fx, the fy, the cxWith the cy
Represent the internal reference of depth camera；The xq、yq、zqRepresent the coordinate of the q；The transposition of the T representing matrixs；
The valid pixel number on the depth image after the start frame depth image and any frame projection is calculated respectively, and by two
Person's ratio is used as similarity.
9. method according to claim 1, it is characterised in that described according to result, is weighted volume data fusion,
Specifically included so as to rebuild indoor full scene threedimensional model：According to the result, using unblind distance function net
Lattice model merges the depth image of each frame, and represents three dimensions using voxel grid, so as to obtain indoor full scene three
Dimension module.
10. method according to claim 9, it is characterised in that according to the result, using unblind apart from letter
Number grid model merges the depth image of each frame, and represents three dimensions using voxel grid, so as to obtain indoor complete field
Scape threedimensional model, specifically includes：
Based on noise behavior and interest region, the unblind distance function is carried out using Volumetric method frameworks
Data weighting is merged；
Mesh model extractions are carried out using Marching cubes algorithms, so as to obtain the indoor full scene threedimensional model.
11. method according to claim 9 or 10, it is characterised in that the unblind distance function according to following formula come
It is determined that：
fi(v)=[K-1zi(u)[uT,1]T]z-[vi]z
Wherein, fiV () represents unblind distance function, namely grid to the distance on object model surface, the positive negative indication grid
It is to be blocked side or in visible side on surface, and zero crossing is exactly the point on surface；The K represents the interior of the camera
Parameter matrix；The u represents pixel；The ziU () represents the corresponding depth values of the pixel u；The viRepresent voxel.
12. methods according to claim 10, it is characterised in that the data weighting fusion is carried out according to following formula：
Wherein, the v represents voxel；The fi(v) and the wiV () represents the corresponding unblind distance of the voxel v respectively
Function and its weight function；The n takes positive integer；The F (v) represent unblind after fusion corresponding to the voxel v away from
From functional value；The W (v) represents the weight of the unblind distance function value corresponding to voxel v after merging；
Wherein, the weight function can determine according to following formula：
Wherein, the diRepresent the radius in interest region；The δsIt is the noise variance in depth data；The w is constant.
A kind of 13. systems that indoor full scene three-dimensional reconstruction is carried out based on consumer level depth camera, it is characterised in that the system
System includes：
Acquisition module, for obtaining depth image；
Filtration module, for carrying out self adaptation bilateral filtering to the depth image；
Segment fusion and registration module, segment fusion and registration for carrying out view-based access control model content to filtered depth image
Treatment；
Volume data Fusion Module, for according to result, being weighted volume data fusion, so as to rebuild indoor full scene three
Dimension module.
14. systems according to claim 13, it is characterised in that the filtration module specifically for：
Self adaptation bilateral filtering is carried out according to following formula：
Wherein, the u and ukAny pixel and its field pixel on the depth image are represented respectively；The Z (u) and
Z (the uk) the correspondence u and u is represented respectivelykDepth value；It is describedRepresent corresponding depth value after filtering；It is described
W is represented in fieldOn normalization factor；The wsWith the wcIt is illustrated respectively in the Gauss of spatial domain and codomain filtering
Kernel function.
15. systems according to claim 13, it is characterised in that the segment fusion and registration module specifically for：Base
Range image sequence is segmented in vision content, and each segmentation closed between segment fusion, and the segmentation
Ring detects that the result to closed loop detection does global optimization.
16. systems according to claim 15, it is characterised in that the segment fusion and registration module also particularly useful for：
View-based access control model content detection automatic segmentation method is segmented to range image sequence, by similar depth image content point
In being segmented at one, segment fusion is carried out to each segmentation, determine the transformation relation between the depth image, and according to described
Transformation relation does closed loop detection between section and section, to realize global optimization.
17. systems according to claim 16, it is characterised in that the segment fusion is specifically included with registration module：
Camera posture information acquiring unit, for using Kintinuous frameworks, carries out visual odometry estimation, obtains every frame depth
Camera posture information under degree image；
Segmenting unit, for that according to the camera posture information, will be thrown per the corresponding cloud data of frame depth image is counter by described
Shadow carries out similarity-rough set, and work as under initial coordinate system with the depth image obtained after projection and the depth image of initial frame
When similarity is less than similarity threshold, camera pose is initialized, be segmented；
Registration unit, for extracting PFFH geometric descriptions in each segmentation cloud data, and is carried out thick between every two sections
Registration, and smart registration is carried out using GICP algorithms, obtain the matching relationship between section and section；
Optimization unit, for posture information and the described section of matching relationship and section between using each segmentation, builds figure simultaneously
Figure optimization is carried out using G2O frameworks, the camera trace information after being optimized, so as to realize the global optimization.
18. systems according to claim 17, it is characterised in that the segmenting unit is specifically included：
Computing unit, the similarity for calculating every frame depth image and the first frame depth image；
Judging unit, for judging the similarity whether less than similarity threshold；
Segmentation subelement, for when the similarity is less than similarity threshold, being segmented to the range image sequence；
Processing unit, for using next frame depth image as the start frame depth image of next segmentation, and repeating calculating
Unit and judging unit, until having processed all frame depth images.
19. systems according to claim 13, it is characterised in that the volume data Fusion Module specifically for：According to institute
Result is stated, the depth image of each frame is merged using unblind distance function grid model, and carry out table using voxel grid
Show three dimensions, so as to obtain indoor full scene threedimensional model.
20. systems according to claim 19, it is characterised in that the volume data Fusion Module is specifically included：
Weighted Fusion unit, for based on noise behavior and interest region, being carried out using Volumetric method frameworks described
Unblind distance function data weighting is merged；
Extraction unit is described indoor complete so as to obtain for carrying out Mesh model extractions using Marching cubes algorithms
Scene threedimensional model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710051366.5A CN106910242B (en) | 2017-01-23 | 2017-01-23 | Method and system for carrying out indoor complete scene three-dimensional reconstruction based on depth camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710051366.5A CN106910242B (en) | 2017-01-23 | 2017-01-23 | Method and system for carrying out indoor complete scene three-dimensional reconstruction based on depth camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106910242A true CN106910242A (en) | 2017-06-30 |
| CN106910242B CN106910242B (en) | 2020-02-28 |
Family
ID=59207090
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710051366.5A Active CN106910242B (en) | 2017-01-23 | 2017-01-23 | Method and system for carrying out indoor complete scene three-dimensional reconstruction based on depth camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106910242B (en) |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107067470A (en) * | 2017-04-05 | 2017-08-18 | 东北大学 | Portable three-dimensional reconstruction of temperature field system based on thermal infrared imager and depth camera |
| CN107833270A (en) * | 2017-09-28 | 2018-03-23 | 浙江大学 | Real-time object dimensional method for reconstructing based on depth camera |
| CN108053476A (en) * | 2017-11-22 | 2018-05-18 | 上海大学 | A kind of human parameters measuring system and method rebuild based on segmented three-dimensional |
| CN108133496A (en) * | 2017-12-22 | 2018-06-08 | 北京工业大学 | A kind of dense map creating method based on g2o Yu random fern |
| CN108227707A (en) * | 2017-12-25 | 2018-06-29 | 清华大学苏州汽车研究院（吴江） | Automatic Pilot method based on laser radar and end-to-end deep learning method |
| CN108537876A (en) * | 2018-03-05 | 2018-09-14 | 清华-伯克利深圳学院筹备办公室 | Three-dimensional rebuilding method, device, equipment based on depth camera and storage medium |
| CN108550181A (en) * | 2018-03-12 | 2018-09-18 | 中国科学院自动化研究所 | It is tracked and dense method for reconstructing, system and equipment online in mobile device |
| CN108564616A (en) * | 2018-03-15 | 2018-09-21 | 中国科学院自动化研究所 | Method for reconstructing three-dimensional scene in the rooms RGB-D of fast robust |
| CN108961176A (en) * | 2018-06-14 | 2018-12-07 | 中国科学院半导体研究所 | Range gating three-dimensional imaging is adaptive bilateral with reference to restorative procedure |
| WO2019042028A1 (en) * | 2017-09-01 | 2019-03-07 | 叠境数字科技（上海）有限公司 | All-around spherical light field rendering method |
| CN109472820A (en) * | 2018-10-19 | 2019-03-15 | 清华大学 | Monocular RGB-D camera real-time face method for reconstructing and device |
| CN109492656A (en) * | 2017-09-11 | 2019-03-19 | 百度在线网络技术（北京）有限公司 | Method and apparatus for output information |
| CN109737974A (en) * | 2018-12-14 | 2019-05-10 | 中国科学院深圳先进技术研究院 | A kind of 3D navigational semantic map updating method, device and equipment |
| CN109819173A (en) * | 2017-11-22 | 2019-05-28 | 浙江舜宇智能光学技术有限公司 | Depth integration method and TOF camera based on TOF imaging system |
| CN110007754A (en) * | 2019-03-06 | 2019-07-12 | 清华大学 | The real-time reconstruction method and device of hand and object interactive process |
| CN110148217A (en) * | 2019-05-24 | 2019-08-20 | 北京华捷艾米科技有限公司 | A kind of real-time three-dimensional method for reconstructing, device and equipment |
| CN112053435A (en) * | 2020-10-12 | 2020-12-08 | 武汉艾格美康复器材有限公司 | Self-adaptive real-time human body three-dimensional reconstruction method |
| CN112598778A (en) * | 2020-08-28 | 2021-04-02 | 国网陕西省电力公司西咸新区供电公司 | VR three-dimensional reconstruction technology based on improved texture mapping algorithm |
| CN113902846A (en) * | 2021-10-11 | 2022-01-07 | 岱悟智能科技(上海)有限公司 | Indoor three-dimensional modeling method based on monocular depth camera and mileage sensor |
| CN113989451A (en) * | 2021-10-28 | 2022-01-28 | 北京百度网讯科技有限公司 | High-precision map construction method and device and electronic equipment |
| CN115358156A (en) * | 2022-10-19 | 2022-11-18 | 南京耀宇视芯科技有限公司 | Adaptive indoor scene modeling and optimization analysis system |
| CN116563118A (en) * | 2023-07-12 | 2023-08-08 | 浙江华诺康科技有限公司 | Endoscopic image stitching method and device and computer equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101751697A (en) * | 2010-01-21 | 2010-06-23 | 西北工业大学 | Three-dimensional scene reconstruction method based on statistical model |
| CN103559737A (en) * | 2013-11-12 | 2014-02-05 | 中国科学院自动化研究所 | Object panorama modeling method |
| CN103927717A (en) * | 2014-03-28 | 2014-07-16 | 上海交通大学 | Depth image recovery method based on improved bilateral filters |
| CN105913489A (en) * | 2016-04-19 | 2016-08-31 | 东北大学 | Indoor three-dimensional scene reconstruction method employing plane characteristics |
| CN106056664A (en) * | 2016-05-23 | 2016-10-26 | 武汉盈力科技有限公司 | Real-time three-dimensional scene reconstruction system and method based on inertia and depth vision |
-
2017
- 2017-01-23 CN CN201710051366.5A patent/CN106910242B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101751697A (en) * | 2010-01-21 | 2010-06-23 | 西北工业大学 | Three-dimensional scene reconstruction method based on statistical model |
| CN103559737A (en) * | 2013-11-12 | 2014-02-05 | 中国科学院自动化研究所 | Object panorama modeling method |
| CN103927717A (en) * | 2014-03-28 | 2014-07-16 | 上海交通大学 | Depth image recovery method based on improved bilateral filters |
| CN105913489A (en) * | 2016-04-19 | 2016-08-31 | 东北大学 | Indoor three-dimensional scene reconstruction method employing plane characteristics |
| CN106056664A (en) * | 2016-05-23 | 2016-10-26 | 武汉盈力科技有限公司 | Real-time three-dimensional scene reconstruction system and method based on inertia and depth vision |
Non-Patent Citations (3)
| Title |
|---|
| SHAHRAM IZADI等: "《KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera》", 《UIST’11》 * |
| THOMAS WHELAN等: "《Kintinuous: Spatially Extended KinectFusion》", 《COMPUTER SCIENCE AND ARTIFICIAL INTELLIGENCE LABORATORY》 * |
| 陈晓明，蒋乐天，应忍冬: "《基于Kinect深度信息的实时三维重建和滤波算法研究》", 《计算机应用研究》 * |
Cited By (39)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107067470A (en) * | 2017-04-05 | 2017-08-18 | 东北大学 | Portable three-dimensional reconstruction of temperature field system based on thermal infrared imager and depth camera |
| CN107067470B (en) * | 2017-04-05 | 2019-09-06 | 东北大学 | Portable three-dimensional reconstruction of temperature field system based on thermal infrared imager and depth camera |
| GB2584753A (en) * | 2017-09-01 | 2020-12-16 | Plex Vr Digital Tech Shanghai Co Ltd | All-around spherical light field rendering method |
| US10909752B2 (en) | 2017-09-01 | 2021-02-02 | Plex-Vr Digital Technology (Shanghai) Co., Ltd. | All-around spherical light field rendering method |
| GB2584753B (en) * | 2017-09-01 | 2021-05-26 | Plex Vr Digital Tech Shanghai Co Ltd | All-around spherical light field rendering method |
| WO2019042028A1 (en) * | 2017-09-01 | 2019-03-07 | 叠境数字科技（上海）有限公司 | All-around spherical light field rendering method |
| CN109492656B (en) * | 2017-09-11 | 2022-04-29 | 阿波罗智能技术(北京)有限公司 | Method and apparatus for outputting information |
| CN109492656A (en) * | 2017-09-11 | 2019-03-19 | 百度在线网络技术（北京）有限公司 | Method and apparatus for output information |
| CN107833270B (en) * | 2017-09-28 | 2020-07-03 | 浙江大学 | Real-time object three-dimensional reconstruction method based on depth camera |
| CN107833270A (en) * | 2017-09-28 | 2018-03-23 | 浙江大学 | Real-time object dimensional method for reconstructing based on depth camera |
| CN109819173B (en) * | 2017-11-22 | 2021-12-03 | 浙江舜宇智能光学技术有限公司 | Depth fusion method based on TOF imaging system and TOF camera |
| CN108053476B (en) * | 2017-11-22 | 2021-06-04 | 上海大学 | Human body parameter measuring system and method based on segmented three-dimensional reconstruction |
| CN108053476A (en) * | 2017-11-22 | 2018-05-18 | 上海大学 | A kind of human parameters measuring system and method rebuild based on segmented three-dimensional |
| CN109819173A (en) * | 2017-11-22 | 2019-05-28 | 浙江舜宇智能光学技术有限公司 | Depth integration method and TOF camera based on TOF imaging system |
| CN108133496A (en) * | 2017-12-22 | 2018-06-08 | 北京工业大学 | A kind of dense map creating method based on g2o Yu random fern |
| CN108227707A (en) * | 2017-12-25 | 2018-06-29 | 清华大学苏州汽车研究院（吴江） | Automatic Pilot method based on laser radar and end-to-end deep learning method |
| CN108227707B (en) * | 2017-12-25 | 2021-11-26 | 清华大学苏州汽车研究院（吴江） | Automatic driving method based on laser radar and end-to-end deep learning method |
| CN108537876A (en) * | 2018-03-05 | 2018-09-14 | 清华-伯克利深圳学院筹备办公室 | Three-dimensional rebuilding method, device, equipment based on depth camera and storage medium |
| CN108537876B (en) * | 2018-03-05 | 2020-10-16 | 清华-伯克利深圳学院筹备办公室 | Three-dimensional reconstruction method, device, equipment and storage medium |
| CN108550181B (en) * | 2018-03-12 | 2020-07-31 | 中国科学院自动化研究所 | Method, system and equipment for online tracking and dense reconstruction on mobile equipment |
| CN108550181A (en) * | 2018-03-12 | 2018-09-18 | 中国科学院自动化研究所 | It is tracked and dense method for reconstructing, system and equipment online in mobile device |
| CN108564616B (en) * | 2018-03-15 | 2020-09-01 | 中国科学院自动化研究所 | Fast robust RGB-D indoor three-dimensional scene reconstruction method |
| CN108564616A (en) * | 2018-03-15 | 2018-09-21 | 中国科学院自动化研究所 | Method for reconstructing three-dimensional scene in the rooms RGB-D of fast robust |
| CN108961176A (en) * | 2018-06-14 | 2018-12-07 | 中国科学院半导体研究所 | Range gating three-dimensional imaging is adaptive bilateral with reference to restorative procedure |
| CN108961176B (en) * | 2018-06-14 | 2021-08-03 | 中国科学院半导体研究所 | Self-adaptive bilateral reference restoration method for range-gated three-dimensional imaging |
| CN109472820A (en) * | 2018-10-19 | 2019-03-15 | 清华大学 | Monocular RGB-D camera real-time face method for reconstructing and device |
| CN109737974A (en) * | 2018-12-14 | 2019-05-10 | 中国科学院深圳先进技术研究院 | A kind of 3D navigational semantic map updating method, device and equipment |
| CN110007754A (en) * | 2019-03-06 | 2019-07-12 | 清华大学 | The real-time reconstruction method and device of hand and object interactive process |
| CN110148217A (en) * | 2019-05-24 | 2019-08-20 | 北京华捷艾米科技有限公司 | A kind of real-time three-dimensional method for reconstructing, device and equipment |
| CN112598778A (en) * | 2020-08-28 | 2021-04-02 | 国网陕西省电力公司西咸新区供电公司 | VR three-dimensional reconstruction technology based on improved texture mapping algorithm |
| CN112598778B (en) * | 2020-08-28 | 2023-11-14 | 国网陕西省电力公司西咸新区供电公司 | VR three-dimensional reconstruction method based on improved texture mapping algorithm |
| CN112053435A (en) * | 2020-10-12 | 2020-12-08 | 武汉艾格美康复器材有限公司 | Self-adaptive real-time human body three-dimensional reconstruction method |
| CN113902846A (en) * | 2021-10-11 | 2022-01-07 | 岱悟智能科技(上海)有限公司 | Indoor three-dimensional modeling method based on monocular depth camera and mileage sensor |
| CN113902846B (en) * | 2021-10-11 | 2024-04-12 | 岱悟智能科技(上海)有限公司 | Indoor three-dimensional modeling method based on monocular depth camera and mileage sensor |
| CN113989451A (en) * | 2021-10-28 | 2022-01-28 | 北京百度网讯科技有限公司 | High-precision map construction method and device and electronic equipment |
| CN113989451B (en) * | 2021-10-28 | 2024-04-09 | 北京百度网讯科技有限公司 | High-precision map construction method and device and electronic equipment |
| CN115358156A (en) * | 2022-10-19 | 2022-11-18 | 南京耀宇视芯科技有限公司 | Adaptive indoor scene modeling and optimization analysis system |
| CN115358156B (en) * | 2022-10-19 | 2023-03-24 | 南京耀宇视芯科技有限公司 | Adaptive indoor scene modeling and optimization analysis system |
| CN116563118A (en) * | 2023-07-12 | 2023-08-08 | 浙江华诺康科技有限公司 | Endoscopic image stitching method and device and computer equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106910242B (en) | 2020-02-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106910242A (en) | The method and system of indoor full scene three-dimensional reconstruction are carried out based on depth camera | |
| CN111815757B (en) | Large member three-dimensional reconstruction method based on image sequence | |
| CN103106688B (en) | Based on the indoor method for reconstructing three-dimensional scene of double-deck method for registering | |
| CN101404091B (en) | Three-dimensional human face reconstruction method and system based on two-step shape modeling | |
| CN105005755B (en) | Three-dimensional face identification method and system | |
| CN103400409B (en) | A kind of coverage 3D method for visualizing based on photographic head attitude Fast estimation | |
| CN101398886B (en) | Rapid three-dimensional face identification method based on bi-eye passiveness stereo vision | |
| CN109544677A (en) | Indoor scene main structure method for reconstructing and system based on depth image key frame | |
| CN104616286B (en) | Quick semi-automatic multi views depth restorative procedure | |
| CN107369183A (en) | Towards the MAR Tracing Registration method and system based on figure optimization SLAM | |
| CN107146201A (en) | A kind of image split-joint method based on improvement image co-registration | |
| CN103700101B (en) | Non-rigid brain image registration method | |
| CN109658515A (en) | Point cloud gridding method, device, equipment and computer storage medium | |
| CN104794722A (en) | Dressed human body three-dimensional bare body model calculation method through single Kinect | |
| CN106997605A (en) | It is a kind of that the method that foot type video and sensing data obtain three-dimensional foot type is gathered by smart mobile phone | |
| CN106155299B (en) | A kind of pair of smart machine carries out the method and device of gesture control | |
| CN113298934B (en) | Monocular visual image three-dimensional reconstruction method and system based on bidirectional matching | |
| CN103854301A (en) | 3D reconstruction method of visible shell in complex background | |
| WO2018133119A1 (en) | Method and system for three-dimensional reconstruction of complete indoor scene based on depth camera | |
| CN107369131A (en) | Conspicuousness detection method, device, storage medium and the processor of image | |
| Long et al. | Neuraludf: Learning unsigned distance fields for multi-view reconstruction of surfaces with arbitrary topologies | |
| CN103454276A (en) | Textile form and style evaluation method based on dynamic sequence image | |
| CN107358645A (en) | Product method for reconstructing three-dimensional model and its system | |
| Xue et al. | Symmetric piecewise planar object reconstruction from a single image | |
| CN110176064A (en) | A kind of photogrammetric main object automatic identifying method for generating threedimensional model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CB03 | Change of inventor or designer information |
Inventor after: Gao Wei Inventor after: Li Jianwei Inventor after: Wu Yihong Inventor before: Li Jianwei Inventor before: Gao Wei Inventor before: Wu Yihong |
|
| CB03 | Change of inventor or designer information |