CN103106688A - Indoor three-dimensional scene rebuilding method based on double-layer rectification method - Google Patents

Abstract

The invention belongs to the crossing field of computer vision and intelligent robots, relates to an indoor three-dimensional scene rebuilding method based on a double-layer rectification method and solves the problems that an existing indoor scene rebuilding method is expensive in required equipment, high in computation complexity and poor in real-time performance. The indoor three-dimensional scene rebuilding method based on the double-layer rectification method comprises Kinect calibration, SURF feature point extraction and matching, mapping from a feature point pair to a three-dimensional space point pair, three dimensional space point double-layer rectification based on random sample consensus (RANSAC) and inductively coupled plasma (ICP) methods and scene updating. According to the indoor three-dimensional scene rebuilding method based on the double-layer rectification method, the Kinect is adopted to obtain environmental data, and the double-layer rectification method is provided based on the RANSAC and the ICP. Indoor three-dimensional scene rebuilding which is economical and rapid is achieved, and the real-time performance of rebuilding algorithm and the rebuilding precision are effectively improved. The indoor three-dimensional scene rebuilding method based on the double-layer rectification method is applicable to the service robot field and other computer vision fields which are relative to the three-dimensional scene rebuilding.

Description

Indoor method for reconstructing three-dimensional scene based on double-deck method for registering

Technical field

The invention belongs to the crossing domain of computer vision and intelligent robot, relate to a kind of indoor environment three-dimensional reconstruction, relate in particular to a kind of method for reconstructing of the indoor scene on a large scale based on double-deck method for registering.

Background technology

In recent years, along with the development of infotech, the demand of 3 D scene rebuilding technology is constantly increased, economy indoor method for reconstructing three-dimensional scene fast becomes the guardian technique problem that numerous areas needs to be resolved hurrily.In family's service robot field, the Intelligent home service robot market demand that aging population causes is day by day strong.At present, on market, most of service robot is because can't providing single simple service by the perception three-dimensional environment under specific scene, and this problem is seriously restricting the development of home services Robot industry.

3 D scene rebuilding is one of the study hotspot problem in the fields such as computer vision, intelligent robot, virtual reality.Traditional three-dimensional rebuilding method can be divided into two classes according to the mode difference of obtaining three-dimensional data: based on the three-dimensional rebuilding method of laser scanner technique with based on the three-dimensional rebuilding method of vision.Still deposit larger limitation for the existing method of indoor large-scale 3 D scene rebuilding problem.

Based on the three-dimensional rebuilding method of laser, obtain depth data or the range image of scene by laser scanner, utilize the registration of depth data to realize aliging of frame data and global data.So just obtained the geological information of three-dimensional scenic, need to obtain the texture information of scene and be mapped to and reconstruct on geometric model by increasing a video camera, this will solve one by the mapping problems of photo to how much.Although can obtain the 3-D geometric model of degree of precision based on the three-dimensional rebuilding method of laser scanning, but the difficulty of texture is larger, thereby it is more difficult to generate realistic three-dimensional model, the laser equipment price is high simultaneously, generally be applied in the fields such as digital archaeology, topographic(al) reconnaissance, digital museum, be difficult at large-scale civil area universal.

Three-dimensional rebuilding method based on vision, namely adopt computer vision methods to carry out the object dimensional Model Reconstruction, refer to utilize digital camera as imageing sensor, the technology such as integrated use image processing vision calculating are carried out three-dimensional non-contact measurement, obtain the three-dimensional information of object with computer program.It is advantageous that not limited by body form, rebuild speed, can realize full-automatic or semi-automatic modeling etc., is an important development direction of three-dimensional reconstruction.Difference according to using the video camera number can be divided into monocular vision method, binocular vision method, trinocular vision method or used for multi-vision visual method.The monocular vision method uses a video camera to carry out three-dimensional reconstruction, derives depth information by two dimension characteristic of image, and these two dimensional characters comprise light and shade, texture, focus, profile etc.Its advantage is that device structure is simple, uses single width or several images just can reconstruct the object dimensional model.But more satisfactoryization of condition that usually requires, practical situations is not very desirable, the reconstruction effect is general.The binocular vision method also claims stereo vision method, and binocular parallax information is converted to depth information.Its advantage is that method is ripe, can stably obtain reconstructed results preferably; Not enough is that operand is still bigger than normal, and in the situation that the larger reconstruction of parallax range successful reduction.The basic thought of used for multi-vision visual method is to provide extra constraint by increasing video camera, avoids problem in binocular vision with this.Its advantage is to rebuild effect to be better than the binocular vision method, but device structure is more complicated, and cost is higher, and is also more difficult in control.

In recent years, along with RGBD(colour and the degree of depth) development of sensor technology, the Kinect that releases such as Microsoft is for 3 D scene rebuilding provides new scheme.At present about the research of Kinect three-dimensional rebuilding method having obtained some achievements aspect the three-dimensional reconstruction of single object, still be in the starting stage in the research aspect the indoor scene reconstruction.The people such as RichardA.Newcombe adopt Kinect to obtain environmental information, utilize the ICP method, realize the three-dimensional reconstruction of environmental information.Because the method realizes on GPU hardware, the GPU hardware configuration to be had relatively high expectations, and be subjected to the restriction of GPU internal memory, the scope that can only rebuild 3m * 3m * 3m can't satisfy the demand of indoor three-dimensional scenic establishment on a large scale.

Summary of the invention

In order to overcome the problem that exists in above-mentioned three-dimensional rebuilding method, the invention provides a kind of economy fast, based on the indoor method for reconstructing three-dimensional scene of double-deck method for registering.

The technical solution used in the present invention is as follows:

Utilize Kinect to obtain RGB and the depth image information of environment, by extracting the SURF unique point of RGB image, with Feature Points Matching information as associated data, in conjunction with random sampling consistance (Random sample Consensus, RANSAC) point of proximity (the Iterative closest point of method and iteration, ICP) method proposes the double-deck method for registering of a kind of three-dimensional data.The method mainly comprises following content: the first, and utilize the RANSAC method to obtain the rotation translation transformation matrix of adjacent two frames (Frame-To-Frame) three-dimensional data, accumulate the relative position variation that this result is obtained Kinect.By setting threshold, increase by one frame data when the Kinect change in location surpasses a certain size are key frame (KeyFrame) with this data setting and complete first registration; The second, utilize the ICP method to obtain the accurate transformation matrix of adjacent key frame (KeyFrame-To-KeyFrame), complete accuracy registration.Utilize KeyFrame data that double-deck method for registering obtains and the transformation matrix between adjacent KeyFrame, complete the reconstruction of three-dimensional environment.

Based on the indoor method for reconstructing three-dimensional scene of double-deck method for registering, it comprises the following steps:

Step 1 is carried out Kinect and is demarcated.

In image measurement process and machine vision applications, for determining three-dimensional geometry position and its mutual relationship between corresponding point in image of surperficial certain point of space object, must set up the geometric model of camera imaging, these geometric model parameters consist of camera parameters.These parameters must just can obtain with calculating by experiment under most of conditions, and this process of finding the solution parameter just is referred to as camera calibration (or camera calibration).In image measurement and machine vision applications, the demarcation of camera parameters is unusual the key link, and the precision of its calibration result and stability directly affect the accuracy of net result.

Kinect is the XBOX360 body sense periphery peripheral hardware of a kind of Microsoft issue, and the degree of depth and colour (RGB) image information is provided simultaneously.Depth information utilizes thermal camera to adopt active mode to obtain, each frame is comprised of the 640*480 pixel, and the investigation depth scope is 0.5 ~ 4.0 meter, and the regulation of longitudinal angle scope is 43 °, the lateral angles scope is 57 °, can obtain the depth information of object in 6 square metres of scopes.The RGB camera of a 640*480 pixel is housed on Kinect simultaneously.Provide simultaneously this characteristic of RGB information and depth information most important for three-dimensional reconstruction, facilitate depth information to align with RGB information.

The calibrating parameters of Kinect sensor comprises thermal camera (depth transducer) internal reference, three parts of the outer ginseng between RGB video camera internal reference and thermal camera and RGB video camera.The present invention adopts the plane reference method of Zhang Zhengyou that the RGB video camera is demarcated.The data that outer ginseng between thermal camera intrinsic parameter and thermal camera and RGB video camera uses official of Microsoft to provide.

Step 2, the extraction of unique point and coupling.

Feature extraction: by analysis image information, determine whether each point in image belongs to a characteristics of image.The result of feature extraction is that the point on image is divided into different subsets, and these subsets often belong to isolated point, continuous curve or continuous zone.SURF(Speeded-Up RobustFeatures) unique point is the most popular method of present computed image feature, and the feature that the method is extracted has that yardstick is constant, the performance of invariable rotary, simultaneously illumination variation and affine, perspective transform is had unchangeability.SURF all surmounts or approaches the same class methods that in the past proposed, and have obvious advantage on computing velocity aspect 3 of multiplicities, uniqueness, robustness.

The present invention extracts the SURF unique point of RGB image, comprises that feature point detection and unique point describe two parts.Employing is carried out Feature Points Matching based on the nearest neighbor algorithm of Euclidean distance, utilizes the data structure of K-D tree to search for, and is more right than determining whether to accept this coupling according to the distance of nearest two unique points.

Step 3, images match are put the three-dimensional coordinate mapping.

Set up transformational relation between the plane of delineation and space three-dimensional point coordinate according to the calibration model of Kinect, determine that three dimensions puts the projection model of the plane of delineation, with following function representation:

u＝π(p)

Wherein, p is the three dimensions point, and u is plane of delineation coordinate, and π (p) representation space three-dimensional point is to the mapping function of the plane of delineation.Coupling by image characteristic point obtains in the corresponding point of the plane of delineation pair, and the projection model that utilizes three dimensions to put the plane of delineation obtains three dimensions point coordinate corresponding to image characteristic point, further obtains three-dimensional point corresponding to two frame data pair.

Step 4 is based on the double-deck registration of the three dimensions point of RANSAC and ICP.

Registration refers to the interior coupling with the different images geographic coordinate that different imaging means was obtained of the same area.Comprise geometric correction, projective transformation and three kinds of processing of unified engineer's scale.Registration results is expressed as matrix:

T _cw＝[R _cw,t _cw]

Wherein, subscript " cw " expression is tied to current Kinect coordinate system, R from world coordinates _cwThe expression world coordinates is tied to the rotation matrix of current coordinate system, t _cwThe expression world coordinates is tied to the translation of current coordinate system.T _cwDescribed Kinect and rotated translation relation under world coordinate system.Put p under the Kinect coordinate system _cTo world coordinates p _wTransformation relation be:

p _c＝T _cwp _w

The problem high for three dimensional point cloud method for registering complexity, that calculated amount is large the present invention is based on RANSAC and ICP method, proposes a kind of double-deck method for registering, is comprised of first registration and accuracy registration two parts.First registration adopts RANSAC, to obtain KeyFrame and Relative Transformation matrix; Adopt ICP to realize accuracy registration, realize the alignment of three-dimensional data points and provide three-dimension varying information accurately for upgrading three-dimensional scenic on the basis of first registration.

Step 5, scene update.

Each frame three-dimensional data of obtaining by Kinect approximately comprises 250,000 points.There is very large information redundancy in adjacent two frame data, in order to improve the sharpness of reconstructed results, and to the description that the three-dimensional map that generates provides an essence to want, the burden of minimizing system aspect internal memory, the present invention adopts KeyFrame Data Update three-dimensional scenic.

The invention has the beneficial effects as follows: adopt Kinect to obtain environmental data, for the characteristics of Kinect sensor, propose a kind of double-deck method for registering based on RANSAC and ICP, realize quick indoor 3 D scene rebuilding on a large scale.Effectively solve three-dimensional rebuilding method cost and real time problems, improved reconstruction precision.

Description of drawings

Fig. 1 is the indoor method for reconstructing three-dimensional scene block diagram based on Kinect;

Fig. 2 is Kinect coordinate system schematic diagram;

Fig. 3 is the double-deck method for registering process flow diagram based on RANSAC and ICP;

Fig. 4 creates the actual environment schematic diagram of three-dimensional scenic for using the present invention: in figure, (a) for the experiment real scene, (b) be the two-dimensional geometry schematic diagram of experimental situation;

Fig. 5 creates the result schematic diagram of three-dimensional scenic for using the present invention.

Embodiment

The present invention is described in further detail by reference to the accompanying drawings.As shown in Figure 1, the present invention includes following step:

Step 1 is carried out Kinect and is demarcated, and concrete grammar is as follows:

(1) print a chessboard template.The present invention adopts an A4 paper, chessboard be spaced apart 0.25cm.

(2) from a plurality of angle shot chessboards.During shooting, should allow chessboard take screen, and 8 template picture be taken altogether in each angle that guarantees chessboard in screen as far as possible.

(3) detect unique point in image, i.e. each black point of crossing of chessboard.

(4) obtain the parameter that Kinect demarcates.

The internal reference matrix K of thermal camera _ir:

$K_{\mathrm{ir}}=\left[\begin{array}{ccc}{f}_{\mathrm{uIR}}& 0& u_{\mathrm{IR}}\\ 0& f_{\mathrm{vIR}}& v_{\mathrm{IR}}\\ 0& 0& 1\end{array}\right]$

Wherein, (f _uIR, f _vIR) be the focal length of thermal camera, value (5,5), (u _IR, v _IR) be thermal camera as the planar central coordinate, value (320,240).

The Intrinsic Matrix K of RGB video camera _c:

$K_c=\left[\begin{array}{ccc}{f}_u& 0& u_0\\ 0& f_v& v_0\\ 0& 0& 1\end{array}\right]$

Wherein, (f _u, f _v) be the focal length of RGB video camera, (u ₀, v ₀) be that the RGB video camera is as the planar central coordinate.

External parameter between thermal camera and RGB video camera is:

T＝[R _IRc,t _IRc]

Wherein, R _IRcBe rotation matrix, t _bcTranslation vector, the parameter of directly using official of Microsoft to provide:

$R_{\mathrm{IRc}}=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$

t _IRc＝[0.075 0] ^T

In the present invention, the Kinect coordinate system as shown in Figure 2, for y axle positive dirction, is upwards forward z axle positive dirction, is to the right the x positive dirction.The initial point position of Kinect is set as the world coordinate system initial point, and the X of world coordinate system, Y, Z direction are identical with x, y, the z direction of Kinect initial point position.

Step 2, the extraction of unique point and coupling, method is as follows:

(1) obtain integral image.Integral image refer to calculate given all pixels of gray level image accumulation and, for the integration I (X) of certain some X=(x, y) in image be:

$I_{\mathrm{\Σ}}\left(X\right)=\underset{i=0}{\overset{i\≤x}{\mathrm{\Σ}}}\underset{j=0}{\overset{j\≤y}{\mathrm{\Σ}}}I(i,j)$

Calculate the gray-scale value sum of a rectangular area with 3 plus and minus calculations in integral image, irrelevant with the area of rectangle.Can see in the step of back, the convolution mask that uses in the SURF feature point extraction is frame shape template, has greatly improved operation efficiency.

(2) ask for approximate Hessian matrix H _ApproxFor certain some X=(x, y) in image I, the Hessian matrix H (X, s) on the s yardstick that X is ordered is defined as:

$H(X,s)=\left[\begin{array}{cc}{L}_{\mathrm{xx}}(X,s)& L_{\mathrm{xy}}(X,s)\\ L_{\mathrm{xy}}(X,s)& L_{\mathrm{yy}}(X,s)\end{array}\right]$

Wherein, L _xx(X, s), L _xy(X, s), L _yy(X, s) expression Gauss second-order partial differential coefficient is in the convolution of X place and image I.Use the second order Gauss filtering in the approximate Hessian of replacement of square frame filtering matrix.The value of frame shape Filtering Template after with image convolution is respectively D _xx, D _yy, D _xy, further replace L with them _xx, L _yy, L _xyObtain approximate Hessian matrix H _Approx, its determinant is:

det(H _approx)＝D _xxD _yy-(wD _xy) ²

Wherein, w is weight coefficient, and value is 0.9 in enforcement of the present invention.

(3) location feature point.The feature point detection of SURF is based on the Hessian matrix, according to the local maximum location feature point position of Hessian matrix determinant.

With the frame shape wave filter of different size, original image is processed and obtained the yardstick image pyramid, according to H _ApproxObtain the extreme value that the scalogram picture is located at (X, s).

Use frame shape wave filter to build metric space, in every single order, select the scalogram picture of 4 layers, the structure parameter on 4 rank sees Table 1.

Size (the unit: s) of 16 templates in quadravalence before table 1 metric space

Use H _ApproxMatrix is obtained extreme value, in 3 dimension (X, s) metric spaces, each regional area of 3 * 3 * 3 is carried out non-maximum value suppress (keep maximum value, other values are set to 0).Elect response as unique point greater than the point of 26 neighborhood values.Utilize the quadratic fit function that unique point is accurately located, fitting function D (X) is:

$D\left(X\right)=D+\frac{{\∂D}^T}{\∂X}X+\frac{1}{2}{X}^T\frac{\∂D}{{\∂X}^2}X$

So far, position, the yardstick information (X, s) of unique point have been obtained.

(4) determine the direction character of unique point.With the Haar wavelet filter, circular neighborhood is processed, obtained the response of the corresponding x of each point, y direction in this neighborhood.Choose the Gaussian function (σ gets 2s, and s is yardstick corresponding to this unique point) centered by unique point, these responses are weighted, the vector of search length maximum, its direction is the corresponding direction of this unique point.

(5) construction feature description vectors.Determine a foursquare neighborhood centered by unique point, the length of side is got 20s, is the unique point direction setting y direction of principal axis of this neighborhood.Square area is divided into 4 * 4 sub regions, processes with the Haar wavelet filter in each subregion that (Haar small echo template size is 2s * 2s).Use d _xThe little wave response of Haar of expression horizontal direction is used d _yThe little wave response of Haar of expression vertical direction.For all d _x, d _yIn order to the Gaussian function weighting centered by unique point, the σ of this Gaussian function is 3.3s.In every sub regions respectively to d _x, d _y, d _x|, d _y| summation obtains 4 dimensional vector V (∑ d _x, ∑ d _y, ∑ d _y|, ∑ d _y|).The vector of 4 * 4 sub regions is coupled together just obtained one 64 vector of tieing up, this vector has rotation, yardstick unchangeability, then after carrying out normalization, has illumination invariant.So far, obtained the proper vector of Expressive Features point.

(6) characteristic matching.Employing is based on the nearest neighbor method of Euclidean distance, utilize the K-D tree to search in image to be matched, find with benchmark image in the nearest the first two unique point of unique point Euclidean distance, if minimum distance less than the proportion threshold value (0.7) of setting, is accepted this a pair of match point except the value that closely obtains in proper order.

Step 3, images match are put the three-dimensional coordinate mapping.

According to calibrating parameters, Kinect depth image and RGB image mid point mapping calculation method are as follows:

1 p=(x in depth image _d, y _d) coordinate P under the Kinect coordinate system _3D=(x, y, z) is:

$\left\{\begin{array}{c}{P}_{3D}.x=(x_d-u_{\mathrm{IR}})\×P_{3D}.z/f_{\mathrm{uIR}}\\ P_{3D}.y=(y_d-v_{\mathrm{IR}})\×P_{3D}.z/f_{\mathrm{vIR}}\\ P_{3D}.z=\mathrm{depth}(x_d,y_d)\end{array}\right.$

Wherein, depth (x _d, y _d) depth value of expression depth image mid point p.

So, derive the corresponding 3D coordinate of pixel of RGB image, and then obtain the coordinate (x in the RGB image _rgb, y _rgb).Computing formula is as follows:

$\left\{\begin{array}{c}{x}_{\mathrm{rgb}}=(P_{3D}^{\′}.x*f_u/P_{3D}^{\′}.z)+u_0\\ y_{\mathrm{rgb}}=(P_{3D}^{\′}.y*f_v/P_{3D}^{\′}.z)+v_0\end{array}\right.$

Wherein, P ' _3D ^TR _IRc* P _3D ^T+ t _IRc

According to above-mentioned conversion relation, the matching double points that obtains in step 2 is converted to three dimensions point right.

Step 4, based on the double-deck registration of the three dimensions point of RANSAC and ICP, method comprises the following steps as shown in Figure 3:

(1) first registration.The corresponding point that Feature Points Matching obtains are to existing larger mistake coupling.Using RANSAC in the first registration stage, to remove the three dimensions point of mistake coupling right, finds the imperial palace point set that satisfies transformation model and estimate transformation matrix T by iteration.Accumulation KeyFrame obtains the transformation matrix of the relative KeyFrame of current Kinect to each Relative Transformation matrix of current data.Go out translational movement and the anglec of rotation mould value of Kinect according to this matrix computations, compare with the threshold value of setting, judging whether to choose this frame is KeyFrame.In embodiment, the threshold value setting of translational movement is 0.4, and angle threshold is 40 degree.

The detailed process of RANSAC is as follows:

1) from from the initial N of reference point collection A and point set B subject to registration to choosing at random 7 pairs of data three-dimensional matching double points;

2) utilize basis matrix to find the solution 7 methods of minimal configuration, be it is calculated that the transformation matrix T of benchmark point set and point set data subject to registration by 7 logarithms of choosing ^AB

3) utilize transformation matrix T ^ABWith Characteristic of Image point set subject to registration

A remaining N-7 three-dimensional point in (expression comprises the point set B subject to registration of N point)

Transform under reference point cloud coordinate system;

4) the point set P ' after computational transformation _N-7With the benchmark point set Between error of coordinate;

5) from N to finding out the unique point of error of coordinate in certain threshold value matching double points to number, be designated as i;

6) repeat 1) ~ 5) step n(n sets by the user, in the present embodiment, iterations is set as 50) inferior, make the i value obtain maximum set for imperial palace point set, be interior point, all the other N-i are Mismatching point, are exterior point.Utilize imperial palace point set to estimate the least square solution of transformation model, as the transformation matrix T of current adjacent two frame data.

(2) accuracy registration.Obtain KeyFrame and Relative Transformation matrix thereof through first registration, the present invention adopts the accurate transformation matrix of ICP calculating K eyFrame, and first registration results as the priori conversion, is tried to achieve the accurate transformation matrix of KeyFrame-To-KeyFrame.

In the depth map of Kinect, pixel value is that 0 zone is invalid metrical information.Obtain effective information and invalid information in depth map for convenience of describing, be defined as follows function for above-mentioned phenomenon:

Wherein, X is picture planimetric coordinates point.

In order to obtain k Kinect pose constantly

According to the transformational relation of Kinect coordinate system and world coordinate system, set up following energy function:

$E=\min \underset{p^k\∈\mathrm{\Ω}}{\mathrm{\Σ}}|T_{\mathrm{cw}}^k{p}_w-p^k|$

Wherein, p _wBe the point under world coordinate system, p ^kBe the point under current coordinate system, Ω has the set of the pixel of significant depth value in the k moment plane of delineation, that is:

Ω={ p ^k| u=π (p ^k) and M (u)=1}

The above-mentioned energy function of setting up is the mathematical description of three-dimensional ICP method.In the ICP algorithm, by the minimization energy function, obtain the pose under world coordinate system at k moment Kinect.Usually the ICP algorithm is under the prerequisite of supposition relative pose, constantly sets up the corresponding relation between the some cloud, and by optimizing corresponding point error iterative.Therefore in the solution procedure of ICP algorithm, initial relative pose is set and is played vital effect, and inappropriate initial pose will make the ICP algorithm be absorbed in local optimum, can't obtain correct result.Based on the ICP algorithm of unordered some cloud, along with an increase of cloud quantity, the space complexity of algorithm and time complexity will significantly improve, and greatly reduce the execution efficient of this algorithm.Therefore initial relative pose is the prerequisite of setting up corresponding relation between the some cloud, plays vital effect in the iterative process of ICP method.The Relative Transformation that this method obtains first registration is as initial relative pose, to obtain the optimal estimation of current KeyFrame.

Suppose in the side-play amount of the k-1 moment and k moment Kinect to be

$T_{\mathrm{cw}}^k=T_{\mathrm{inc}}^k{T}_{\mathrm{cw}}^{k-1}$

Kinect is at x, y, and the rotation amount on the z direction of principal axis (α, beta, gamma) and the translational movement on three directions are (t _x, t _y, t _z).When enough hour of above-mentioned two vectors, launch according to the single order Taylor's formula, make x=(α, beta, gamma, t _x, t _y, t _z):

$T_{\mathrm{inc}}^k=\exp \left(x\right)$

$=\left[\begin{array}{ccc}1& \mathrm{\γ}& -\mathrm{\β}\\ -\mathrm{\γ}& 1& \mathrm{\α}\\ \mathrm{\β}& -\mathrm{\α}& 1\end{array}\right.\left|\begin{array}{c}{t}_x\\ t_y\\ t_z\end{array}\right]$

$=\left[R_{\mathrm{inc}}\right|t_{\mathrm{inc}}]$

World coordinates for the k moment spatial point of obtaining is

Under the coordinate system of k-1 Kinect during the moment, energy function is transformed to so with this spot projection:

$E=\min \underset{p^k\∈\mathrm{\Ω}}{\mathrm{\Σ}}\left|\right|T_{\mathrm{cw}}^k{p}_w-p^k\left|\right|$

$=\min \underset{p^k\∈\mathrm{\Ω}}{\mathrm{\Σ}}\left|\right|T_{\mathrm{inc}}^k{T}_{\mathrm{cw}}^{k-1}{p}_w-p^k\left|\right|$

$=\min \underset{p^k\∈\mathrm{\Ω}}{\mathrm{\Σ}}\left|\right|T_{\mathrm{inc}}^k{p}_w^{k-1}{p}^k\left|\right|$

Ω={ p ^k| u=π (p ^k) and M (u)=1}

Wherein, p _wAnd p ^kBe corresponding point,

Be p _wCoordinate under k-1 moment camera coordinates is.

By

$T_{\mathrm{inc}}^k{p}_w^{k-1}=R_{\mathrm{inc}}{p}_w^{k-1}+t_{\mathrm{inc}}=G\left(p_w^{k-1}\right)x+p_w^{k-1}$

Obtain the final expression of energy function:

$\underset{x\∈\mathrm{se}\left(3\right)}{\min }\underset{\mathrm{\Ω}}{\mathrm{\Σ}}\left|\right|G\left(p_w^{k-1}\right)x+p_w^{k-1}{p}^k\left|\right|$

Ω={ p ^k| u=π (p ^k) and M (u)=1}

Wherein, $\left(p_w^{k-1}\right)\left[{\left[p_w^{k-1}\right]}_{\×}\right|I_{3\×3}],[{p_w^{k-1}]}_{\×}$ Serve as reasons

The antisymmetric matrix that consists of.The threshold value of setting energy function is 0.05, utilizes Cholesky to decompose and obtains hexa-atomic group of solution x=(α, beta, gamma, t _x, t _y, t _z), be mapped to special European group (rigid motion group) SE (3) space in Lie group, and can obtain current Kinect pose in conjunction with the pose of k-1 moment Kinect.

Step 5, scene update.

The renewal of scene is divided into two kinds of situations, and a kind of is to carry out for the first time scene update, and this moment, the set positions with Kinect was the initial point of world coordinate system, and added the current contextual data of obtaining; Another kind is newly-increased frame KeyFrame data, according to formula Current newly-increased KeyFrame data are transformed in world coordinate system, complete the renewal of contextual data.

The below provides and uses the method for the invention is carried out the three-dimensional environment establishment under indoor true environment a experiment embodiment.

The depth camera that experiment is adopted is Kinect-XBOX360, and the RGB image resolution ratio is 640 * 480, and the highest frame frequency is 30fps.Indoor environment as shown in Figure 4, Fig. 4 (a) for the experiment real scene, Fig. 4 (b) is the two-dimensional geometry schematic diagram of experimental situation.During experiment, hand-held Kinect, begin from starting point, stops by behind the fixed route walking place of reaching home, and synchronously increases progressively in the process of walking the generation global map, the scope of 9m * 9m in the map covering chamber of establishment.Fig. 5 is the reconstructed results schematic diagram.

Experimental result shows, the method for the invention can be used on a large scale that indoor three-dimensional scenic creates, and has higher precision and good real-time.

The above is only preferred embodiment of the present invention, is not for limiting protection scope of the present invention, all any modifications of doing within the spirit and principles in the present invention, is equal to and replaces and improvement etc., within all should being included in protection scope of the present invention.

Claims (2)

1. indoor method for reconstructing three-dimensional scene based on double-deck method for registering is characterized in that comprising the following steps:

Step 1 is carried out Kinect and is demarcated, and method is as follows:

(1) print a chessboard template;

(2) from a plurality of angle shot chessboards;

(3) detect unique point in image, i.e. each black point of crossing of chessboard;

(4) obtain the parameter that Kinect demarcates:

The internal reference matrix K of thermal camera _ir:

$K_{\mathrm{ir}}=\left[\begin{array}{ccc}{f}_{\mathrm{uIR}}& 0& u_{\mathrm{IR}}\\ 0& f_{\mathrm{vIR}}& v_{\mathrm{IR}}\\ 0& 0& 1\end{array}\right]$

Wherein, (f _uIR, f _vIR) be the focal length of thermal camera, value (5,5), (u _IR, v _IR) be thermal camera as the planar central coordinate, value (320,240);

The Intrinsic Matrix K of RGB video camera _c:

$K_c=\left[\begin{array}{ccc}{f}_u& 0& u_0\\ 0& f_v& v_0\\ 0& 0& 1\end{array}\right]$

Wherein, (f _u, f _v) be the focal length of RGB video camera, (u ₀, v ₀) be that the RGB video camera is as the planar central coordinate;

External parameter between thermal camera and RGB video camera is:

T＝[R _IRc,t _IRc]

Wherein, R _IRcBe rotation matrix, t _IRcTranslation vector, the parameter of directly using official of Microsoft to provide:

$R_{\mathrm{IRc}}=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$

t _IRc＝[0075 0 0] ^T

The Kinect coordinate system for y axle positive dirction, is upwards forward z axle positive dirction, is to the right the x positive dirction; The initial point position of Kinect is set as the world coordinate system initial point, and the X of world coordinate system, Y, Z direction are identical with x, y, the z direction of Kinect initial point position;

Step 2, the extraction of unique point and coupling, method is as follows:

(1) obtain integral image: the integration I (X) for certain some X=(x, y) in image is:

$I_{\mathrm{\Σ}}\left(X\right)=\underset{i=0}{\overset{i\≤x}{\mathrm{\Σ}}}\underset{j=0}{\overset{j\≤y}{\mathrm{\Σ}}}I(i,j)$

Calculate the gray-scale value sum of a rectangular area with 3 plus and minus calculations in integral image, irrelevant with the area of rectangle;

(2) ask for approximate Hessian matrix H _Approx: for certain some X=(x, y) in image I, the Hessian matrix H (X, s) on the s yardstick that X is ordered is defined as:

$H(X,s)=\left[\begin{array}{cc}{L}_{\mathrm{xx}}(X,s)& L_{\mathrm{xy}}(X,s)\\ L_{\mathrm{xy}}(X,s)& L_{\mathrm{yy}}(X,s)\end{array}\right]$

Wherein, L _xx(X, s), L _xy(X, s), L _yy(X, s) expression Gauss second-order partial differential coefficient is in the convolution of X place and image I; Use the second order Gauss filtering in the approximate Hessian of replacement of square frame filtering matrix, the value of frame shape Filtering Template after with image convolution is respectively D _xx, D _yy, D _xy, further replace L with them _xx, L _yy, L _xyObtain approximate Hessian matrix H _Approx, its determinant is:

det(H _approx)＝D _xxD _yy-(wD _xy) ²

Wherein, w is weight coefficient;

(3) location feature point: the feature point detection of SURF is based on the Hessian matrix, according to the local maximum location feature point position of Hessian matrix determinant;

With the frame shape wave filter of different size, original image is processed and obtained the yardstick image pyramid, according to H _ApproxObtain the extreme value that the scalogram picture is located at (X, s);

Use frame shape wave filter to build metric space, in every single order, select the scalogram picture of 4 layers, use H _ApproxMatrix is obtained extreme value, in 3 dimension (X, s) metric spaces, each regional area of 3 * 3 * 3 is carried out non-maximum value suppress; Elect response as unique point greater than the point of 26 neighborhood values; Utilize the quadratic fit function that unique point is accurately located, fitting function D (X) is:

$D\left(X\right)=D+\frac{{\∂D}^T}{\∂X}X+\frac{1}{2}{X}^T\frac{\∂D}{{\∂X}^2}X$

Thereby obtain position, the yardstick information (X, s) of unique point;

(4) determine the direction character of unique point: with the Haar wavelet filter, circular neighborhood is processed, obtained the response of the corresponding x of each point, y direction in this neighborhood; Choose the Gaussian function centered by unique point, σ gets 2s, and s is yardstick corresponding to this unique point, these responses are weighted, and the vector of search length maximum, its direction is the corresponding direction of this unique point;

(5) construction feature description vectors: determine a foursquare neighborhood centered by unique point, the length of side is got 20s, is the unique point direction setting y direction of principal axis of this neighborhood; Square area is divided into 4 * 4 sub regions, processes with the Haar wavelet filter in each subregion, Haar small echo template size is 2s * 2s; Use d _xThe little wave response of Haar of expression horizontal direction is used d _yThe little wave response of Haar of expression vertical direction is for all d _x, d _yIn order to the Gaussian function weighting centered by unique point, the σ of this Gaussian function is 3.3s; In every sub regions respectively to d _x, d _y, d _x|, d _y| summation obtains 4 dimensional vector V (∑ d _x, ∑ d _y, ∑ d _y|, ∑ d _y|), the vector of 4 * 4 sub regions being coupled together just obtained one 64 vector of tieing up, this vector has rotation, yardstick unchangeability, after normalization, has illumination invariant; This vector is the proper vector of Expressive Features point;

(6) characteristic matching: adopt the nearest neighbor method based on Euclidean distance, utilize the K-D tree to search in image to be matched, find with benchmark image in the nearest the first two unique point of unique point Euclidean distance, if minimum distance less than the proportion threshold value of setting, is accepted this a pair of match point except the value that closely obtains in proper order;

Step 3, images match are put the three-dimensional coordinate mapping, and method is as follows:

Ask 1 p=(x in depth image _d, y _d) coordinate P under the Kinect coordinate system _3D=(x, y, z):

$\left\{\begin{array}{c}{P}_{3D}.x=(x_d-u_{\mathrm{IR}})\×P_{3D}.z/f_{\mathrm{uIR}}\\ P_{3D}.y=(y_d-v_{\mathrm{IR}})\×P_{3D}.z/f_{\mathrm{vIR}}\\ P_{3D}.z=\mathrm{depth}(x_d,y_d)\end{array}\right.$

Wherein, depth (x _d, y _d) depth value of expression depth image mid point p;

Obtain coordinate (x in the RGB image by the corresponding 3D coordinate of RGB image pixel _rgb, y _rgb):

$\left\{\begin{array}{c}{x}_{\mathrm{rgb}}=(P_{3D}^{\′}.x*f_u/P_{3D}^{\′}.z)+u_0\\ y_{\mathrm{rgb}}=(P_{3D}^{\′}.y*f_v/P_{3D}^{\′}.z)+v_0\end{array}\right.$

Wherein, P ' _3D ^T=R _IRc* P _3D ^T+ t _IRc

According to above-mentioned conversion relation, the matching double points that obtains in step 2 is converted to three dimensions point right;

Step 4, based on the double-deck registration of three dimensions point of RANSAC and ICP method, method is as follows:

(1) first registration: using RANSAC in the first registration stage, to remove the three dimensions point of mistake coupling right, finds the imperial palace point set that satisfies transformation model and estimate transformation matrix T by iteration; Accumulation KeyFrame obtains the transformation matrix of the relative KeyFrame of current Kinect to each Relative Transformation matrix of current data; Go out translational movement and the anglec of rotation mould value of Kinect according to this matrix computations, compare with the threshold value of setting, judging whether to choose this frame is KeyFrame;

(2) accuracy registration: for obtaining k Kinect pose constantly

According to the transformational relation of Kinect coordinate system and world coordinate system, set up following energy function:

$E=\min \underset{p^k\∈\mathrm{\Ω}}{\mathrm{\Σ}}|T_{\mathrm{cw}}^k{p}_w-p^k|$

Wherein, p _wBe the point under world coordinate system, p ^kBe the point under current coordinate system, Ω has the set of the pixel of significant depth value in the k moment plane of delineation, that is:

Ω={ p ^k| u=π (p ^k) and M (u)=1}

Wherein, M(X) obtain the function of effective information and invalid information in depth map for describing:

Wherein, X is picture planimetric coordinates point;

Suppose in the side-play amount of the k-1 moment and k moment Kinect to be

:

$T_{\mathrm{cw}}^k=T_{\mathrm{inc}}^k{T}_{\mathrm{cw}}^{k-1}$

As Kinect at x, y, the rotation amount on the z direction of principal axis (α, beta, gamma) and the translational movement on three directions are (t _x, t _y, t _z) enough hour, launch according to the single order Taylor's formula, make x=(α, beta, gamma, t _x, t _y, t _z):

$T_{\mathrm{inc}}^k=\exp \left(x\right)$

$=\left[\begin{array}{ccc}1& \mathrm{\γ}& -\mathrm{\β}\\ -\mathrm{\γ}& 1& \mathrm{\α}\\ \mathrm{\β}& -\mathrm{\α}& 1\end{array}\right.\left|\begin{array}{c}{t}_x\\ t_y\\ t_z\end{array}\right]$

$=\left[R_{\mathrm{inc}}\right|t_{\mathrm{inc}}]$

World coordinates for the k moment spatial point of obtaining is

Under the coordinate system of k-1 Kinect during the moment, energy function is transformed to this spot projection:

$E=\min \underset{p^k\∈\mathrm{\Ω}}{\mathrm{\Σ}}\left|\right|T_{\mathrm{cw}}^k{p}_w-p^k\left|\right|$

$\underset{p^l\∈\mathrm{\Ω}}{\mathrm{\Σ}}\left|\right|T_{\mathrm{inc}}^k{T}_{\mathrm{cw}}^{k-1}{p}_w-p^k\left|\right|$

$=\min \underset{p^k\∈\mathrm{\Ω}}{\mathrm{\Σ}}\left|\right|T_{\mathrm{inc}}^k{p}_w^{k-1}{p}^k\left|\right|$

Ω={ p ^k| u=π (p ^k) and M (u)=1}

Wherein, p _wAnd p ^kBe corresponding point,

Be p _wCoordinate under k-1 moment camera coordinates is;

By

$T_{\mathrm{inc}}^k{p}_w^{k-1}=R_{\mathrm{inc}}{p}_w^{k-1}+t_{\mathrm{inc}}=G\left(p_w^{k-}1\right)x+p_w^{k-1}$

Obtain the final expression of energy function:

$\underset{x\∈\mathrm{se}\left(3\right)}{\min }\underset{\mathrm{\Ω}}{\mathrm{\Σ}}\left|\right|G\left(p_w^{k-1}\right)x+p_w^{k-1}{p}^k\left|\right|$

Ω={ p ^k| u=π (p ^k) and M (u)=1}

Wherein, $G\left(p_w^{k-1}\right)=\left[\right[p_w^{k-1}{]}_{\×}\left|I_{3\×3}\right],[p_w^{k-1}{]}_{\×}$ Serve as reasons The antisymmetric matrix that consists of;

Set the threshold value of energy function, utilize Cholesky to decompose and obtain hexa-atomic group of solution x=(α, beta, gamma, t _x, t _y, t _z), be mapped to special European group SE (3) space in Lie group, and can obtain current Kinect pose in conjunction with the pose of k-1 moment Kinect;

Step 5, scene update, method is as follows:

The renewal of scene is divided into two kinds of situations, and a kind of is to carry out for the first time scene update, and this moment, the set positions with Kinect was the initial point of world coordinate system, and added the current contextual data of obtaining; Another kind is newly-increased frame KeyFrame data, according to formula

Current newly-increased KeyFrame data are transformed in world coordinate system, complete the renewal of contextual data.

2. the indoor method for reconstructing three-dimensional scene based on double-deck method for registering according to claim 1, is characterized in that, the method that matrix T is changed in the described application of step 4 RANSAC changes persuing is as follows:

(1) from from the initial N of reference point collection A and point set B subject to registration to choosing at random 7 pairs of data three-dimensional matching double points;

(2) utilize basis matrix to find the solution 7 methods of minimal configuration, be it is calculated that the transformation matrix T of benchmark point set and point set data subject to registration by 7 logarithms of choosing ^AB

(3) utilize transformation matrix T ^ABWith Characteristic of Image point set subject to registration

In a remaining N-7 three-dimensional point

Transform under reference point cloud coordinate system;

(4) the point set P ' after computational transformation _N-7With the benchmark point set

Between error of coordinate;

(5) from N to finding out the unique point of error of coordinate in certain threshold value matching double points to number, be designated as i;

(6) repeat (1) ~ (5) n time, make set that the i value obtains maximum for imperial palace point set, be interior point, all the other N-i are Mismatching point, are exterior point; Utilize imperial palace point set to estimate the least square solution of transformation model, as the transformation matrix T of current adjacent two frame data.

Images