CN104346829A - Three-dimensional color reconstruction system and method based on PMD (photonic mixer device) cameras and photographing head - Google Patents

Abstract

The invention discloses a three-dimensional color reconstruction system and method based on a PMD camera and a photographing head. The system comprises an independent calibration module, an integrated calibration module and a three-dimensional reconstruction module, wherein the independent calibration module is used for calibrating the PMD camera and the photographing head to obtain inner parameters, outer parameters and distortion parameters of the PMD camera and the photographing head; the integrated calibration module is used for solving corrected images corresponding to images collected by the PMD camera and the photographing head respectively, and a relative rotation torque and a translation vector between a coordinate system of the PMD camera and a coordinate system of the photographing head according to the inner parameters, the outer parameters and the distortion parameters; the three-dimensional reconstruction module is used for selecting pixel points at four edges in the corrected images, solving corresponding points on the photographing head according to the relative rotation torque and the translation vector to obtain the corresponding image edges of the images collected by the photographing head, cutting the images and storing the cut images as new images. According to the three-dimensional color reconstruction system and method, the three-dimensional color reconstruction is implemented on the basis of the PMD camera and the photographing head.

Description

Based on color three dimension reconstructing system and the method thereof of PMD camera and camera

Technical field

The invention belongs to Automatic Measurement Technique field, relate to a kind of three-dimensional reconstruction system and the method thereof that merge the positional information of PMD camera (Photonic Mixer Device, PMD) and the color information of colour imagery shot particularly.

Background technology

Three-dimensional reconstruction, refer to and the mathematical model being applicable to computer representation and process is set up to object in three dimensions, being the basis processing, operate and analyze its character under computer environment to it, is also the gordian technique setting up the virtual reality expressing objective world in a computer.In machine vision, three-dimensional reconstruction refers to the process of the image reconstruction three-dimensional information according to single-view or multi views, due to the INFORMATION OF INCOMPLETE of single video, therefore three-dimensional reconstruction needs to utilize experimental knowledge, and the three-dimensional reconstruction of multi views (the binocular location of similar people) is relatively easy, its method is first demarcated video camera, namely calculates the image coordinate system of video camera and the relation of world coordinate system, then utilizes the information reconstruction in multiple two dimensional image to go out three-dimensional information.

At present, traditional robotic vision system is all the work such as the identification location of carrying out target based on monocular vision or binocular vision, and this vision system exists following problem to the identification of target and location: the distance of binocular vision system in space orientation between General Requirements sensor and target is greater than 50cm.And the array configuration General Requirements position transducer of monocular vision and position transducer and the distance of target are less than 100cm; This all greatly limit three-dimensional reconstruction system and the application of method in real work.

Because, for the parallel shafts stereo visual system of standard, namely two camera coordinate systems are parallel to each other and mutually only at their X or the Binocular Stereo Vision System of Y direction translation one segment distance, depth recovery wherein and three-dimensional rebuilding method are (Ramesh Jain known in the art, Rangachar Kasturi, Brian G Schunck.Machine Vision, McGraw-Hill, 1995).He is the image utilizing two cameras simultaneously to gather, two images have certain pixel deviations in X-axis or Y direction, when carrying out depth recovery and three-dimensional reconstruction by the three-dimensional rebuilding method of standard, this just requires will there be common target in two images simultaneously collected, namely require that target has certain horizontal range to the plane of camera system, horizontal range between this Distance geometry two cameras has certain relation, and General Requirements target is greater than 50cm to the horizontal range of camera plane.And for the three-dimensional rebuilding method of monocular vision combination, because the sensors such as use location carry out the degree of depth of perception target, so be also have requirement to target to the horizontal range of camera plane, this distance of General Requirements is less than 100cm.So, the stereo visual system adopted in stereoscopic vision at present and method, the distance of target to sensor be there are certain requirements, and images match will be carried out according to the complexity of target and be also one and easily produce error and time-consuming process, and affect reconstruction precision.Therefore, this area needs a kind of novel depth recovery and three-dimensional reconstruction system and method.

Summary of the invention

The object of the present invention is to provide a kind of color three dimension reconstructing system based on PMD camera and camera and method, rebuild for realizing color three dimension based on PMD camera and camera.

To achieve these goals, the invention provides a kind of color three dimension reconstructing system based on PMD camera and camera, it is characterized in that, comprising:

Independent demarcating module, for demarcating PMD camera, camera, obtains described PMD camera, the inner parameter of described camera, external parameter and distortion parameter;

Combination demarcating module, for the image gathered respectively described PMD camera, described camera, solve relative rotation matrices between image after corresponding correction, described PMD camera, described camera coordinate system and translation vector according to described inner parameter, described external parameter and described distortion parameter;

Three-dimensional reconstruction module, for choosing the pixel at four edges in image after described correction, and solve the corresponding point on described camera according to described relative rotation matrices and translation vector, obtain image border corresponding in the image of described camera collection, and after being partitioned into, be stored as new image.

Described color three dimension reconstructing system, wherein, described independent demarcating module is also for being optimized described inner parameter, external parameter.

Described color three dimension reconstructing system, wherein, described combination demarcating module also for: after described camera, PMD camera are fixing up and down, gather at least three images from different perspectives simultaneously, detect the unique point of often opening in photo, utilizing the orthogonality of rotation matrix and described inside and outside parameter and distortion parameter according to described unique point, by solving linear equation, asking for described relative rotation matrices and translation vector.

Described color three dimension reconstructing system, wherein, described three-dimensional reconstruction module obtains the corresponding point of image on described camera after described correction in the following ways:

P _web(X _c，Y _c，Z _c)＝RP _pmd(X ₁，Y ₁，Z ₁)+T

Wherein, (X ₁, Y ₁, Z ₁) be the edge pixel point of PMD collected by camera to the rear image of correction, (X _c, Y _c, Z _c) be the corresponding point on described camera, R, T are relative rotation matrices between two camera coordinate systems and translation matrix, P _pmdthe volume coordinate representing under PMD camera coordinates system a bit, P _wedbe the volume coordinate representing under camera coordinate system a bit, subscript pmd represents that, under PMD coordinate system, subscript web represents under camera coordinate system.

Described color three dimension reconstructing system, wherein, described three-dimensional reconstruction module also for: utilize described inner parameter, external parameter obtain image after distortion correction; Solve the common field range of described PMD camera, described camera; Solve the RGB chromatic information that XYZ space positional information is corresponding, utilize 4 weighted sums to solve; Utilize bilinear interpolation method to obtain high resolving power XYZ-RGB image information according to described common field range, complete three-dimensional reconstruction.

To achieve these goals, the present invention also provides a kind of color three dimension method for reconstructing based on PMD camera and camera, it is characterized in that, comprising:

Step one, demarcates PMD camera, camera, obtains described PMD camera, the inner parameter of described camera, external parameter;

Step 2, to the image that described PMD camera, described camera gather respectively, relative rotation matrices between image after corresponding correction, camera coordinates system and translation vector is solved by the inner parameter of described PMD camera, described video camera and distortion parameter, and trimming edge quadrate;

Step 3, choose the pixel at four edges in image after described correction, and solve the corresponding point on described camera according to described relative rotation matrices and translation vector, obtain image border corresponding in the image of described camera collection, and after being partitioned into, be stored as new image.

Described color three dimension method for reconstructing, wherein, in described step one, comprising:

B1. scaling board is obtained;

B2. move scaling board or camera, take at least 8 photos from different perspectives;

B3. all unique points of often opening in photo are detected;

B4. utilizing the orthogonality of rotation matrix according to described unique point, by solving linear equation, obtaining described inner parameter and external parameter.

Described color three dimension method for reconstructing, wherein, in described step 2, comprising:

C1. scaling board is obtained;

C2. move scaling board or camera, take at least 3 photos from different perspectives simultaneously;

C3. all unique points of often opening in photo are detected;

C4. utilize described inside and outside parameter to carry out distortion correction to image, utilizing the orthogonality of rotation matrix according to described unique point, by solving linear equation, obtaining described rotation matrix and translation matrix;

C5. the described rotation matrix after utilizing least square method optimization to optimize and translation matrix.

Described color three dimension method for reconstructing, wherein, in described step 3, comprising:

Obtain the corresponding point of image on described camera after described correction in the following ways:

P _web(X _c，Y _c，Z _c)＝RP _pmd(X ₁，Y ₁，Z ₁)+T

Wherein, (X ₁, Y ₁, Z ₁) be the edge pixel point of PMD collected by camera to the rear image of correction, (X _c, Y _c, Z _c) be the corresponding point on described camera, R, T are relative rotation matrices between two camera coordinate systems and translation matrix, P _pmdthe volume coordinate representing under PMD camera coordinates system a bit, P _wedbe the volume coordinate representing under camera coordinate system a bit, subscript pmd represents that, under PMD coordinate system, subscript web represents under camera coordinate system.

Described color three dimension method for reconstructing, wherein, in described step 3, comprising:

D1. utilize described inner parameter, external parameter obtains image after distortion correction;

D2. the common field range of described PMD camera, described camera is solved;

D3. solve the RGB chromatic information that XYZ space positional information is corresponding, utilize 4 weighted sums to solve;

D4. utilize bilinear interpolation method to obtain high resolving power XYZ-RGB image information according to described common field range, complete three-dimensional reconstruction.

The invention provides a kind of novel color three dimension reconstructing system based on PMD camera and camera and method, greatly adapt to the application of field of agricultural robots, reach wider to the measurement range of target, measuring accuracy is higher, stability is better, and simple to operate, quick, and do not need to carry out real-time matching to image.

Method of the present invention, compared with classic method, has the following advantages:

1. accuracy is good, and repeatability is high: carry out three-dimensional reconstruction to scene internal object in different distance, its positioning precision and hand dipping related coefficient are 0.999, and standard deviation is 1.0cm;

2. speed of rebuilding is fast: three-dimensional rebuilding method mainly relies on non-online correction parameter, so just can reconstruct scene three-dimensional information in visual field in 2s;

3. reliability is high, can use in the environment of outdoor orchard: the method three-dimensional reconstruction mainly utilizes specific wavelength laser time of flight (TOF) principle to set up spatial information, in outdoor orchard, therefore utilize the method still can obtain reliable and stable spatial positional information;

4. simple to operate: three-dimensional rebuilding method has write in compiled software, only needing to carry out simple training to operating personnel can operate;

5. expansibility is strong: when the larger single vision system of field range can not cover, set of system conbined usage can be increased, only need to utilize off-line type two camera to rotate and translation matrix correction method, just can obtain rotation between two systems and translation Input matrix just can increase field range as parameter, can jointly use by 5 these subsystems at most simultaneously.

Method energy of the present invention well three-dimensional reconstruction visual scene, thus ensure that the normal operation of robotic vision system to a great extent, simultaneously also for vision measurement technology provides a kind of reference measure method and apparatus.

Accompanying drawing explanation

Figure 1A is the color three dimension reconstructing system structural drawing based on PMD camera and camera of the present invention;

Figure 1B is the color three dimension reconstructing system structural drawing based on PMD camera and camera of the present invention;

Fig. 2 is the color three dimension method for reconstructing process flow diagram based on PMD camera and camera of the present invention;

Fig. 3 is that schematic diagram and schematic diagram are demarcated in combination of the present invention;

Fig. 4 is interpolation by weighted average method schematic diagram of the present invention;

Fig. 5 is bilinear interpolation method principle schematic;

Fig. 6 A-Fig. 6 D is that the color three dimension based on PMD camera and camera of the present invention rebuilds structural representation;

Fig. 7 be the measurement result of three-dimensional reconstruction of the present invention and hand dipping adopt to same target compare schematic diagram.

Embodiment

Describe the present invention below in conjunction with the drawings and specific embodiments, but not as a limitation of the invention.

As shown in Figure 1A, be the color three dimension reconstructing system structural drawing based on PMD camera and camera of the present invention, Figure 1B is the outside schematic diagram of the color three dimension reconstructing system based on PMD camera and camera of the present invention.

This reconstructing system 100 comprises: module 40 is set up in single camera demarcation 10, combination demarcating module 20, three-dimensional reconstruction module 30 and system.

Independent demarcating module 10, for demarcating single camera (common camera 1, PMD camera 3), solves the inside and outside parameter that common camera 1, PMD camera 3 are respective.

Combination demarcating module 20, combination for carrying out system is demarcated, for common camera 1, PMD camera 3 take the photograph two width images in visual field, image after the correction of its correspondence is solved by the intrinsic parameter of PMD camera 3 and distortion parameter, relative rotation matrices between camera coordinates system and translation vector, and trimming edge quadrate.

Three-dimensional reconstruction module 30, in image after calibration, chooses the pixel at four edges and the corresponding point solved in common camera 1, finds common camera 1 to gather image border corresponding in image, and is stored as new image after being partitioned into.

Because the resolution of common camera 1, PMD camera 3 is different, pixel in the pixel of PMD camera 3 and common camera 1 is not one_to_one corresponding, so correspondence point in common camera 1 needs to solve corresponding colouring information (RGB information) through interpolation method, solve being employed herein 4 weighted sum methods.

PMD camera 3 obtains object distance by flight (coming and going) time of detecting optical pulses, and obtains the depth information of entire image simultaneously.

System sets up module 40, for setting up color three dimension system and verifying, according to the XYZRGB information of solved PMD camera 3 correspondence, in conjunction with RGB information required by common camera 1, obtain corresponding RGB and XYZ interpolating matrix, wherein interpolation method employs bilinear interpolation, finally can try to achieve the RGB-XYZ information jointly collected with PMD camera 3 in common camera 1, realizes color three dimension and rebuilds.

The basic functional principle of reconstructing system 100 is:

PMD camera 3 and common camera about 1 are placed, fixed by web member between camera 4, PMD camera 3 and common camera 1 trigger dynamic acquisition image simultaneously, image is gathered by data acquisition and analytic system 40 and is preserved, be supplied to central processing unit, mutual relationship in the three-dimensional space position that can be calculated PMD camera 3 and common camera 1 of method for reconstructing between color information, thus set up color three dimension spatial information, for application in agricultural such as the navigation of robot, obstacle, identification provide corresponding reference information to positioning system etc.

Common camera 1, it is as camera, can use any color camera or IP Camera, as acquisition field of view chromatic information, for three-dimensional reconstruction.

Controlled LASER Light Source 2, selects the LASER Light Source of different wave length to be used as depth information in visual field and measures, selected 870nm light source here, carries out depth information measurement accuracy high, reproducible, and be applicable to agriculture outdoor environment based on laser time of flight principle.

PMD camera 3, can the depth information of all pixels and gray scale intensity signal in acquisition field of view, and image resolution ratio 200*200, gray level image is used for self-correcting or combination is demarcated, and depth information is used for three-dimensional reconstruction, scope 0.3-7m on probation.

Web member 4 between camera, for common camera 1, PMD camera 3 being fixed, ensureing that demarcating its relative position rear can not change, thus ensureing the precision of three-dimensional reconstruction.

Device outer connector 5, ensures its safety and stable easy link for whole system being fixed on other object.

PMD camera power supply interface 6 is the interfaces for providing power supply to PMD camera 3 and light source.

Light source input control interface 7 is the interfaces for receiving light source control signal.

Light source exports control interface 8, according to Software for Design and system requirements, exports the interface to light source control signal, mainly controls the intensity of light source, ensures the accuracy of the information that collects.

PMD camera data interface 9, controls for computing machine and miscellaneous equipment the communication interface gathering spatial position data, employing be Mini b USB2.0 interface.

General camera data-interface 10 is the communication interfaces controlling to gather color data for computing machine and miscellaneous equipment, employing be generic USB 2.0 interface.

In said system:

Common camera 1, for gathering chromatic information, has selected sieve skill C920 camera here, ultimate resolution 1280*960,30fps;

Controlled LASER Light Source 2, mainly uses the laser of infrared portion as light source, can improve system depth information stability and repeatability;

PMD camera 3, main uses the CMOS chip PMD19K-3S that can gather infrared light supply, is increase target three-dimensional information, and 5 different PMD cameras can be used jointly to form three-dimensional reconstruction system simultaneously.

As shown in Figure 2, be the color three dimension method for reconstructing process flow diagram based on PMD camera and camera of the present invention.In conjunction with Figure 1A, Figure 1B, the flow process of this method for reconstructing comprises the following steps:

Step 201, the demarcation of single camera, solves the respective inside and outside parameter of common camera 1, PMD camera 3 and distortion parameter;

Step 202, the combination of system is demarcated, for common camera 1, PMD camera 3 take the photograph two width images in visual field, image after the correction of its correspondence is solved by the inside and outside parameter of PMD camera 3 and distortion parameter, relative rotation matrices between camera coordinates system and translation vector, and trimming edge quadrate;

Step 203, the three-dimensional reconstruction of system, in image after calibration, chooses the pixel at four edges and the corresponding point solved in common camera 1, finds common camera 1 to gather image border corresponding in image, and is stored as new image after being partitioned into;

Because the resolution of common camera 1, PMD camera 3 is different, pixel in the pixel of PMD camera 3 and common camera 1 is not one_to_one corresponding, so correspondence point in common camera 1 needs to solve corresponding colouring information (RGB information) through interpolation method, solve being employed herein 4 weighted sum methods;

Step 204, set up color three dimension system and verify, according to the XYZRGB information of solved PMD camera 3 correspondence, in conjunction with RGB information required by common camera 1, obtain corresponding RGB and XYZ interpolating matrix, wherein interpolation method employs bilinear interpolation, finally can try to achieve the RGB-XYZ information jointly collected with PMD camera 3 in common camera 1, realizes color three dimension and rebuilds.

In one embodiment, in step 201, the single camera scaling method in common camera 1, PMD camera 3 is comprised the following steps (with reference to " study OpenCV(Chinese edition) " book, Chapter 11, camera model and demarcation, the 406th page):

A1. print a chessboard pattern or circle dot matrix image, and be attached in one piece of plane as scaling board;

A2. move scaling board or camera, take several photos (being no less than 8) from different perspectives;

A3. detect and often open all angle points in photo or round dot central point;

A4. when not considering radial distortion, utilizing the orthogonality of rotation matrix, by solving linear equation, obtaining five inner parameters and the external parameter of camera.

A5. least square method is utilized to estimate the coefficient of radial distortion of camera;

A6. inside and outside parameter is optimized.

According to the mathematical model of camera, because chessboard is mobile in every width image, namely in each visual field, 5 inner parameters and external parameter be calculated.And the angle point detected on each image, linear equation is set up by the distance in mathematic modeling and known correction plate between angle point (or round dot central point), owing to being collected by camera plurality of pictures, so equation is redundancy, utilize least square method optimization can obtain inside and outside parameter net result.

In above-mentioned steps A4, five inner parameters of camera comprise: under pixel meaning, X, Y-direction focal distance f c (1), fc (2), graphic based point coordinate value cc (1), cc (2), distortion parameter kc; The outer parameter of camera mainly comprises rotation matrix and translation matrix.

Focal distance f c (1), fc (2), be actually the physics focal length of lens and the product of imager each unit correspondence direction size, thus convert the focal length under metric unit to focal length under pixel meaning, conveniently directly calculate in calibration process.

Graphic based point coordinate value cc (1), cc (2) are the vertical projection of projection centre on imaging plane, are also the centers of radial distortion.

Distortion parameter kc, distortion mainly contains two classes: radial distortion and tangential distortion, wherein radial distortion is about camera lens principal axis symmetry, being the main source of distortion, owing to needing to use nonlinear optimization algorithm to the demarcation of camera when considering nonlinear distortion, introducing too much distortion factor, not only often can not improve precision, can cause the instability of system on the contrary, so under normal circumstances, tangential distortion can be ignored.

Rotation matrix and translation matrix, to the image of the certain objects that each width obtains, can fasten in camera coordinates the relative position describing object with the peaceful in-migration of rotation.

In one embodiment, in step 202, system in combination is demarcated and is solved common camera 1, the rotation matrix of PMD camera 3 space coordinates and translation matrix, need common camera 1, PMD camera 3 gathers scaling board (chessboard pattern or circular dot pattern) simultaneously, then utilize known features point on scaling board to carry out the outer parameter of computing system, key step is as follows:

B1. print a chessboard pattern or circle dot matrix image, and be attached in one piece of plane as scaling board;

B2. move scaling board or camera, several photos (being no less than 3) taken by two video cameras simultaneously from different perspectives;

B3. all unique points (angle point or round dot central point) of often opening in photo are detected;

B4. utilizing known camera inside and outside parameter to carry out distortion correction to image, utilize the orthogonality of rotation matrix according to the unique point detected, by solving linear equation, obtaining rotation matrix and the translation matrix of two cameras;

B5. the rotation matrix after utilizing least square method optimization to optimize and translation matrix.

As above the realization of step please refer to " study OpenCV(Chinese edition) " book, the 12nd chapter, projection and 3D vision, the 452nd page.In space, the mathematical relation of Two coordinate system is obtained by rotation matrix and translation matrix, with O _lwith O _rcoordinate system represents the space coordinates of two cameras respectively, using wherein 1 P as the angle point (or round dot central point) in image, linear equation is set up by the distance in the inside and outside parameter of the relation between two coordinate systems, camera itself and known correction plate between angle point (or round dot central point), solve the relation between Two coordinate system, i.e. rotation matrix R and translation matrix T, owing to being double camera collection plurality of pictures, so the result of finally trying to achieve is through least square method optimization, the result that can be optimized.

In one embodiment, in step 202, carrying out trimming edge quadrate is common method after distortion correction, is got rid of at insignificant edge in the image after correction, leaves rectangular view data and facilitate later stage computing.Carrying out in marginalisation, only needing to calculate in trimming process the edge of filling meaningless pixel can be got rid of.

In one embodiment, in step 203, the three-dimensional reconstruction of colored (RGB) and locus (XYZ), the inside and outside parameter of two video cameras up and down needing utilization to try to achieve, rotation and translation matrix (alpha_c, fc, cc, kc, R _sand T _s) jointly solve, key step is as follows:

C1. camera parameter is utilized to obtain image after distortion correction;

C2. the common field range of two video cameras is solved;

C3. solve the RGB chromatic information that XYZ space positional information is corresponding, utilize 4 weighted sums to solve;

C4. utilize bilinear interpolation method to obtain high resolving power XYZ-RGB image information according to common field range, three-dimensional reconstruction completes.

Because the physical parameters such as two camera lens, resolution and focal length are not identical, so its field range is different.According to three-dimensional rebuilding method, namely common field range can calculate PMD collected by camera and correspond to the point in image the point that color camera gathers the chromatic information in image, does not then belong to common field range as do not solved color image information corresponding to depth point.The image of common field range is retained, utilizes bilinear interpolation method to obtain high-resolution image information.

Further, in said method, locating correlativity with hand dipping is 0.999, and model criteria deviation is 1.0cm;

Further, in said method, in robot navigation, target identification and when using in locating, after camera chain and world coordinate system correlation parameter are brought into, video camera three-dimensional scenic can be set up in real time, and utilize other software to calculate the relevant informations such as required guidance path or target location.

Further, in said method, can fast, accurately and efficiently three-dimensional reconstruction is carried out to target in the visual field.

Below in conjunction with a specific embodiment, the process that the color three dimension based on PMD camera and camera of the present invention is rebuild is described in detail:

(1) demarcation of single camera

Print a chessboard pattern (or dot pattern), and be attached in one piece of plane as scaling board; Mobile scaling board or camera, gather several images (being no less than eight) from different perspectives; PMD collected by camera to plot of light intensity picture or the coloured image that collects of general camera in utilize system software to detect often to open all angle points (or central point of dot matrix image) in photo; When not considering radial distortion, utilizing the orthogonality of rotation matrix, by solving linear equation, trying to achieve five inner parameters of two cameras and the external parameter of system respectively; Least square method is utilized to estimate the coefficient of radial distortion of camera; Finally, inside and outside parameter is optimized; Finally, solve the inside and outside parameter after two cameras optimizations and distortion parameter, corresponding each camera comprises alpha_c, fc, cc, kc, R _sand T _s;

According to the mathematical model of camera, because chessboard is mobile in every width image, namely in each visual field, 5 inner parameters and external parameter be calculated.And the angle point detected on each image, linear equation is set up by the distance in mathematic modeling and known correction plate between angle point (or round dot central point), owing to being collected by camera plurality of pictures, so equation is redundancy, utilize least square method optimization can obtain inside and outside parameter net result.

The combination of (2) two cameras is demarcated

After common camera 1, PMD camera about 3 two camera are fixing, gather several images (being no less than three) from different perspectives simultaneously, utilize system software to detect often to open all angle points (or central point of dot matrix image) in photo, according to the orthogonality of rotation matrix and two cameras inside and outside parameter and the distortion parameter separately of having tried to achieve, by solving linear equation, ask for the relative rotation matrices between two camera coordinate systems and translation vector R, T, as shown in Figure 3;

In figure 3, depict combination and demarcate schematic diagram and schematic diagram, wherein figure (a) is the calculating schematic diagram for rotation and translation matrix between camera, and figure (b) is the fusion schematic diagram of xyz information and rgb information.

Combination is demarcated and is mainly comprised two contents:

(1) as shown in Fig. 3 (a), after common camera 1, PMD camera 3 liang of cameras are combined up and down and fix, acquire totally 11 different attitude chessboard correction plate images, solve the rotation and translation matrix that two camera coordinate systems are corresponding;

(2) as shown in Fig. 3 (b), the rgb Information Fusion schematic diagram in the xyz information of PMD camera 3 and color camera, wherein coordinate system X ₁y ₁z ₁o ₁the coordinate system of PMD camera 3, coordinate system X ₂y ₂z ₂o ₂the coordinate system of color camera, P _pmd(X ₁, Y ₁, Z ₁) be at X ₁y ₁z ₁o ₁under coordinate system a bit, target solves P _pmdpoint is corresponding to X ₂y ₂z ₂o ₂the rgb colouring information of the coordinate points under coordinate system and correspondence thereof.

(3) three-dimensional rebuilding method of system

Under the calibrated system of combination, gather image, parse image after the correction of its correspondence respectively by the inside and outside parameter and distortion parameter of trying to achieve two cameras; Choose the pixel (X at four edges of PMD plot of light intensity picture after correcting ₁, Y ₁, Z ₁), and according to the relative rotation matrices between two camera coordinate systems and translation matrix R, T, solve the corresponding point (X in common camera 1 _c, Y _c, Z _c), be computing method as shown in Equation 1;

P _web(X _c, Y _c, Z _c)=RP _pmd(X ₁, Y ₁, Z ₁)+T (formula 1)

Wherein, (X ₁, Y ₁, Z ₁) be edge pixel point after PMD camera 3 collects light intensity image rectification, (X _c, Y _c, Z _c) be the corresponding point in common camera 1, R, T are relative rotation matrices between two camera coordinate systems and translation matrix, P _pmdthe volume coordinate representing under PMD camera coordinates system a bit, P _wedit is the volume coordinate representing under camera coordinate system a bit.Subscript pmd represents that, under PMD coordinate system, subscript web represents under camera coordinate system.

Next, P is solved _wedunder relation between the image pixel value that arrives of the volume coordinate of trying to achieve and camera collection, image color value of information rgb spatial point xyz under pmd coordinate system is combined, XYZ-RGB image information can be tried to achieve.

According to the inside and outside parameter of common camera 1, alpha_c, fc, cc, kc and the volume coordinate (X tried to achieve _c, Y _c, Z _c) calculate pixel (x in its correspondence image _p, y _p), by a bit (X in color camera space _c, Y _c, Z _c) convert a bit (x in plane picture to _p, y _p), shown in following formula 2;

$x_n=\left[\begin{array}{c}{X}_c/Z_c\\ Y_c/Z_c\end{array}\right]=\left[\begin{array}{c}x\\ y\end{array}\right]$

r ²＝x ²+y ²

$x_d=\left[\begin{array}{c}{x}_d\left(1\right)\\ x_d\left(2\right)\end{array}\right]=(1+\mathrm{kc}\left(1\right)r^2+\mathrm{kc}\left(2\right)r^4+\mathrm{kc}\left(5\right)r^6)x_n+\mathrm{dx}$ (formula 2)

$\mathrm{dx}=\left[\begin{array}{c}2\mathrm{kc}\left(3\right)\mathrm{xy}+\mathrm{kc}\left(4\right)(r^2+{2x}^2)\\ 2\mathrm{kc}\left(4\right)\mathrm{xy}+\mathrm{kc}\left(3\right)(r^2+{2y}^2)\end{array}\right]$

$\left\{\begin{array}{c}{x}_p=\mathrm{fc}\left(1\right)x_d\left(1\right)+\mathrm{cc}\left(1\right)\\ y_p=\mathrm{fc}\left(2\right)x_d\left(2\right)+\mathrm{cc}\left(2\right)\end{array}\right.$

X _nrepresent a bit (X in space _c, Y _c, Z _c) project to point corresponding in plane, x _drepresent that camera distortion corrects the Taylor series expansion of mathematical formulae, dx represents the remainder of taylor series expansion.

Corresponding point (X in common camera 1 _c, Y _c, Z _c), according to the inside and outside parameter of common camera 1, fc, cc, kc and the volume coordinate (X tried to achieve _c, Y _c, Z _c) calculate pixel (x in its correspondence image _p, y _p)

Because two resolution of video camera are different, in the pixel of PMD camera 3 and common camera 1, pixel is not one_to_one corresponding, institute needs to solve through interpolation method in the hope of correspondence pixel in common camera 1, just can obtain the colouring information (RGB information) of its correspondence, solve being employed herein 4 weighted sum methods, as shown in Figure 4.Known (x _p, y _p), (x ₀, y ₀), (x ₁, y ₁), (x can be tried to achieve respectively _p, y _p) to the distance of nearest four pixels: D ₀₀, D ₀₁, D ₁₀, D ₁₁, according to weighted sum formula, can (x be tried to achieve _p, y _p) put corresponding RGB information r _p, g _p, b _p, formula 3 is as follows;

$r_p=\frac{D_{00}}{D_{00}+D_{01}+D_{10}+D_{11}}{r}_{00}+\frac{D_{01}}{D_{00}+D_{01}+D_{10}+D_{11}}{r}_{\text{01}}+\frac{D_{10}}{D_{00}+D_{01}+D_{10}+D_{11}}{r}_{10}+\frac{D_{11}}{D_{00}+D_{01}+D_{10}+D_{11}}{r}_{11}$

$g_p=\frac{D_{00}}{D_{00}+D_{01}+D_{10}+D_{11}}{g}_{00}+\frac{D_{01}}{D_{00}+D_{01}+D_{10}+D_{11}}{g}_{\text{01}}+\frac{D_{10}}{D_{00}+D_{01}+D_{10}+D_{11}}{g}_{10}+\frac{D_{11}}{D_{00}+D_{01}+D_{10}+D_{11}}{g}_{11}$

$b_p=\frac{D_{00}}{D_{00}+D_{01}+D_{10}+D_{11}}{b}_{00}+\frac{D_{01}}{D_{00}+D_{01}+D_{10}+D_{11}}{b}_{\text{01}}+\frac{D_{10}}{D_{00}+D_{01}+D_{10}+D_{11}}{b}_{10}+\frac{D_{11}}{D_{00}+D_{01}+D_{10}+D_{11}}{b}_{11}$

Wherein, known (x _p, y _p), (x ₀, y ₀), (x ₁, y ₁), (x can be tried to achieve respectively _p, y _p) to the distance of nearest four pixels: D ₀₀, D ₀₁, D ₁₀, D ₁₁, according to weighted sum formula, can (x be tried to achieve _p, y _p) put corresponding RGB information r _p, g _p, b _p,

(r ₀₀, g ₀₀, b ₀₀) be (x ₀, y ₀) some rgb colouring information;

(r ₀₁, g ₀₁, b ₀₁) be (x ₀, y ₁) some rgb colouring information;

(r ₁₀, g ₁₀, b ₁₀) be (x ₁, y ₀) some rgb colouring information;

(r ₁₁, g ₁₁, b ₁₁) be (x ₁, y ₁) some rgb colouring information.

Finally, a bit (X in PMD camera ₁, Y ₁, Z ₁) corresponding RGB information is (r _p, g _p, b _p).

(4) color three dimension reconstructing system and checking is set up

According to the XYZRGB information of solved PMD camera 3 correspondence, the XYZRGB information being equal to pixel with PMD camera 3 can be obtained, for improving image resolution ratio, utilize bilinear interpolation method, shown in Figure 5, interpolation is carried out to RGBXYZ matrix, finally can try to achieve the RGB-XYZ information jointly collected with PMD camera 3 in common camera 1, realize color three dimension to rebuild, shown in following Fig. 6 A-Fig. 6 D.

In the bilinear interpolation interpolator arithmetic of image, new-create pixel numerical value value (x0 in target image, y0), be drawn by weighted average calculation in the value of 4 adjacent pixels in 2X2 region that it closes on by source images, be illustrated in figure 5 its schematic diagram.Arthmetic statement is as follows:

Wherein, it is floating type that float represents these data, and it is integer type that int represents these data.

(1) source images and wide with the high ratio of target image is calculated.

W0: the width representing source images

H0: the height representing source images

W1: the width representing target image

H1: the height representing target image

fw=(w0-1)/(w1-1)；

fh=(h0-1)/(h1-1)；

(2) for a point (x0, y0) of target image, calculate the respective coordinates in source images, result is floating number.

float x0=x*fw；

float y0=y*fh；

int x1=int(x0)；

int x2=x1+1；

int y1=int(y0)；

int y2=y1+1；

(3) ask weight ratio s1, s2, s3, s4 of closing on shared by four points, close on four points and be respectively (x1, y1), (x2, y2), (x1, y2), (x2, y1).

fx1=x0-x1；

fx2=1.0f-fx1；

fy1=y0-y1；

fy2=1.0f-fy1；

float s1=fx1*fy1；

float s2=fx2*fy1；

float s3=fx2*fy2；

float s4=fx1*fy2；

Finally, the coordinate figure obtaining this point is represented with value (coordinate), then: value (x0, y0)=value (x2, y2) * s1+value (x1, y2) * s2+value (x1, y1) * s3+value (x2, y1) * s4.

As shown in Figure 6A, be original color image that system color collected by camera arrives; Fig. 6 B is PMD camera and the color camera plot of light intensity picture that arrives of triggering collection simultaneously; Fig. 6 C is the depth image that PMD camera collects simultaneously; Fig. 6 D is the rgb image produced after the colouring information of system globe area Fig. 6 A, 6B, 6C and depth information with xyz information.Wherein, original color image has wider field range in the lateral direction, get rid of after information fusion, the pixel of depth image and plot of light intensity picture is one to one, and due to system be fix up and down, so its depth information has wider field range in below, owing to there is no corresponding informance in coloured image, so it is shown as black in reconstruction image.

The checking of three-dimensional reconstruction system measuring accuracy.System carries out three-dimensional reconstruction three different distance to same target, hand dipping is carried out to the target in scene, and compare with the measurement result of system three-dimensional reconstruction, hand dipping and systematic survey correlativity are 0.999, standard deviation is 1.0cm, mean deviation is 0.9cm, and result is as shown in following table 1 and Fig. 7.

Table 1

The present invention is owing to installing simple, easy operation, fast response time, the target detection unit of field, industrialized agriculture and orchard robot can be widely used in, also can carry out real-time obstacle to ensure the safety of mobile device and people in some production works to moving vehicle or robot.The present invention not only can carry out three-dimensional reconstruction and detection to target, also can be used in the navigation of moving target and obstacle and monitor in real time its state, to ensure its safety and stability run.

The present invention proposes a kind of dynamic three-dimensional reconstruction system and method combined based on PMD camera and colour imagery shot.This system is made up of PMD camera, camera, computing machine and data acquisition and analysis software, and three-dimensional reconstruction comprises the demarcation of camera system, and image information collecting and different scale images information three-dimensional are rebuild.PMD camera obtains object distance by flight (coming and going) time of detecting optical pulses, and obtains the depth information of entire image simultaneously.System basic functional principle is: PMD camera and camera are placed up and down, fixed by link, two cameras trigger dynamic acquisition image simultaneously, image is gathered by data acquisition software and preserves, be supplied to central processing unit, mutual relationship in the three-dimensional space position that can be calculated PMD camera and colour imagery shot of method for reconstructing between color information, thus set up color three dimension spatial information, for application in agricultural such as the navigation of robot, obstacle, identification provide corresponding reference information to positioning system etc.

The depth image of two width different resolutions and coloured image are carried out information fusion by the present invention, thus obtain colored three-dimensional information, the accuracy mainly utilizing PMD camera depth information to measure, drastically increase efficiency and the precision of three-dimensional reconstruction, equipment needed thereby is simple to operate, installation is easy; The Image Acquisition in early stage and the data analysis in later stage are combined, substantially increases the dependable with function of system.

As the novel target three-dimensional reconstruction system of one and method, the method introduces the information fusion three-dimensional rebuilding method that PMD camera and common color camera based on laser time of flight combine, wherein PMD camera is not singly to provide the depth data of three-dimensional reconstruction, and the stability that plot of light intensity picture corresponding to depth information can improve three-dimensional rebuilding method can be collected, thus significantly improve the precision of three-dimensional reconstruction.Application multi-visual information merges three-dimensional reconstruction algorithm, by the information that fathoms of PMD camera, the high stable characteristic of intensity signal and general camera fast and the feature of colour merge, to achieve in agricultural the high precision of the target colors such as fruit, identify and location fast.

System architecture building form of the present invention, by the PMD camera of employing laser time of flight principle and the textural association of colour imagery shot, utilize method for reconstructing find the relation of the spatial coordinated information of PMD camera and the color information of color camera and merge, thus guarantee system quick and precisely obtain spatial color three-dimensional information.

Combination scaling method of the present invention, carry out combination by the plot of light intensity picture and common camera adopting PMD camera to demarcate, two cameras using chessboard or dot matrix correction plate to combine calibration resolution not identical rotate and translation matrix, ensure the unification mutually of Two coordinate system information.

Color three dimension method for reconstructing of the present invention, it is the inside and outside parameter in conjunction with two cameras based on the three dimensional space coordinate information of PMD camera, change into the pixel coordinate of color camera as calculated, and the chromatic information that contact is corresponding with pixel, thus complete the color reconstruction of three-dimensional information in the visual field.

Certainly; the present invention also can have other various embodiments; when not deviating from the present invention's spirit and essence thereof; those of ordinary skill in the art are when making various corresponding change and distortion according to the present invention, but these change accordingly and are out of shape the protection domain that all should belong to the claim appended by the present invention.

Claims (10)

1., based on a color three dimension reconstructing system for PMD camera and camera, it is characterized in that, comprising:

Independent demarcating module, for demarcating PMD camera, camera, obtains described PMD camera, the inner parameter of described camera, external parameter and distortion parameter;

Combination demarcating module, for the image gathered respectively described PMD camera, described camera, solve relative rotation matrices between image after corresponding correction, described PMD camera, described camera coordinate system and translation vector according to described inner parameter, described external parameter and described distortion parameter;

Three-dimensional reconstruction module, for choosing the pixel at four edges in image after described correction, and solve the corresponding point on described camera according to described relative rotation matrices and translation vector, obtain image border corresponding in the image of described camera collection, and after being partitioned into, be stored as new image.

2. color three dimension reconstructing system according to claim 1, is characterized in that, described independent demarcating module is also for being optimized described inner parameter, external parameter.

3. color three dimension reconstructing system according to claim 1, it is characterized in that, described combination demarcating module also for: after described camera, PMD camera are fixing up and down, gather at least three images from different perspectives simultaneously, detect the unique point of often opening in photo, utilizing the orthogonality of rotation matrix and described inside and outside parameter and distortion parameter according to described unique point, by solving linear equation, asking for described relative rotation matrices and translation vector.

4. the color three dimension reconstructing system according to claim 1,2 or 3, is characterized in that, described three-dimensional reconstruction module obtains the corresponding point of image on described camera after described correction in the following ways:

P _web(X _c，Y _c，Z _c)＝RP _pmd(X ₁，Y ₁，Z ₁)+T

Wherein, (X ₁, Y ₁, Z ₁) be the edge pixel point of PMD collected by camera to the rear image of correction, (X _c, Y _c, Z _c) be the corresponding point on described camera, R, T are relative rotation matrices between two camera coordinate systems and translation matrix, P _pmdthe volume coordinate representing under PMD camera coordinates system a bit, P _wedbe the volume coordinate representing under camera coordinate system a bit, subscript pmd represents that, under PMD coordinate system, subscript web represents under camera coordinate system.

5. the color three dimension reconstructing system according to claim 1,2 or 3, is characterized in that, described three-dimensional reconstruction module also for: utilize described inner parameter, external parameter obtain image after distortion correction; Solve the common field range of described PMD camera, described camera; Solve the RGB chromatic information that XYZ space positional information is corresponding, utilize 4 weighted sums to solve; Utilize bilinear interpolation method to obtain high resolving power XYZ-RGB image information according to described common field range, complete three-dimensional reconstruction.

6., based on a color three dimension method for reconstructing for PMD camera and camera, it is characterized in that, comprising:

Step one, demarcates PMD camera, camera, obtains described PMD camera, the inner parameter of described camera, external parameter;

Step 2, to the image that described PMD camera, described camera gather respectively, relative rotation matrices between image after corresponding correction, camera coordinates system and translation vector is solved by the inner parameter of described PMD camera, described video camera and distortion parameter, and trimming edge quadrate;

Step 3, choose the pixel at four edges in image after described correction, and solve the corresponding point on described camera according to described relative rotation matrices and translation vector, obtain image border corresponding in the image of described camera collection, and after being partitioned into, be stored as new image.

7. color three dimension method for reconstructing according to claim 6, is characterized in that, in described step one, comprising:

B1. scaling board is obtained;

B2. move scaling board or camera, take at least 8 photos from different perspectives;

B3. all unique points of often opening in photo are detected;

B4. utilizing the orthogonality of rotation matrix according to described unique point, by solving linear equation, obtaining described inner parameter and external parameter.

8. color three dimension method for reconstructing according to claim 6, is characterized in that, in described step 2, comprising:

C1. scaling board is obtained;

C2. move scaling board or camera, take at least 3 photos from different perspectives simultaneously;

C3. all unique points of often opening in photo are detected;

C4. utilize described inside and outside parameter to carry out distortion correction to image, utilizing the orthogonality of rotation matrix according to described unique point, by solving linear equation, obtaining described rotation matrix and translation matrix;

C5. the described rotation matrix after utilizing least square method optimization to optimize and translation matrix.

9. the color three dimension method for reconstructing according to claim 6,7 or 8, is characterized in that, in described step 3, comprising:

Obtain the corresponding point of image on described camera after described correction in the following ways:

P _web(X _c，Y _c，Z _c)＝RP _pmd(X ₁，Y ₁，Z ₁)+T

Wherein, (X ₁, Y ₁, Z ₁) be the edge pixel point of PMD collected by camera to the rear image of correction, (X _c, Y _c, Z _c) be the corresponding point on described camera, R, T are relative rotation matrices between two camera coordinate systems and translation matrix, P _pmdthe volume coordinate representing under PMD camera coordinates system a bit, P _wedbe the volume coordinate representing under camera coordinate system a bit, subscript pmd represents that, under PMD coordinate system, subscript web represents under camera coordinate system.

10. the color three dimension method for reconstructing according to claim 6,7 or 8, is characterized in that, in described step 3, comprising:

D1. utilize described inner parameter, external parameter obtains image after distortion correction;

D2. the common field range of described PMD camera, described camera is solved;

D3. solve the RGB chromatic information that XYZ space positional information is corresponding, utilize 4 weighted sums to solve;

D4. utilize bilinear interpolation method to obtain high resolving power XYZ-RGB image information according to described common field range, complete three-dimensional reconstruction.