CN103971405A - Method for three-dimensional reconstruction of laser speckle structured light and depth information - Google Patents

Abstract

The invention provides a method for three-dimensional reconstruction of laser speckle structured light and depth information. The three-dimensional reconstruction technology is an important subject for machine vision research and refers to the content that a three-dimensional space geometrical shape of a three-dimensional body is restored through images of the three-dimensional body. Generally, three-dimensional reconstruction is conducted through the binocular parallax principle of a binocular camera or through a triangulation method or space codes are obtained through the structured light and the depth information is obtained through the triangulation method. The method aims at obtaining the depth information through the laser speckle structured light, a similar invention such as the kinect of the Microsoft Corporation also obtains the depth information (namely different depths are matched through a cross-correlation function of laser speckles) of an object through the method, and the difference is an algorithm for obtaining the depth information through the speckles. According to the method, parallel code number sorting is conducted on each pixel block one by one by a thinning window through multiple support vector machines, so that the depth of each pixel window is obtained, coordinates under a world coordinate system of the object are obtained by inversely solving a camera model through the depth information, and therefore the depth information with the higher accuracy can be obtained.

Description

The three-dimensional rebuilding method of a kind of laser speckle structured light and depth information

Technical field

The invention belongs to machine vision research field, is a kind of three-dimensional reconstruction based on laser speckle structured light.

Background technology

Three-dimensional reconstruction is one of important topic of machine vision research, refers to that its image by three-dimensional body recovers the three dimensions geometric configuration of three-dimensional body.The mode of general three-dimensional reconstruction has by the binocular parallax principle of twin camera passes through triangulation, or obtains space encoding by structured light, obtains depth information by triangulation.

The invention is intended to obtain depth information with a kind of laser speckle structured light, similarly invention as the kinect of Microsoft be also the depth information (cross correlation function by laser speckle mates different depth) that obtains object by this kind of method, different is by the algorithm of the depth information of reentrying after speckle, the present invention's proposition encodes to obtain the degree of depth of each pixel window by many support vector machine parallel sorting to individual element piece by refinement window, obtain the coordinate of object under world coordinate system by the anti-camera model of separating of depth information.

Three-dimensional reconstruction is generally the world coordinates that solves object point according to the video camera of having demarcated by the inside and outside portion of known demarcation parameter, and solution procedure has determined that because of the restriction of camera model and solving equation group the result solving can only be a ray equation, and cannot directly obtain by its equation 3 world coordinatess of object point.The invention is intended to propose a kind of new mode and obtain the depth coordinate of object point, and then carry out 3 coordinates under the three-dimensional world coordinate system of direct solution object point by the equation of linear camera model and radial distortion.

When laser is by coarse transparent surface (as frosted glass) and while being incident upon body surface, can observe the light and shade spot of random distribution at body surface, i.e. laser speckle.The generation of this laser speckle is in the time that Ear Mucosa Treated by He Ne Laser Irradiation is on rough surface, on surface, every bit is all wanted scattered light, and all receiving the irradiation of these coherent scattering light, a space point just forms laser speckle, speckle field is divided into two kinds by light path, a kind of speckle field is propagated and is formed (also referred to as objective speckle) in free space, another kind forms (also referred to as subjective speckle) by lens imaging, and that the present invention uses is the latter.

And the regular speckle forming for space each point, wherein comprise the depth information in every, space, catch formed speckle by thermal camera, by the extraction of feature, the training of sorter (SVM), finally can obtain the depth information comprising in the speckle forming about every, space.

By obtained depth information, pass through formula $\mathrm{\λ}\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_u& s& u_o\\ 0& f_v& v_o\\ 0& 0& 1\end{array}\right]\left(\begin{array}{cc}R& T\end{array}\right)\left[\begin{array}{c}{x}_w\\ y_w\\ z_w\\ 1\end{array}\right]\begin{array}{c}=K\end{array}\left(\begin{array}{cc}R& T\end{array}\right)\left[\begin{array}{c}{x}_w\\ y_w\\ z_w\\ 1\end{array}\right]$ Carry out the three-dimensional coordinate of solution room each point.

Summary of the invention

What camera calibration process adopted conventionally is classical pin-hole imaging model, and this model is generally described with following formula:

$\mathrm{\λ}\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_u& s& u_o\\ 0& f_v& v_o\\ 0& 0& 1\end{array}\right]\left(\begin{array}{cc}R& T\end{array}\right)\left[\begin{array}{c}{x}_w\\ y_w\\ z_w\\ 1\end{array}\right]\begin{array}{c}=K\end{array}\left(\begin{array}{cc}R& T\end{array}\right)\left[\begin{array}{c}{x}_w\\ y_w\\ z_w\\ 1\end{array}\right]---\left(1\right)$

Wherein: in space, the homogeneous coordinates of any point P in world coordinate system are P _w(x _wy _wz _w1) ^t, the homogeneous coordinates in image coordinate system are p (u v 1) ^t.λ is arbitrary scale factor; K is camera intrinsic parameter matrix, and wherein s is the pattern distortion factor, f _u, f _vfor the unable coordinate of image picture point in u direction and v direction is to the scale-up factor of image pixel coordinate, i.e. effective focal length, (u _ov _o) be the image coordinate of primary optical axis and plane of delineation intersection point.R is a rotation matrix that 3*3 unit is orthogonal; T is a translation vector; Meanwhile, (R, T) is the position of camera coordinate system with respect to world coordinate system.

By above-mentioned formula, the in the situation that of known inside and outside parameter, can obtain two system of equations, wherein unknown is the object point coordinate under world coordinate system, so, if solved and must first be tried to achieve a coordinate by above-mentioned formula, the present invention solves all the other two coordinates by the unknown number number that solves depth information and reduce in above-mentioned equation.

Use laser speckle to encode to space, the present invention uses subjective speckle, i.e. the speckle that scioptics imaging forms is encoded to the space of certain angle, obtains coded image by infrared camera.

Be used as the training set of proper vector as SVM by the feature extraction of the coded image to different distance, send in SVM and train, it should be noted that, each sub-picture is all the conduct training set of the affiliated SVM of distance separately, will obtain like this one group of SVM, also here determine for the precision that finally solves the depth distance obtaining, distance interval in the scope of speckle encoding between different images is less naturally better, but obtain simultaneously the number of SVM also can become many thereupon, like this can the cost more time on the time of compute depth distance subsequently, have not little impact for the real-time of image.

The calculating of depth distance is the test set that is used as SVM by the feature extraction of speckle test pattern, encode by multiclass SVM and many SVM are carried out to computing simultaneously and finally obtain one group of binary coding, be multiplied by quality coefficient by this group coding and obtain concrete depth information.In the time obtaining the depth information of target area, the zones of different even distance between different pixels is different, want to obtain desirable depth information, need to be in the time of coupling SVM, extract computing to making window in test set, even for accurately will be to each speckle, pixel be used as window, in window, carry out feature extraction, as the training set of SVM.

After Depth Information Acquistion, pass through $\mathrm{\λ}\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_u& s& u_o\\ 0& f_v& v_o\\ 0& 0& 1\end{array}\right]\left(\begin{array}{cc}R& T\end{array}\right)\left[\begin{array}{c}{x}_w\\ y_w\\ z_w\\ 1\end{array}\right]\begin{array}{c}=K\end{array}\left(\begin{array}{cc}R& T\end{array}\right)\left[\begin{array}{c}{x}_w\\ y_w\\ z_w\\ 1\end{array}\right]$ Formula is brought to solve corresponding all the other coordinates into and is obtained the three-dimensional coordinate under object point world coordinate system; Can calculate easily rotation matrix and translation matrix by suitably choosing world coordinate system, when mobile, the three-dimensional reconstruction of object is fine so.

Brief description of the drawings

Fig. 1 main flow chart of the present invention.

The infrared SVM calibration process of Fig. 2 the present invention schematic diagram.

Fig. 3 test set of the present invention is sent into the schematic diagram of classifying in SVM.Fig. 4 feature extraction of the present invention with organize schematic diagram.

Certain SVM training process flow diagram of Fig. 5 the present invention.

Fig. 6 SVM of the present invention trains process flow diagram in batches.

Fig. 7 svm classifier process flow diagram of the present invention.

Embodiment

Basic hardware used in the present invention:

1, infrared camera;

2, generating laser;

3,3 of diffractive optical components;

Below in conjunction with accompanying drawing, elaborate embodiments of the present invention:

Fig. 1 has shown that the present invention is to three-dimensional reconstruction flow process, and embodiment is described as follows:

1, algorithm flow of the present invention is as follows:

(1) demarcate first.Demarcate first the stated accuracy having determined in use procedure below, because field involved in the present invention is at three-dimensional reconstruction, stated accuracy claimed range is wider, can choose the scaling method of different accuracy under different three-dimensional reconstruction scenes.

(2) choose suitable world coordinate system.Three-dimensional scenic is carried out in process of reconstruction, sometimes need attitude adjustment or movement to video camera; For not needing in mobile situation, rotation matrix and translation matrix not variation there is no impact to later stage calculating three-dimensional coordinate; Situation about moving for needs as long as choose suitable world coordinate system, solves rotation and translation matrix by analyzing the motion track of video camera in video camera moving process.

(3) laser speckle structured light projection.The laser that laser generator produces obtains required speckle pattern by optics (as diffractive optical components) scattering, each zero level energy for safe laser regulation is no more than the 0.4mw upper limit, we need special optics that the laser zero level energy scattering out is reduced to this below scope, the optical design (number of patent application CN200880119911) of the patent of applying at home such as Prime Sensez for zero level is reduced.

(4) as Fig. 2, the laser speckle scattering out has certain calibration range to space, this scope can obtain by the scattering angle of diffuse optical parts, within the scope of this, the specific demarcation thing of the interior use that keeps at a certain distance away carries out mark to speckle, if spread is that-30 degree (comprise upper and lower to 30 degree, left and right), depth distance is 0.5-3.5 rice, can get speckle image one time every 1mm with the demarcation thing that is greater than range of scatter angles, and this spacing distance also determines to ask for the precision of depth distance below.

(5) in step 4, in the scope of 3 meters, get and once demarcate speckle image every 1mm, have like this 300 sub-pictures, every width image is carried out to feature extraction, by PCA method, the speckle image of window is carried out to feature extraction, simultaneously for make up PCA some time the inefficacy of catching to feature, make 300 points of training sets by the feature such as brightness, diameter of extracting speckle.As Fig. 4

(6), in step 5, the flow process of PCA algorithm is to its normalization matrix X separately of the every width image calculation in 300 width images _i ^*, calculate X _i ^*covariance matrix C; Covariance matrix is carried out to Eigenvalues Decomposition, choose maximum p eigenwert characteristic of correspondence vector composition projection matrix; Here the standard of selected characteristic value is to be comprehensively greater than 90% of all eigenwert summations for selected eigenwert; Original sample matrix is carried out to projection, obtain its major component S ^*.The concrete formula using is as follows:

${x^{*}}_i=x_i-\stackrel{\‾}{X},$ Wherein $\stackrel{\‾}{X}=\frac{\underset{i=1}{\overset{N}{\mathrm{\Σ}}}{x}_i}{N}$

$C=\frac{S^{T}S}{N-1}$

(C-λ _i)p＝0

S ^*＝S*p

(7) because of the defect of PCA self, under certain conditions, PCA method can can't be very effective to the extraction of some feature, form composite character so need to add other features to be used as characteristic attribute in major component feature set, concrete is exactly by speckle size, brightness jointly forms composite character as new feature and major component feature and is used as training set, the method of its mixing is that its speckle characteristics of major component of extracting speckle place block of pixels by judgement is consistent, as: the matrix that certain speckle place picture element matrix is formed for a 115*109, if be selected in the corresponding major component feature of this matrix so, speckle attributive character in the time mixing in this matrix is consistent.

(8) training set is sent in SVM and trained, SVM is that a quantity is followed the equal SVM group of training set quantity here, and in this group SVM, each SVM trains a training set, and the organizational form between SVM is considered parallel or tandem.As Fig. 6.

(9) in step 8, be consistent to the training of each SVM, as Fig. 5 specific practice is, the kernel function of first selecting SVM to use, is used RBF kernel function here, and its RBF kernel functional parameter σ initial value is set; Its feature space of calculation of parameter using the feature set of extracting in step 7 as RBF; Solve Lagrange-duality antithesis factor-alpha, calculate sorting parameter w, b by α; The disaggregated model calculating is calculated to its average accuracy by the method for crosscheck, if average accuracy is greater than 90%, training finishes, otherwise, to kernel functional parameter, σ resets, and so circulation of training again, until find the kernel functional parameter σ that average accuracy is the highest.The part formula of wherein using is as follows:

$K(x_1,x_2)=e^{(-\frac{{\left|\right|x_1-x_2\left|\right|}^2}{2{\mathrm{\σ}}^2})}$

${{\mathrm{\α}}_i}^{\mathrm{new}}={{\mathrm{\α}}_i}^{\mathrm{old}}+\frac{y_i(E_i-E_j)}{\mathrm{\η}}$

E _i=r _i-y _i, η=K (x ₁, x ₁)+K (x ₁, x ₁)-2K (x ₁, x ₂), r is geometric distance, y is tag along sort.

$w=\underset{i=1}{\overset{m}{\mathrm{\Σ}}}{\mathrm{\α}}_i{y}_i{x}_i,b=-\frac{{\max }_{i:y^i=-1}{w}^T{x}_i+{\min }_{:y^i=1}{w}^T{x}_i}{2}$

(10) by the projection of step 3, can obtain equally the speckle image to certain particular space, classification process is also from here on.As Fig. 7

(11) pixel of supposing speckle image is 1080*768, need to carry out window to the speckle image obtaining, in order to obtain the accurate as far as possible depth information of every bit, can consider on ageing basis as far as possible by window drawdown ratio as the window of choosing be 6*6, speckle image is also just divided into 180*128 piece.

(12) each piece window is carried out to feature extraction, here feature used is that the feature extracted with step 5 is consistent, suppose that the feature of extracting is one 5 dimension row vector, pass through so a series of processing, the image of this width 1080*768 has just been divided into the test set of 180*128*5 part.

(13) test set of step 9 is sent in SVM and classified, each test set can obtain the binary coding of a group 300, as Fig. 3.

(14) binary coding obtaining in step 10 is passed through to formula D=L _s+ B*dist wherein D represents depth distance, L _sthe initial distance that represents the depth distance in step 4, B represents that the binary coding of SVM is converted to the number of 10 systems, dist represents spacing distance.Try to achieve the depth distance of every window by formula, the distance between every window is tried to achieve to the depth information of each pixel by image interpolation algorithm.

(15) pass through formula $\mathrm{\λ}\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_u& s& u_o\\ 0& f_v& v_o\\ 0& 0& 1\end{array}\right]\left(\begin{array}{cc}R& T\end{array}\right)\left[\begin{array}{c}{x}_w\\ y_w\\ z_w\\ 1\end{array}\right]\begin{array}{c}=K\end{array}\left(\begin{array}{cc}R& T\end{array}\right)\left[\begin{array}{c}{x}_w\\ y_w\\ z_w\\ 1\end{array}\right],$ The three-dimensional coordinate of obtaining every some pixel carries out scene rebuilding.

Claims (10)

1. the three-dimensional rebuilding method that obtains depth information based on laser speckle structured light by support vector machine, is characterized in that, comprises

Following committed step:

Step 1: camera calibration;

Step 2: choose suitable coordinate system;

Step 3: laser speckle structured light projection;

Step 4: obtain and demarcate speckle image and speckle test pattern;

Step 5: feature extraction;

Step 6: SVM is training in batches;

Step 7: the test pattern that step 4 is obtained carries out window;

Step 8: feature extraction;

Step 9: svm classifier;

Step 10: according to classification results compute depth.

2. according to claim 1, the three-dimensional rebuilding method that obtains depth information by support vector machine based on laser speckle structured light is characterized in that: in step 1, the camera of demarcating is infrared camera and common CCD camera, scaling method can be used but not limited to following methods: two-step approach, three-step approach, least square method etc.

3. according to claim 1, the three-dimensional rebuilding method that obtains depth information by support vector machine based on laser speckle structured light is characterized in that: in step 2, in the time choosing world coordinate system, generally choose the camera coordinate system at thermal camera place, making has controlled moving can calculate fast rotation matrix and translation matrix once video camera.

4. according to claim 1, the three-dimensional rebuilding method that obtains depth information by support vector machine based on laser speckle structured light is characterized in that: in step 3, laser speckle projection will be passed through the diffractive optical components group of particular design, to at least pass through the diffractive optical components (DOE) of three parts specific to the laser penetrating from laser maker, by first component, for by laser light scattering certain angle, diffraction goes out the speckle pattern of given shape simultaneously; The 2nd DOE position can be contained the pattern that first component institute diffraction goes out substantially, by the diffraction of the 2nd DOE, speckle is done to optical convolution and form specific pattern, that spot size is after projecting to the requirement of these two DOE, the spot size catching after returning by target area scattering is at least greater than or is substantially all greater than the size of a pixel, the 3rd optics selects to have with the 2nd DOE the physical dimension in same space cycle, but design DOE transport function, make to be less than 0.1mw by its zero level energy.

5. according to claim 1, the three-dimensional rebuilding method that obtains depth information by support vector machine based on laser speckle structured light is characterized in that: in step 4, for demarcating speckle image, get the sampled images that dist is interval, every width image will cover whole demarcation planes, make sampled images speckle can fully characterize the speckle characteristics of certain distance, according to claim 1, the three-dimensional rebuilding method that obtains depth information by support vector machine based on laser speckle structured light is characterized in that: in step 5, for the pattern collecting, carry out following feature extraction but be not limited to these: PCA major component, spot size, brightness, gray shade scale, and coefficient of autocorrelation and Hurst index between them.

6. according to claim 1, the three-dimensional rebuilding method that obtains depth information by support vector machine based on laser speckle structured light is characterized in that: in step 6, the feature of each width figure under dist distance is sent in following SVM group as training set: the parameter of each SVM is all consistent, and use RBF kernel function, make its kernel functional parameter σ and regularization parameter c optimum by SVM cross validation, optimum standard is that its training result is more than 90%.

7. according to claim 1, the three-dimensional rebuilding method that obtains depth information by support vector machine based on laser speckle structured light is characterized in that: in step 7, described window method can arbitrarily be chosen, also can realize self-adapting window by window function, no matter but which kind of window method is all the consideration based on computing time and precision, post-processed speed at least reaches 60ms/f.

8. according to claim 1, the three-dimensional rebuilding method that obtains depth information by support vector machine based on laser speckle structured light is characterized in that: in step 8, related feature extraction is consistent with the feature extraction described in claim 6, can be, but not limited to following characteristics: spot size, brightness, gray shade scale, and coefficient of autocorrelation and Hurst index between them.

9. according to claim 1, the three-dimensional rebuilding method that obtains depth information by support vector machine based on laser speckle structured light is characterized in that: in step 9, test set is sent in SVM group, and the classification time for each window sample in certain SVM is no more than 0.015ms.

10. according to claim 1, the three-dimensional rebuilding method that obtains depth information by support vector machine based on laser speckle structured light is characterized in that: in step 10, the two-stage system data that obtain according to SVM group, by formula D=L _s+ B*dist calculates the depth information of each window representative, then in window pixel, carries out interpolation according to selected window, obtains the depth information of each pixel.