Summary of the invention
Technical matters: the purpose of this invention is to provide a kind of three-dimensional rebuilding method based on regional depth rim detection and binocular solid coupling, to and possess based on the window of regional depth rim detection structure, normalized crosscorrelation coefficient and quicken the more Stereo Matching Algorithm of the confidence spread optimization method good combination of new system, and be applied to binocular vision system, rapidly and accurately the three-dimensional model of reconstructed object thing.
Technical scheme: at first carry out camera calibration, two width of cloth images of photographic subjects thing from different perspectives then, carrying out polar curve according to nominal data proofreaies and correct and image transformation, in the fixedly neighborhood of each pixel, utilize the search of second-order differential edge detection operator then and make up a virtual sealing depth edge, form corresponding support window; Use support pixel in the support window by the coupling cost between the normalized crosscorrelation coefficient calculations center pixel then, and, then utilize the confidence spread optimization algorithm to obtain parallax as the data item of global energy optimization equation.Introduce a kind of parallax exact algorithm of estimating based on sub-pix afterwards and repair some discontinuous quantification parallax value; At last, according to the calibration information of binocular camera and the pixel parallax three-dimensional point cloud of reconstructed object thing as a result, obtain Three-dimension Target information.
This three-dimensional rebuilding method contains the following step successively:
Step 1) camera calibration: use the binocular camera of two horizontal positioned to take the scaling board that posts gauge point, the position of at first adjusting binocular camera makes its optical axis approximate parallel, then from about two angles take scaling board simultaneously, then according to the scaling board image, respectively two video cameras are demarcated, obtained intrinsic parameter matrix A separately
L, A
RWith outer parameter matrix [R
Lt
L], [R
Rt
R];
Step 2) object Image Acquisition: keep the shooting angle of video camera constant, two width of cloth images of photographic subjects thing, wherein left lens shooting is left image, right lens shooting be right image;
The step 3) polar curve is proofreaied and correct and image transformation: the camera interior and exterior parameter matrix that obtains based on calibration process in the step 1), the utilization method for correcting polar line rotates two width of cloth object images respectively, form parallel polar curve, convert the original image of object to image to be matched according to the polar curve after proofreading and correct then, make matched pixel corresponding in two images to be matched have identical ordinate;
Step 4) is constructed support window based on the regional depth rim detection: the closed boundary of the support window of each pixel p in the image to be matched is regarded as the set of a class degree of depth transition pixel, utilize the second-order differential edge detection operator from pixel p, to around search the frontier point of degree of depth transition pixel as support window, connect these frontier points, form the border of a virtual closed edge, form the support window W of pixel p as window
pIn like manner, the support window W of each pixel q in another image to be matched
qCan construct in the same way;
Step 5) is calculated the coupling cost, comprises following two sub-steps:
Step 5.1: appointing the width of cloth in the image to be matched of getting the correction of two width of cloth process polar curve is benchmark image, and another width of cloth is a registering images; Step 4) obtains that coordinate is (x, the support window W of any pixel p y) in the benchmark image
p, and the support window W of possible match point q on the corresponding polar curve of p in the registering images
q, the q point coordinate be (x+d, y), d ∈ [d wherein
Mind
Max], d
MinBe minimum parallax value, d
MaxBe the maximum disparity value;
Step 5.2: in benchmark image, choose size and shape and support window W
pAnd W
qThe window of common factor window unanimity
In registering images, choose size and shape and support window W
pAnd W
qThe window of common factor window unanimity
According to formula
Calculate and support pixel i
lAnd i
rThe normalized crosscorrelation coefficient, as benchmark pixel p when parallax is d, the coupling cost of corresponding registration pixel q, be designated as Φ (p, d), I (i
l) and I (i
r) be respectively pixel i
lAnd i
rGray-scale value;
With
Be respectively benchmark image and registering images at the common factor window
With
Interior average gray;
Step 6) confidence spread optimized energy function: iteration is propagated degree of confidence message in benchmark image, can obtain the energy optimization of entire image, employing is quickened more, and new system passes to next pixel with degree of confidence message from a pixel: when iteration t each time, on left and right, upper and lower four direction, spread news respectively, and before accepted the direction of message along a pixel, upgraded the degree of confidence message of this pixel immediately; After message propagation finishes, calculate the degree of confidence of each pixel p this moment, choosing the parallax value with maximum confidence is the parallax d of this pixel
p
Step 6.1: the level and smooth cost of calculating pixel
(s p) has determined neighbor p and the s support for the degree of confidence message propagation to level and smooth cost Ψ, at first uses formula
Calculate level and smooth cost Ψ between each pixel p and its right neighbor and the following neighbor (p, s), wherein, Δ I
PsBe the gray scale difference value between pixel p and the s, by Δ I
Ps=| I (s)-I (p) | calculate, I (s) and I (p) are respectively the gray-scale value of pixel s and p; L is a static threshold, and u and P are level and smooth compensation term, all is static constant; d
sThe parallax value of pixel s in the time of for this reason, d is the parallax value to be matched of pixel p;
Step 6.2: the degree of confidence message of calculating pixel
When the t time iteration propagated degree of confidence message, the message that pixel s passes to neighbor p was designated as
The coupling cost Φ (s, the d that have comprised pixel s
s), level and smooth cost Ψ (s, p), and three message that pixel is accepted when the t-1 time iteration that (do not comprise p) in 4 neighborhoods of pixel s:
Wherein, Φ (s, d
s) be that the to be matched pixel of pixel s in registering images is corresponding to parallax value d
sThe coupling cost, obtain by step 5); Ψ (s, p)=(p s) is level and smooth cost to Ψ; N (s) p represent the neighbor of s but do not comprise p;
Step 6.3: the degree of confidence of calculating pixel is also obtained parallax
According to the acceleration of message new system more,, give adjacent left pixel with the message propagation of each pixel successively from the rightest pixel of benchmark image; From the most left pixel, give adjacent right pixel with the message propagation of each pixel successively again; Pass through the propagation of lower and upper both direction again by this rule, promptly finish 1 iteration of the degree of confidence message of entire image and upgrade; After T time iteration is upgraded, the degree of confidence b of pixel p
p(d) by
Calculate, Φ (p is the coupling cost of pixel p d), obtains by step 5,
Neighbor s passes to the degree of confidence message of pixel p for this moment, then calculates all parallax d ∈ [d in the disparity range
Mind
Max] corresponding degree of confidence, d
MinBe minimum parallax value, d
MaxBe the maximum disparity value, choose the parallax d that the parallax value with maximum confidence is a pixel p
p, by
Calculate;
The step 7) parallax is accurate: utilize sub-pix to estimate
The integer parallax value that step 6) is obtained is converted into the decimal form, and obtains more accurate sub-pix parallax Dispartiy (p), as the final parallax value of pixel p, wherein, three discrete parallax value d
p, d
p+ 1, d
p-1 obtains Φ (p, d by step 6
p), Φ (p, d
p+ 1) and Φ (p, d
p-1) the coupling cost that obtains for step 5;
The three-dimensional point cloud of step 8) reconstructed object thing: P
LBe the pixel of impact point P correspondence in benchmark image, P
RBe the matched pixel of impact point P correspondence in registering images, P
LCoordinate be (x
L, y
L), P
RCoordinate be (x
R, y
R), pass through x
R=x
L+ Disparity (P
L) and y
R=y
LObtain Disparity (P
L) the pixel P that obtains for step 7)
LFinal parallax value, camera interior and exterior parameter matrix that obtains according to step 1) and pixel matching are to (P then
L, P
R) calculate the three-dimensional coordinate of impact point P; Calculate the three-dimensional coordinate of all impact points, and then obtain the three-dimensional point cloud model and the three-dimensional information of whole object.
Wherein, step 4) comprise following three sub-steps based on regional depth rim detection structure support window,
Step 4.1: two images to be matched after proofreading and correct through polar curve are carried out medium filtering, improve picture quality, reduce noise to the edge detection operator Effect on Performance;
Step 4.2: the summit on eight directions of search support window
In two images to be matched, from the horizontal direction of each pixel p, be rotated counterclockwise successively 45 ° can supported window 8 zenith directions.At each zenith directions k (k=0,1,2...7) the last second-order differential operator that passes through
Begin to search for regional depth marginal point h (θ on this direction from central point p
k), wherein,
Be pixel (p+h
I+1) second order gray scale differential value on direction k, I (p+h
I+1), I (p+h
I-1) and I (p+h
i) be respectively pixel (p+h
I+1), (p+h
I-1) and (p+h
i) gray-scale value, h
i∈ 1,2,3...}
k, θ
k(k=0,1, be eight zenith directions of support window 2...7), when satisfying formula
The time, h (θ
k)=h
i, and the summit of going up support window as direction k;
Step 4.3: step-length is h (θ on eight directions of connection pixel p
k) the summit, form the regional depth edge that p is ordered, choosing its area surrounded is corresponding support window W
p
Beneficial effect: compared with prior art, the present invention has following advantage: based on the support window of regional depth rim detection structure based on hypothesis: the pixel of the similar degree of depth has similar parallax in the neighborhood, utilize the size and the shape of the variation constraint window of depth information in the zone, comprise the support pixel that has the similar degree of depth to center pixel.Simultaneously in conjunction with normalized crosscorrelation coefficient calculations method, can overcome effectively that unbalanced illumination obtains stable coupling cost to the influence of three-dimensional reconstruction system in the actual environment, and be applicable to follow-up energy optimization algorithm.Possess and quicken the texture information that the confidence spread optimization algorithm of new system more makes full use of all pixels of entire image and seek optimum solution,, repair the match information of occluded pixels, reduced coupling uncertainty and error with the minimization of energy function.The parallax exact method of estimating based on sub-pix can be converted into the parallax value of integer form more accurate decimal, obtains more level and smooth object three-dimensional point cloud.The present invention fully combines edge detecting technology, calculates and the advantage of confidence spread optimization algorithm based on the parallax of depth information, overcome the subject matter that runs into when existing Stereo Matching Technology is applied in actual three-dimensional reconstruction system, when guaranteeing to rebuild accuracy, improved the arithmetic speed and the automaticity of process of reconstruction.
Embodiment
With reference to the accompanying drawings, specific embodiments of the present invention are done more detailed description.The programming implementation tool is selected Visual C++2003.net and OpenCV Flame Image Process function library for use, selects for use people's face of low texture and the uneven significantly watermelon of illumination as the three-dimensional reconstruction target in the indoor environment.
Fig. 1 is entire flow figure of the present invention.
Fig. 2 is system model of the present invention and principle schematic.Use two ccd video cameras respectively to take a width of cloth target image, O simultaneously from two different angles respectively
L, O
RBe respectively the photocentre of two video cameras, I
L, I
RBe respectively the imaging plane of two video cameras, P is a space object point for the treatment of on the object of reconstruct, P
L, P
RBe object point P imaging point on two video camera imaging planes respectively.This is a pair of match point by the same space object point imaging point on the different cameras imaging plane.Appoint and to get wherein that a width of cloth is a benchmark image, another width of cloth is a registering images, and the process of search for corresponding match point for each pixel in the benchmark image in registering images is called three-dimensional the coupling.After obtaining the matching relationship of pixel,,, carry out reverse computing, just can obtain the 3 d space coordinate of corresponding object point, thereby realize the three-dimensional reconstruction of image in conjunction with demarcating the camera interior and exterior parameter that obtains according to system model.
Fig. 4 is the schematic diagram of regional depth edge detecting technology, and the coordinate center is the center pixel of each support window, and ordinate is a pixel depth, and horizontal ordinate is the space length at neighborhood interior pixel and center.The image border is the set that the pixel of transition appears in colouring information, and they not only appear between the target, also appear between the pixel.And the closed boundary of the support window of each pixel can be regarded the set of this class transition pixel as.These window edge color of pixel information transition occurs having between the pixel of the similar degree of depth and the dissimilar degree of depth to center pixel.The regional depth edge detecting technology utilizes edge detection operator from each pixel, to around search the frontier point of degree of depth transition pixel as the support window of this pixel, in Fig. 4, be labeled as " ".Connect these frontier points, form a virtual closed edge, the fixedly neighborhood of this pixel is divided into outer peripheral areas and central area, select the central area then, and use pixel wherein to calculate the coupling cost as support window.Therefore, as Fig. 5 (a) (b), the regional depth edge that the fixedly neighborhood U of each pixel is configured is divided into two targets: as the central area W and the outer peripheral areas W-U of support window.Edge detection operator mainly comprises single order and second-order differential operator.When carrying out degree of depth transition pixel detection, only set up and searching to have the frontier point of the pixel of maximum differential value as support window based on the window of single order differential boundary operator; And the second-order differential boundary operator only need be sought the frontier point of second-order differential zero crossing as support window.
Fig. 5 is the support window synoptic diagram of pixel.Shown in Fig. 5 (a), p is pixel in the benchmark image, W
pFor with p being a support window zone based on depth edge detection structure of center pixel; Fig. 5 (b) is the support window in the registering images, and the frame of broken lines inner region is that the disparity range of the match point of benchmark pixel p correspondence is the hunting zone, and the match point of p drops in this scope, and q is a pixel in the hunting zone, W
qFor with q being the support window zone of center pixel, dashed region among Fig. 5 (c)
Be size in the benchmark image and shape and support window W
pAnd W
qThe window of common factor window unanimity; Dashed region among Fig. 5 (d)
Be size in the registering images and shape and support window W
pAnd W
qThe window of common factor window unanimity.The coupling cost Φ that is the pixel q of d with its parallax distance in pixel p and the registering images in the benchmark image (p d) is defined as:
Wherein, I (i
l) and I (i
r) be respectively pixel i
lAnd i
rGray-scale value;
With
Be respectively benchmark image and registering images at the common factor window
With
Interior average gray.
Method of the present invention specifically comprises following step:
Step 1: camera calibration
Use the binocular camera of two horizontal positioned to take the scaling board that posts gauge point, at first adjust the position of binocular camera, make the substantially parallel and left and right sides mirror of its optical axis be in suitable shooting angle, take scaling board simultaneously from two angles then.Adopt list of references " A Flexible New Technique for CameraCalibration " (Zhang Z Y among the present invention, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000,20 (11): a kind of calibration algorithm that is called the plane template method that proposes 1330-1334) is demarcated two video cameras according to the scaling board image respectively, obtains intrinsic parameter matrix A separately
L, A
RWith outer parameter matrix [R
Lt
L], [R
Rt
R].
Step 2: object Image Acquisition
Keep the shooting angle of video camera constant, two width of cloth images of photographic subjects thing, wherein left lens shooting is left image, right lens shooting be right image;
Step 3: polar curve is proofreaied and correct and image transformation
Fig. 3 is that polar curve is proofreaied and correct synoptic diagram.According to the polar curve constraint principles, certain pixel is positioned on the polar curve of this pixel correspondence in the matched pixel on the right figure among the left figure.And in the binocular reconstructing system of reality, because two video cameras can not guarantee absolute parallel shooting, need to introduce list of references " A compact algorithm for rectification ofstereo pairs.Machine Vision and Applications " (Fusiello A, Trucco E, Verri A.2000,12 (1): the polar curve correcting algorithm that proposes 16-22), the camera interior and exterior parameter that obtains according to step 1 calibration process, by suitable rotation, in two width of cloth images, form parallel polar curve, according to corrected polar curve conversion original image, finally form the image to be matched of object then.Corresponding matched pixel has identical ordinate in two images to be matched, thereby the search procedure of solid coupling has been dropped to one dimension from two dimension, has improved the operation efficiency and the precision of subsequent step.
Step 4:, comprise following three sub-steps according to regional depth edge detecting technology structure support window
Step 4.1: two images to be matched are carried out medium filtering, improve picture quality, reduce the influence of noise to edge detection operator.
Step 4.2: the summit on eight directions of search support window
In the image to be matched that after proofreading and correct, obtains through step 3 polar curve, from the horizontal direction of each pixel p, by shown in Figure 5, be rotated counterclockwise successively 45 ° can supported window 8 zenith directions.At each zenith directions k (k=0,1,2...7) the last second-order differential operator that passes through
Begin to search for regional depth marginal point h (θ on this direction from central point p
k), wherein,
Be pixel (p+h
I+1) second order gray scale differential value on direction k, I (p+h
I+1), I (p+h
I-1) and I (p+h
i) be respectively pixel (p+h
I+1), (p+h
I-1) and (p+h
i) gray-scale value, h
i∈ 1,2,3...}
k, θ
k(k=0,1,2...7) be eight zenith directions of support window, satisfy formula by working as
The time, h (θ
k)=h
i, and the summit of going up support window as direction k.
Step 4.3: step-length is h (θ on eight directions of connection pixel p
k) the summit, form the regional depth edge of p, the neighborhood that is fixed is divided into central area and outer peripheral areas.Choosing the central area is corresponding support window W
pThen obtain the support window W of pixel q in another image to be matched in the same way
q
Step 5: calculate the coupling cost, comprise following two sub-steps:
Step 5.1: appoint and to get that a width of cloth is a benchmark image in the image that two width of cloth proofread and correct through polar curve, another width of cloth is a registering images.Through the calculating of step 4.3, can obtain in the benchmark image coordinate for (x, the support window of any pixel p y) are W
p, and the support window W of possible match point q in the disparity range D in the registering images
qWherein, the q point coordinate be (x+d, y), d ∈ D, D=[d
Mind
Max], d
MinBe minimum parallax value, d
MaxBe the maximum disparity value.
Step 5.2: choose size and shape and W
pAnd W
qThe window of common factor window unanimity, by formula (1) calculates the normalized crosscorrelation coefficient of wherein supporting pixel, the coupling cost when being d as the pixel p parallax, be designated as Φ (p, d).
Step 6: confidence spread optimized energy function
Confidence spread optimization algorithm iteration in Markov Random Fields neighbours domain system is propagated degree of confidence message.The algorithm core is the message that propagates into next pixel p when the t time iteration from a pixel s, is designated as
Upgrade with the suitable manner iteration.Quickening more, when the new system decision passes to next pixel again with message.As shown in Figure 6, when iteration t each time, this system spreads news on left and right, upper and lower four direction respectively, and has before accepted the direction of message along a pixel, upgrades this pixel immediately.The advantage of this more new system is to carry out fast the iterative process of confidence spread optimization algorithm, and iteration all need be according to the order of Fig. 6 each time, the message of all pixels in the update image.Pixel s passes to pixel p when t iteration message comprised pixel s coupling cost Φ (s, d), level and smooth cost Ψ (s, p), and three message that pixel is accepted when t-1 iteration that (do not comprise p) in 4 neighborhoods of pixel s.
Step 6.1: the level and smooth cost of calculating pixel
(s p) has determined neighbor p and the s support for message propagation to level and smooth cost Ψ, has limited the message transmission between the discontinuous pixel of colouring information.Because Ψ (s, p)=(p, s), and in message process, level and smooth cost keeps constant to Ψ, so at first calculate the level and smooth cost between each pixel s and its right neighbor and the following neighbor.Use
Calculating Ψ (s, p).Wherein, Δ I
SpBe the gray scale difference value between pixel p and the s, by Δ I
Sp=| I (s)-I (p) | calculate, I (s) and I (p) are respectively the gray-scale value of pixel s and p; L is a static threshold, and u and P are level and smooth compensation term, all is static constant; d
sThe parallax value of pixel s in the time of for this reason, d is the parallax value to be matched of pixel p.
Step 6.2: the degree of confidence message of calculating pixel
During t iteration, the degree of confidence message that pixel s passes to pixel p is:
Wherein, Φ (s, d
s) be that the to be matched pixel of pixel s in another width of cloth image is corresponding to parallax d
sThe coupling cost, obtain by step 5; (s p) is level and smooth cost to Ψ; N (s) p represent the neighbor of s but do not comprise p.
Step 6.3: the degree of confidence of calculating pixel is also obtained parallax
The acceleration of pressing Fig. 6 is new system more, gives next pixel with the message propagation of each pixel successively.Through after T the iteration of entire image, the degree of confidence b of pixel p
p(d) by
Calculate, (p is the coupling cost of pixel p d) to Φ, is obtained by step 5;
Neighbor s passes to the degree of confidence message of pixel p for this moment.Then calculate all parallax d ∈ [d in the disparity range
Mind
Max] corresponding degree of confidence, d
MinBe minimum parallax value, d
MaxBe the maximum disparity value, the parallax value d of pixel p
pRequire to select the maximum a posteriori estimator, with the joint probability of maximization Markov random field, by
Calculate.
Step 7: parallax is accurate
The integer parallax value that the parallax of estimating based on sub-pix accurately can obtain step 6 is converted into more accurate decimal, reduces the parallax quantization error of three-dimensional coupling, has protected the three-dimensional information of object.After the step 6, three of each pixel p discrete parallax value d in the benchmark image
p, d
p+ 1, d
p-1 can obtain immediately.Estimate by sub-pix
Can obtain the sub-pix parallax Dispartiy (p) of decimal form, as the final parallax value of benchmark pixel p.Wherein, three discrete parallax value d
p, d
p+ 1, d
p-1 obtains Φ (p, d by step 6
p), Φ (p, d
p+ 1) and Φ (p, d
p-1) the coupling cost of the pixel p that obtains for step 5.
Step 8: three-dimensional information reduction
According to the pixel parallax Dispartiy (p) that step 7 obtains, the coupling in benchmark image and the registering images is to (P
L, P
R) can pass through x
R=x
L+ Disparity (P
L), y
R=y
LObtain P
LCoordinate be (x
L, y
L), P
RCoordinate be (x
R, y
R).Follow the camera intrinsic parameter matrix A that obtains according to step 1
L, A
RWith outer parameter matrix [R
Lt
L], [R
Rt
R], and pixel matching is to (P
L, P
R), just can calculate the three-dimensional coordinate of impact point by the space method that crosses.Fig. 7 is the cross synoptic diagram of method of space, O
L, O
RBe respectively the photocentre of two video cameras, S
L, S
RBe respectively the imaging plane of two video cameras, P
L, P
RIt is a pair of match point in two shot by camera images.Following relation of plane is arranged between the pixel coordinate on object point three-dimensional coordinate and the imaging plane in the space:
Wherein, (u, the v) coordinate of representation space object point imaging element on imaging plane, (X
w, Y
w, Z
w) volume coordinate of expression impact point P, Z
1, Z
2Be respectively the optical axis of two video cameras.Impact point P respectively with video camera photocentre O
L, O
RThe straight line O that is become
LP, O
RThe intersection point of P on imaging plane is exactly its projection P at the plane of delineation
L(u
1, v), P
R(u
2, v).Can obtain straight line O according to formula (2)
LP
L, O
RP
REquation, its intersection point is exactly the world coordinates that P is ordered.In the actual computation process, straight line O
LP
LAnd O
RP
RMay not can just in time intersect, need choose the common vertical line mid point of different surface beeline among Fig. 7 as impact point this moment.