Background technology
Multicamera system (comprising multiple-camera or multi-camera system) can be widely used in three-dimensional reconstruction, vision monitoring, action data during video display, animation and development of games are made is caught, sports analysis and training guidance, and the numerous areas such as gait analysis in the medical research, rehabilitation medical.Using the essential step of multicamera system is that system is demarcated, and promptly determines the projection matrix of each camera or further each projection matrix is decomposed into inside and outside parameter.
Traditional multicamera system scaling method generally is by three-dimensional calibrating block or two dimensional surface calibrating template (Zhang Z Y.Flexible camera calibration by viewing a plane fromunknown orientations, In:Proceedings of the ICCV ' 99, Kerkya, Greece, 1999.666-673) demarcate, because demarcating parts self existence, above-mentioned two classes block, can't calibrate all cameras of multicamera system simultaneously, can only demarcate adjacent two cameras successively, carry out under coordinate transform unification to a coordinate system at last again, thereby this method is very consuming time, and there is cumulative errors usually in the outer parameter of demarcating.Another kind of method is by static demarcating and two steps of dynamic calibration (Vicon Motion Systems to be demarcated by specific demarcation parts in system, VICON 512 manuals, pp.43-52,186-190), these class methods can improve the precision and the efficient of demarcation to a certain extent.This method was utilized by two monumented points (for the redundant information of utilizing data reduces The noise in the static demarcating stage, several monumented points also can be set more) one dimension that constitutes demarcates parts and demarcates, its essence is camera intrinsic parameter to be carried out certain hypothesis utilize self-calibration technology to demarcate, the metric that utilizes the demarcation parts to provide then rises to calibration result under the Euclidean meaning, but there is the shortcoming of the low poor robustness of well-known precision in self-calibration technology, and still there is the lower problem of efficient and precision in this method owing to need utilize two kinds of demarcation parts to demarcate step by step.
In recent years, development and application along with computer vision, people begin to explore new camera marking method, Zhang has proposed to utilize one dimension to demarcate component body video camera has been demarcated (ZhangZ Y.Camera calibration with one-dimensional objects.IEEE Trans.PatternAnalysis and Machine Intelligence, 26 (7): 892-899,2004).But the point that the method for Zhang needs one dimension to demarcate on the parts maintains static, and one dimension demarcates parts and can only rotate around this point of fixity, when specific implementation, is difficult to strict rotating around point of fixity on the one hand, and this can have a strong impact on the precision of demarcation; Operator's health can block some camera on the other hand, can't demarcate total system simultaneously.This has limited the practical application of this method.
Summary of the invention
For overcoming the deficiencies in the prior art, the purpose of this invention is to provide a kind of comparatively easy, practical, fast and have a scaling method of the multicamera system of degree of precision and robustness.
To achieve these goals, an aspect of of the present present invention provides a kind of scaling method of multicamera system, comprises that step is as follows:
Step 1: operation is done rigid motion by the demarcation parts that three of conllinear or three above monumented points constitute in the multicamera system perform region; Each camera that the demarcation parts are done once in the motion multicamera system is respectively gathered piece image, obtains corresponding to the set of diagrams picture of demarcating the current pose of parts; Operation is demarcated parts and is made the multiple series of images that repeatedly motion obtains different poses demarcation parts;
Step 2: the multiple series of images input computing machine of the demarcation parts that multicamera system is taken;
Step 3: each group in the multiple series of images is corresponding to a pose demarcating parts, for the some monumented points on the demarcation parts under a certain pose, each width of cloth from the set of diagrams picture of this pose correspondence extracts the picture point coordinate of this monumented point, and these picture point that will obtain are as set of diagrams picture point correspondence; Have a plurality of monumented points on the parts and demarcate the repeatedly motion of parts do owing to demarcate, then can obtain the picture point coordinate and the multiple series of images point correspondence of all monumented points from multiple series of images;
Step 4: picture point coordinate that step 3 is obtained and corresponding relation and the geological information of demarcating parts carry out the projection matrix that the linear calibration finds the solution multicamera system;
Step 5: the projection matrix result of the multicamera system of finding the solution as initial value, is carried out nonlinear optimization to initial value and obtains more accurate calibration result.
According to embodiments of the invention, described linear calibration has utilized the utmost point geometric relationship between the picture point of all monumented points simultaneously, according to demarcating the geological information that parts provide and the picture point of monumented point, according to the projection meaning, affine meaning and Euclidean meaning successively promote calibration result, and when calibration result is risen to affine meaning by the projection meaning, utilize projective transformation to keep the constant character of double ratio to determine the image coordinate of the infinity point of demarcation parts correspondence, and estimate infinity homographies between first camera and all the other each cameras thus, obtain the normal vector of plane at infinity under the projective coordinate system, and then utilize this normal vector to obtain calibration result under the affine meaning.
According to embodiments of the invention, described linear calibration comprises the steps:
1) determines projection matrix under the projection meaning:, utilize utmost point geometric knowledge in the computer vision to recover the projection matrix of multicamera system under the projection meaning according to the coordinate and the corresponding relation of picture point;
2) determine projection matrix under the affine meaning: utilize and demarcate geological information that parts provide and the picture point of demarcating parts, determine the normal vector of plane at infinity, obtain infinity homographies and promptly be the projective transformation one to one between two planes of delineation being determined by plane at infinity, homography matrix rises to affine meaning with the projection matrix under the projection meaning;
3) determine projection matrix under the Euclidean meaning: the range information of demarcating between the monumented point on the parts is known, utilize the projection matrix under the affine meaning and the image coordinate equationof structure group of monumented point according to this constraint, obtain the projection matrix of the multicamera system under the Euclidean meaning.
According to embodiments of the invention, described nonlinear optimization method comprises: with the projection matrix of each camera and the three-dimensional coordinate of monumented point is optimization variable, with the distance between the re-projection image coordinate of the monumented point of the monumented point image coordinate extracted in the image and estimation be cost function, be that initial value carries out nonlinear optimization and handles with linear calibration's result.
According to embodiments of the invention, described scaling method carries out disposable demarcation to multicamera system integral body.
To achieve these goals, another aspect of the present invention provides a kind of caliberating device of multicamera system, comprising:
Demarcate the perform region that parts are positioned at multicamera system, demarcate parts and fixedly connected with long handle, long handle is used to control the demarcation parts and moves in the perform region of multicamera system; Multicamera system is taken the image of motion demarcation parts in its perform region, is used for multicamera system is demarcated.
According to embodiments of the invention, the spherical more than three or three that contains conllinear on the described demarcation parts constitutes monumented point, is demarcating the distance that is provided with arbitrarily on the parts between the monumented point.
According to embodiments of the invention, adopt luminous, reflective or constitute with the bead of demarcating environmental colors remarkable distinct color being arranged according to the request for utilization spherical.
According to embodiments of the invention, there are right angle member or nut to fixedly connected at an end of handle with the demarcation parts, make and demarcate parts and handle form right angle.
According to embodiments of the invention, in calibration process, demarcate parts and in the perform region, do rigid motion.
Scaling method provided by the invention and device can carry out disposable demarcation to whole multicamera system, and do not need every camera is demarcated respectively or utilized a plurality of demarcation parts to demarcate step by step, having overcome being difficult to that existing scaling method exists realizes, introduces problem such as block, have characteristics such as comparatively easy, practical, that stated accuracy is high, robustness is good, the demarcation efficient and the precision of multicamera system are improved.
Embodiment
Below in conjunction with accompanying drawing the present invention is described in detail, be to be noted that described embodiment only is intended to be convenient to the understanding of the present invention, and it is not played any qualification effect.
Multicamera system requires video camera or camera from a plurality of angle shot objects usually.When object or person to be rebuild are positioned at the perform region of multicamera system, multicamera system can obtain waits to rebuild the image of object or person in the different viewing angles of synchronization, utilize the calibration result of multicamera system can calculate the three-dimensional information of waiting to rebuild object or person, realize targets such as motion-captured.Thereby to demarcate be most important step of fundamental sum of this system.Provided a kind of synoptic diagram of multicamera system caliberating device in Fig. 1,4 shown in the dotted line is the workspace of multicamera system, i.e. the public field of vision of all cameras.
Comprise among Fig. 1: demarcation parts 1, handle 2, operating system 3 or manually-operated, perform region 4, multicamera system 5 with three monumented points, demarcate the perform region 4 that parts 1 are positioned at multicamera system 5, demarcate parts 1 and fixedly connected with long handle 2, long handle 2 is used to control demarcation parts 1 and moves in the perform region 4 of multicamera system 5; Multicamera system 5 is taken the image of the demarcation parts 1 that move in its perform region, be used for multicamera system 5 is demarcated.
The scaling method that the present invention proposes mainly comprises and obtains image, the image input of demarcating parts 1, extracts monumented point image coordinate and structural map picture point correspondence from image, demarcate multicamera system linearly and utilize nonlinear algorithm to optimize step such as calibration result and form, shown in the process flow diagram of Fig. 2 scaling method of the present invention, specifying of each step is as follows:
Step 1. operation is demarcated parts 1 and do general rigid motion in perform region 4, obtains the image of demarcating parts 1:
(three monumented points of minimum needs that the demarcation parts 1 that apparatus of the present invention adopt are made of the monumented point more than three or three of conllinear, suppress to measure The noise for the redundant information of utilizing data and can get monumented point more than three), and the geological information between the monumented point is known, the present invention demarcates shown in the modular construction synoptic diagram as Fig. 3, and demarcate parts 1 and can utilize the right-angle coupler (being cruciform fastening for falsework again) or the nut of standard to install long handle additional as required, the present invention utilizes shown in the demarcation block diagram after right-angle coupler installs handle additional as Fig. 4.This makes operator 3 to operate outside perform region 4 and demarcates parts 1, is avoided operator 3 to block camera 5.Utilize luminous, reflective or with demarcate bead body that environmental colors has remarkable distinct color as the sign of demarcating on the parts 1, with the centre of sphere of bead point as a token of, demarcating the distance that is provided with arbitrarily on the parts between the monumented point.Operator 3 outside perform region 4 holds the long handle 2 control demarcation parts 1 that add of demarcating parts 1 and do general rigid motion in the perform region 4 of multicamera system 5, multicamera system 5 should synchronously be taken the multiple series of images of demarcating parts 1, here be meant the image of taking simultaneously by multicamera system 5 for some athletic postures of demarcating parts 1, not life period delay each other synchronously; Or exist certain hour to postpone when allowing the different cameral photographic images of multicamera system 5, but should be still under the current pose in photographic images markers limiting-members 1 always.
Step 2. input picture:
The image that multicamera system 5 is taken is imported computing machine by scanner or some special purpose interface;
Step 3. is extracted the picture point coordinate of monumented point from the image that multicamera system 5 is taken, structural map picture point correspondence:
Utilize camera to take ball from all angles, ball all is imaged as a circle in the image that obtains, and the picture point of the corresponding centre of sphere in the center of circle.Utilize this character with luminous, reflective or with demarcate bead that environmental colors has remarkable distinct color as the sign of demarcating parts 1, be the monumented point of demarcating parts 1 with the centre of sphere.By the manual picture point that as a token of put in the center of circle of extraction centre of sphere correspondence in image of interactive means; Also can carry out the zone with Gauss's template circle that imaging obtains to bead and detect (blob detection), and be the picture point of monumented point with the regional center by automatic mode.If multicamera system 5 has M camera, demarcating on the parts 1 has Q monumented point, demarcates parts 1 motion N time, and the picture point that then can obtain monumented point is:
X wherein
Mn qThe expression one dimension is demarcated parts 1 at the n time its q monumented point X of motion back
n qThe picture point that under m camera, becomes.Certain motion back is demarcated the picture point that some monumented points are become for 5 times in different cameral on the parts 1 and is constituted set of diagrams picture point correspondence, then can obtain NQ picture group picture point correspondence:
Step 4. linear calibration system
Demarcate to obtain the projective transformation matrix of each camera in the multicamera system 5 in fact exactly or further utilize QR to decompose scheduling algorithm projective transformation matrix is decomposed into inside and outside parameter.Multicamera system 5 is demarcated need be by demarcating parts 1, and only need to install or the systematic parameter of multicamera system 5 is once finished change the time in that multicamera system 5 is initial.The scaling method that the present invention proposes at first utilizes linear method that multicamera system 5 is demarcated, and utilizes nonlinear optimization further to improve stated accuracy again.Linear calibration's method has been utilized the utmost point geometric relationship between the picture point of all monumented points simultaneously, according to demarcating the geological information that parts 1 provide and the picture point of monumented point, according to the projection meaning, affine meaning and Euclidean meaning successively promote calibration result, and when calibration result is risen to affine meaning by the projection meaning, utilize projective transformation to keep the constant character of double ratio to determine the image coordinate of the infinity point of demarcation parts correspondence, and estimate infinity homographies between first camera and all the other each cameras thus, obtain the normal vector of plane at infinity under the projective coordinate system, and then utilize this normal vector to obtain calibration result under the affine meaning.
Linear calibration's process may further comprise the steps:
1) determines projection matrix under the projection meaning.Considering all images point that step 3 obtains simultaneously, is 2,3 and greater than 3 three kinds of situations according to the camera number M of multicamera system 5, calculates fundamental matrix respectively, the three warmers tensor sum is measured matrix, can obtain homography matrix H easily thus
mWith limit e
m, then the projection matrix under the projection meaning is:
P
1=[I|0], P
2=[H
2| e
2] ..., P
M=[H
M| e
M], wherein I is 3 rank unit matrix;
2) determine projection matrix under the affine meaning.Suppose that monumented point number Q gets 3, the distance between adjacent two of three monumented points can be designated as d successively by demarcating geological information that
parts 1 provide
1, d
2, remember that the infinity point of the n time motion back demarcation parts is V
N ∞, the double ratio that constitutes of three monumented points and infinity point then
Because projective transformation keeps double ratio constant, so m (m=1,2 ..., M) double ratio that the picture point of the picture point of three monumented points and infinity point constitutes in the image that obtains of individual camera is
Therefore can calculate the picture point v of the infinity point of demarcating the parts correspondence according to the picture point of monumented point
MnInfinity point all is positioned on the plane at infinity, thereby m (m=2,3,, M) have a homography matrix H who induces by plane at infinity between the infinity point picture point under the infinity point picture point under the individual camera and the 1st camera (here with the 1st camera as benchmark)
M ∞=H
m-e
ma
T(m=2,3 ..., M, wherein vectorial a are the normal vector of plane at infinity under the projective coordinate system), i.e. v
Ln=(H
m-e
ma
T) v
MnUtilize the correspondence between the picture point of infinity point
According to following formula equationof structure group, solve a.Thereby obtain infinity homographies H
M ∞=H
m-e
ma
T, (m=2,3 ..., M).
Utilize H
M ∞Projection matrix under the projection meaning can be risen under the affine meaning:
3) determine projection matrix under the Euclidean meaning.The range information of demarcating between the monumented point on the
parts 1 is known, utilize the projection matrix under the affine meaning and the image coordinate equationof structure group of monumented point according to this constraint, find the solution this and organize the intrinsic parameter K that equation of constraint obtains the 1st camera (benchmark camera), then the projection matrix under the Euclidean meaning is:
Wherein diag (K, 1) expression diagonal element is K, 14 * 4 square formations.Utilize the projection matrix P of image coordinate and estimation
m (e)Can estimate the Euclidean coordinate X of monumented point in three dimensions
n q
Step 5. optimization system calibration result
Multicamera system 5 calibration results that obtain with step 4 are initial value, utilize nonlinear optimization algorithm to find the solution calibration result is optimized, and ask the calibration result under the optimum meaning.
To demarcate parts 1 n (n=1,2 ..., N) the three-dimensional coordinate X of monumented point after the inferior motion
n qCalibrating parameters (the projection matrix P of all cameras in the system with multicamera system 5
m (e)M=1,2, M) as optimization variable, the projection matrix of utilize estimating carries out the re-projection that image transformation obtains monumented point to the three-dimensional symbol point of estimating, with the distance between the monumented point image coordinate of extracting in the monumented point image coordinate of re-projection and the image with as the cost function of optimization, concrete cost function is as follows:
X wherein
n qFor the n time motion back one dimension estimating demarcated the Euclidean coordinate of q monumented point in three dimensions on the parts 1.This is one does not have the constraint nonlinear optimal problem, and multicamera system 5 calibration results that obtain with step 4 are initial value.Above-mentioned nonlinear optimal problem recommends to use row civilian Burger-Marquardt algorithms (the Levenberg-Marquardt algorithm abbreviates the LM algorithm as) to find the solution, and this algorithm is a prior art, is not described in detail in this.
Above-mentioned steps by method described in the present embodiment, can realize the disposable demarcation of all cameras of multicamera system 5, utilize calibration result that the object or person in multicamera system 5 workspaces are carried out three-dimensional reconstruction, reconstruction accuracy can reach submillimeter level in short range.
The above; only be the embodiment among the present invention; but protection scope of the present invention is not limited thereto; anyly be familiar with the people of this technology in the disclosed technical scope of the present invention; can understand conversion or the replacement expected; all should be encompassed in of the present invention comprising within the scope, therefore, protection scope of the present invention should be as the criterion with the protection domain of claims.