GB2380887A - Lens distortion correction using correspondence points within images which are constrained to lie on the same epipolar curve - Google Patents

Lens distortion correction using correspondence points within images which are constrained to lie on the same epipolar curve Download PDF

Info

Publication number
GB2380887A
GB2380887A GB0124608A GB0124608A GB2380887A GB 2380887 A GB2380887 A GB 2380887A GB 0124608 A GB0124608 A GB 0124608A GB 0124608 A GB0124608 A GB 0124608A GB 2380887 A GB2380887 A GB 2380887A
Authority
GB
United Kingdom
Prior art keywords
lens distortion
distortion parameter
images
epipolar
parameter value
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
GB0124608A
Other versions
GB0124608D0 (en
Inventor
Andrew Fitzgibbon
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Oxford University Innovation Ltd
Original Assignee
Oxford University Innovation Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oxford University Innovation Ltd filed Critical Oxford University Innovation Ltd
Priority to GB0124608A priority Critical patent/GB2380887A/en
Publication of GB0124608D0 publication Critical patent/GB0124608D0/en
Priority to PCT/GB2002/004517 priority patent/WO2003034750A1/en
Publication of GB2380887A publication Critical patent/GB2380887A/en
Withdrawn legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/80Geometric correction
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/50Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/50Depth or shape recovery
    • G06T7/55Depth or shape recovery from multiple images
    • G06T7/579Depth or shape recovery from multiple images from motion
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/80Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/10Image acquisition
    • G06V10/12Details of acquisition arrangements; Constructional details thereof
    • G06V10/14Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/147Details of sensors, e.g. sensor lenses
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/98Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N17/00Diagnosis, testing or measuring for television systems or their details

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Quality & Reliability (AREA)
  • Vascular Medicine (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biomedical Technology (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)

Abstract

Image data processing apparatus, including means for computing and accounting for lens distortion given two or more images of a scene and no other information. In particular, the image data processing apparatus of the invention produces one or more estimated lens distortion parameters for insertion in an epipolar constraint equation, the apparatus being arranged to adjust point correspondences between two or more captured images such that they lie on substantially the same epipolar curve.

Description

<Desc/Clms Page number 1>
IMAGE CAPTURE APPARATUS This invention relates generally to image capture apparatus and, more particularly, to image capture apparatus including means for estimating lens distortion parameters from two or more images captured by a camera.
There are many circumstances, for example within the film industry, in which it may be required to reconstruct a three-dimensional scene which is observed in one or more captured images. For instance, it may be required to combine a three-dimensional video sequence of a scene captured within the real world with one or more three dimensional virtual buildings or other objects incorporated therein.
The lens of an image capturing device is generally curved and, as such, an image captured therewith will exhibit a certain amount of"curvature"caused by lens distortion, which curvature does not substantially adversely affect the quality of the captured image itself, but causes the image not to obey the laws of perspective projection, and therefore needs to be accounted for. In order to achieve this, it is known in the art to calibrate the camera capturing the images of the scene so as to apply one or more lens distortion terms to the captured images and thus substantially eliminate, or at least reduce, the above-mentioned curvature therefrom. The result of this (if viewed in terms of a single frame) is the production of an essentially"perspective"image of the same scene in which the comers of the original image have been effectively"pushed out"or"pushed in", as illustrated in Figure I of the drawings.
The amount of curvature created in an image because of lens distortion may typically be
represented by a single value, and the addition of such a single distortion term within the tz > image processing function is known to significantly improve the results of scene reconstruction, particularly over long video sequences.
It should be noted that the largest body of work on the correction of lens distortion deals with camera precalibration, i. e. where the camera is calibrated offline (before the image
<Desc/Clms Page number 2>
sequence is captured therewith). However, the present invention is concerned primarily with the situation where the original camera lens is not available, for example in the case of archive footage, or when using variable lens geometries. Thus, the present invention is concerned with the problem of nonlinear lens distortion in the context of camera selfcalibration and structure from motion. In particular, the recovery of three-dimensional camera motion from two-dimensional point tracks, where there is moderate to severe lens distortion.
Known techniques concerned with online estimation of lens distortion can be divided into two strategies. The first, known as the plumb line method uses straight lines in a scene to provide constraints on the distortion parameters. However, straight lines are not always available in a scene, and when present are not necessarily trivial to detect. As a result, extreme care must often be taken to ensure that real-world curves are not confused with distorted lines.
It is known in the art that lens distortion can be computed given two or more images of the same scene, captured from two or more different respective angles, and no other information, and the second known method is known as bundle adjustment which involves the computation of the fundamental matrix of a pair of cameras. An example of this
method is given in Proceedings International Conference on Pattern Recognition (proc ICPR), 1996"On the Epipolar Geometry Between Two Images With Lens Distortion", Z.
Zhang, in which lens distortion is considered as an integral part of the camera and the rigidity constraints or assumptions required to compute the fundamental matrix are extended to include the parameters of the distortion model. The epipolar geometry between two images with lens distortion is described, and it is established that for a point in one image, its corresponding point in the other image should lie on the so-called epipolar curve. The paper then goes on to investigate the possibility of estimating the distortion parameters and the fundamental matrix based on this generalised epipolar constraint. However, experimental results with computer simulation have shown that the distortion parameters can only be estimated correctly using the disclosed techniques if the
<Desc/Clms Page number 3>
noise in image points is low and the lens distortion is severe. Otherwise it is considered to be better to treat the camera (s) as being free of distortion.
Further, the above-described technique relies on iterative methods to find the distortion parameters. As is usual with such iterative methods, their convergence is not guaranteed, initial estimates must be found, and-although fast within this class of nonlinear techniques - they remain too slow to place in the inner loop of any hypothesise-and-test architecture.
Thus, some known systems are only able to estimate lens distortion parameters satisfactorily in the extreme conditions whereby noise in image points is low and lens distortion is severe, otherwise lens distortion is ignored. However, in many cases, ignoring the lens distortion parameters produces unsatisfactory results and, if accurate camera information is required there is considered to be no recourse but to bundle adjustment (also described in the chapter'Bundle Adjustment: A Modem Synthesis', by W. Triggs, P.
McLauchlan, R. Hartley and A. Fitzgibbon in Vision Algorithms : Theory and Practice, LNCS, Springer Verlag, 2000), initialised with reasonable estimates of camera geometry in the presence of lens distortion which have until now been necessarily partially initially estimated manually by an expert before being inserted into one or more iterative algorithms for a more accurate calculation of these parameters. As a result, currently-known bundle adjustment techniques cannot be used to compute lens distortion parameters simultaneously with unknown camera motion and unknown scene geometry, and must instead be set by automated guesswork, which is slow, requires expert manual input and frequently fails altogether, and we have now devised an arrangement which overcomes the problems outlined above.
In accordance with the present invention, there is provided image data processing apparatus, comprising means for receiving two or more images of a scene captured by one or more image capturing devices, means for identifying point correspondences between said captured images, means for determining a lens distortion parameter value corresponding to lens distortion exhibited in said captured images and adjusting said point correspondences according to said determined lens distortion parameter value such that
<Desc/Clms Page number 4>
said point correspondences lie on substantially the same epipolar curve, said lens distortion parameter value being determined so as to satisfy an epipolar constraint equation defining said epipolar curve, the lens distortion parameter comprising an element of said epipolar constraint equation, the apparatus further comprising means for computing one or more suitable estimated lens distortion parameter values for insertion in said epipolar constraint equation.
In a preferred embodiment of the invention, the means for computing one or more suitable estimated lens distortion parameter values is arranged to compute said values using one or more design matrices, said one or more design matrices preferably being derived from twodimensional point coordinates, beneficially obtained directly from the captured images.
Each point correspondence obtained directly from the captured images comprises a pair of image points (x,, ) and (x,', y,'), where i ranges from 1 to the number of point correspondences, m. Define r, : = X, 2 + y, 2, and r,' : = X,'2 + y,'2.
In one preferred embodiment, from each such point correspondence, three design matrix rows may be defined as follows:
These rows are assembled into m x 9 design matrices Db D2, D3 (i. e. D (1 a E {1, 2, 3}) as follows:
<Desc/Clms Page number 5>
The three design matrices are inserted into a quadratic eigenvalue problem of the form : (D, +D, + À2D3) f= 0 where À denotes the lens distortion parameter and f denotes a 9-vector containing the elements of the fundamental matrix. It is known how to solve such problems. In a second embodiment, from each such point correspondence, three pairs of design matrix rows may be defined as follows:
These pairs of rows are assembled into three design matrices Db D2, D3 each of size 2m x 9, as above. The three design matrices are inserted into a quadratic eigenvalue problem of the form :
where # denotes the lens distortion parameter and h denotes a 9-vector containing the elements of the planar homography.
<Desc/Clms Page number 6>
An embodiment of the present invention will now be described by way of example only and with reference to the accompanying drawings, in which: Figure 1 is a schematic diagram illustrating the effect of applying lens distortion correction to a captured image, it will be shown that distortion can be computed given two or more images of the scene and no other information.
The following description of an exemplary embodiment of the present invention relates primarily to allowing the matching of image pairs via interest-point correspondences. As explained above, the most successful prior art techniques for matching interest points are based on the geometric constraints offered by multiple-view geometry. These tend to be effective because fast linear algorithms exist for the computation of the relationships allowing their computation to form the kernel of RANSAC-based matching algorithms.
However, when images exhibit strong lens distortion, these constraints cannot be applied because the two-view relationships (fundamental matrix, planar homography) are not accurate in the image periphery.
Thus, an object of the present invention is to develop a model for the between-view relations which incorporates lens distortion. In particular, a model is required which admits a direct solution, i. e. computation from point correspondences via well understood, fast, globally convergent, numerical algorithms such as single value decomposition (SVD) or eigenvalue extraction, and the present invention is concerned with the calculation of one or more realistic estimates of camera geometry and correspondences for use in bundle adjustment techniques generally.
In the following, 2D points (in non-homogeneous coordinates) will be denoted by X = (x, y) and x will denote a general vector, including 2D points in homogeneous coordinates. The data used in the algorithm employed in this exemplary embodiment of the invention comprises point correspondences between lens distorted images. As the following deals almost entirely with two-view geometry, primes will be used to indicate a corresponding
<Desc/Clms Page number 7>
point in the second view. Thus, as input we have a set of two-view point correspondences, denoted x x\ The image points observed will be distorted functions of some perspective pinhole points,
which shall be denoted by p I which shal) be denoted by p, so the image point x is the distorted version of the perfect point p. The present invention is only concerned with radial distortion, so that the relationship between x and p is dependent on their distances from the image centre.
Throughout the paper, all these points are expressed in a 2D coordinate system, with origin at the distortion centre. In the absence of any other information, one would fix the distortion centre at the centre of the image, which is considered to be a reasonable approximation.
Given that the distortion centre may be assumed known, it is known in the art how to write the distortion correction (e. g. C. Slama,"Manual ofPhotogrammetry, 4* Edition", American Society of Photogrammetry, Falls Church, VA, USA, 1980) in one of several ways.
The distortion model used in this description of an exemplary embodiment of the present invention is:
and:
In order to compute the fundamental matrix from perfect point correspondences pop'an algorithm may be derived and used, which can be modified to include),, the distortion parameter, such that the fundamental matrix F may be computed from distorted, measurable points xx', as follows.
<Desc/Clms Page number 8>
A point correspondence in pinhole coordinates pop'which corresponds to a real 3D point
which has been imaged by a pair of cameras will satisfy the epipolar constraint. This is embodied in the fundamental matrix, F, for the pair of cameras : p Fp = 0 (3) It is the task of the apparatus of this exemplary embodiment of the present invention to recover F from point correspondences. Writing p = (p, q, 1), and concatenating the rows of F into a nine-vector f, we may rewrite the above constraint as:
Collecting eight such rows into a design matrix D, we obtain an estimate for fby solving Df= 0. This estimate will be greatly improved by truncating the resulting matrix to rank 2.
In order to compute F from the known image coordinates x, we must express equation (3) in terms ofx. Writing the distortion equation (1) projectively, we obtain:
or:
where both x and z are known (i. e. can be computed from image coordinates alone). Then the epipolar constraint is:
<Desc/Clms Page number 9>
or:
which is quadratic in À (or K) and linear in F. Indeed, expanding everything out, we obtain (with r = llxll, and r'= llx'll)
Gathering the three row vectors into three design matrices, we obtain the following quadratic eigenvalue problem (QEP):
Such problems are analogous to standard second order ordinary differential equations (ODE's) (replace k with partial derivative operators), and efficient numerical algorithms are already available in the art for their solution. Equation (4) has a maximum of 18 solutions which can be used as the distortion parameters in the bundle adjustment estimation described above, with one of said solutions providing the closest approximation of lens distortion. However, it has been found that there are at most only 10 solutions, and in practice probably no more than 6 solutions, that are real.
The preceding analysis applies also to the estimation of a plane projective transformation between the images. In this case, each point correspondence adds two rows to the design matrices:
<Desc/Clms Page number 10>
The analogous computation for the trifocal tensor leads to a cubic eigenvalue problem, which is again readily solved.
Thus, in summary, the present invention extends the uncalibrated estimation of geometry from multiple images to include a correction for lens distortion. Its main contribution is considered to be a linear algorithm for the simultaneous estimation of this model and the fundamental matrix. All conventional algorithms for this purpose are iterative.
The present invention enables images which exhibit lens distortion to be matched with the same ease as those which accurately fit the pinhole model. Furthermore, it is possible to use the distortion-aware model to match even low distortion images without overfitting.
A specific embodiment of the present invention has been described herein by way of example only and it will be apparent to a person skilled in the art that modifications and variations may be made to the described embodiment without departing from the scope of the invention.

Claims (6)

  1. CLAIMS: 1. Image data processing apparatus, comprising means for receiving two or more images of a scene captured by one or more image capturing devices, means for identifying point correspondences between said captured images, means for determining a lens distortion parameter value corresponding to lens distortion exhibited in said captured images and adjusting said point correspondences according to said determined lens distortion parameter value such that said point correspondences lie on substantially the same epipolar curve, said lens distortion parameter value being determined so as to satisfy an epipolar constraint equation defining said epipolar curve, the lens distortion parameter comprising an element of said epipolar constraint equation, the apparatus further comprising means for computing one or more suitable estimated lens distortion parameter values for insertion in said epipolar constraint equation.
  2. 2. Apparatus according to claim 1, wherein the lens distortion parameter value is computed using two-dimensional point correspondences or coordinates obtained directly from said captured images.
  3. 3. Apparatus according to claim 2, wherein said lens distortion parameter value is computed using one or more design matrices.
  4. 4. Apparatus according to any one of claim 3, wherein three or more design matrices are generated and inserted into a polynomial eigenvalue of the form:
    where k denotes the lens distortion parameter and f denotes a 9-vector containing the elements of the fundamental matrix.
    <Desc/Clms Page number 12>
  5. 5. Apparatus according to claim 3, wherein three or more design matrices are generated and inserted into a polynomial eigenvalue problem of the form:
    where À denotes the lens distortion parameter and h denotes a 9-vector containing the elements of the planar homography.
  6. 6. Image data processing apparatus substantially as herein described with reference to the accompanying drawings.
GB0124608A 2001-10-13 2001-10-13 Lens distortion correction using correspondence points within images which are constrained to lie on the same epipolar curve Withdrawn GB2380887A (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
GB0124608A GB2380887A (en) 2001-10-13 2001-10-13 Lens distortion correction using correspondence points within images which are constrained to lie on the same epipolar curve
PCT/GB2002/004517 WO2003034750A1 (en) 2001-10-13 2002-10-03 Image capture apparatus

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
GB0124608A GB2380887A (en) 2001-10-13 2001-10-13 Lens distortion correction using correspondence points within images which are constrained to lie on the same epipolar curve

Publications (2)

Publication Number Publication Date
GB0124608D0 GB0124608D0 (en) 2001-12-05
GB2380887A true GB2380887A (en) 2003-04-16

Family

ID=9923769

Family Applications (1)

Application Number Title Priority Date Filing Date
GB0124608A Withdrawn GB2380887A (en) 2001-10-13 2001-10-13 Lens distortion correction using correspondence points within images which are constrained to lie on the same epipolar curve

Country Status (2)

Country Link
GB (1) GB2380887A (en)
WO (1) WO2003034750A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1320813C (en) * 2003-06-20 2007-06-06 北京中星微电子有限公司 A distortion correction method for lens imaging
WO2013015699A1 (en) 2011-07-25 2013-01-31 Universidade De Coimbra Method and apparatus for automatic camera calibration using one or more images of a checkerboard pattern
US8681224B2 (en) 2007-06-26 2014-03-25 Dublin City University Method for high precision lens distortion calibration and removal

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1100048A1 (en) * 1999-11-12 2001-05-16 Société N3DI S.A.R.L. Automatic building process of a digital model using stereoscopic image couples

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07294215A (en) * 1994-04-25 1995-11-10 Canon Inc Method and apparatus for processing image
US6173087B1 (en) * 1996-11-13 2001-01-09 Sarnoff Corporation Multi-view image registration with application to mosaicing and lens distortion correction
JP3020898B2 (en) * 1997-07-22 2000-03-15 株式会社エイ・ティ・アール人間情報通信研究所 A linear estimation method of three-dimensional position by affine camera correction
US6101288A (en) * 1997-07-28 2000-08-08 Digital Equipment Corporation Method for recovering radial distortion parameters from a single camera image
US6137491A (en) * 1998-06-05 2000-10-24 Microsoft Corporation Method and apparatus for reconstructing geometry using geometrically constrained structure from motion with points on planes
US6597816B1 (en) * 1998-10-30 2003-07-22 Hewlett-Packard Development Company, L.P. Correcting distortion in an imaging system using parametric motion estimation

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1100048A1 (en) * 1999-11-12 2001-05-16 Société N3DI S.A.R.L. Automatic building process of a digital model using stereoscopic image couples

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Lens distortion calibration using point correspondences, Stein G.P. Proceedings of IEEE Computer Society Conf. on Computer Vision and Pattern Recogn(see pp 602-604), pp 602 - 608, ISBN 0-8186-7822-4 *
On the epipolar geometry between two images with lens distortion, Zhengyou ZhangProceedings of International Conf. on Pattern Recognition, August 1996, pp 407 - 411, ISBN 0-8186-7282-X *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1320813C (en) * 2003-06-20 2007-06-06 北京中星微电子有限公司 A distortion correction method for lens imaging
US8681224B2 (en) 2007-06-26 2014-03-25 Dublin City University Method for high precision lens distortion calibration and removal
WO2013015699A1 (en) 2011-07-25 2013-01-31 Universidade De Coimbra Method and apparatus for automatic camera calibration using one or more images of a checkerboard pattern

Also Published As

Publication number Publication date
WO2003034750A1 (en) 2003-04-24
GB0124608D0 (en) 2001-12-05

Similar Documents

Publication Publication Date Title
KR100695174B1 (en) Method and apparatus for tracking listener&#39;s head position for virtual acoustics
Kim et al. Dynamic video deblurring using a locally adaptive blur model
Chiuso et al. 3-D motion and structure from 2-D motion causally integrated over time: Implementation
KR101706216B1 (en) Apparatus and method for reconstructing dense three dimension image
US9361706B2 (en) Real-time optical flow sensor design and its application to obstacle detection
US10062151B2 (en) Image deblurring method and apparatus
US10126115B2 (en) Triangulation device, triangulation method, and recording medium recording program therefor
US10853960B2 (en) Stereo matching method and apparatus
US20090092282A1 (en) System and Method for Tracking Objects with a Synthetic Aperture
JP2010500677A (en) Image processing method
CN103530599A (en) Method and system for distinguishing real face and picture face
US10582179B2 (en) Method and apparatus for processing binocular disparity image
EP3166074A1 (en) Method of camera calibration for a multi-camera system and apparatus performing the same
WO2014002521A1 (en) Image processing device and image processing method
JP2016126673A (en) Information processing device, control method therefor, and video camera
Kim et al. Dynamic scene deblurring using a locally adaptive linear blur model
KR100691855B1 (en) Apparatus for extracting features from image information
JP3822482B2 (en) Face orientation calculation method and apparatus
US20170116739A1 (en) Apparatus and method for raw-cost calculation using adaptive window mask
KR20180014149A (en) Apparatus and method for generating depth information
GB2380887A (en) Lens distortion correction using correspondence points within images which are constrained to lie on the same epipolar curve
JP2000353244A (en) Method for obtaining basic matrix, method for restoring euclidean three-dimensional information and device therefor
JP5887974B2 (en) Similar image region search device, similar image region search method, and similar image region search program
TWI687086B (en) Temporally consistent belief propagation system
Anandan et al. Integrating local affine into global projective images in the joint image space

Legal Events

Date Code Title Description
WAP Application withdrawn, taken to be withdrawn or refused ** after publication under section 16(1)