WO1996034365A9 - Apparatus and method for recreating and manipulating a 3d object based on a 2d projection thereof - Google Patents

Apparatus and method for recreating and manipulating a 3d object based on a 2d projection thereof

Info

Publication number
WO1996034365A9
WO1996034365A9 PCT/US1996/005697 US9605697W WO9634365A9 WO 1996034365 A9 WO1996034365 A9 WO 1996034365A9 US 9605697 W US9605697 W US 9605697W WO 9634365 A9 WO9634365 A9 WO 9634365A9
Authority
WO
WIPO (PCT)
Prior art keywords
views
trilinear
scene
generating
tensor
Prior art date
Application number
PCT/US1996/005697
Other languages
French (fr)
Other versions
WO1996034365A1 (en
Filing date
Publication date
Priority claimed from IL11349695A external-priority patent/IL113496A/en
Application filed filed Critical
Priority to EP96913833A priority Critical patent/EP0832471A4/en
Priority to JP8532665A priority patent/JPH11504452A/en
Priority to AU56674/96A priority patent/AU5667496A/en
Publication of WO1996034365A1 publication Critical patent/WO1996034365A1/en
Publication of WO1996034365A9 publication Critical patent/WO1996034365A9/en

Links

Definitions

  • the present invention relates to apparatus and methods for processing 2D projections of 3D objects and particularly to apparatus and methods for geometric analysis of a 2D projection image.
  • the present invention seeks to provide image transfer apparatus and methods which are useful for generating a novel view of a 3D scene from first and second reference views thereof.
  • the present invention also seeks to provide 3D scene reconstruction methods and apparatus for generating a 3D representation of a 3D scene from first, second and third views thereof.
  • the present invention also seeks to provide improved apparatus and methods for processing 2D projections of 3D ob ⁇ jects.
  • the present invention also seeks to provide methods for reconstruction of a 3D object based on a trilinear tensor defined on three views of the 3D object.
  • the present invention additionally seeks to provide methods for image transfer for a 3D object based on a trilinear tensor defined on three views of the 3D object.
  • the present invention also seeks to provide an image transfer method for generating a novel view of a 3D scene from first and second reference views thereof, the method including providing first and second reference views of a 3D scene, employ ⁇ ing geometric information regarding the first reference view, second reference view and novel view, respectively, to generate a trilinear tensor representing the geometric relationship between the first, second and novel views and generating the novel view by computing a multiplicity of novel view locations each corre ⁇ sponding to different first and second corresponding locations in the first and second reference views respectively based on the first and second corresponding locations and the trilinear ten ⁇ sor.
  • the step of providing may, for example, comprise scan ⁇ ning in the first and second reference images.
  • the step of employing may, for example, include the step of generating a set of first, second and third corresponding locations in the first reference view, second reference view and novel view, respectively.
  • an image transfer method for generating a novel view of a 3D scene from first and second reference views thereof including providing first and second reference views of a 3D scene, employing geometric infor ⁇ mation regarding the first reference view, second reference view and novel view, respectively, to generate a trilinear tensor representing the geometric relationship between the first, second and novel views, and generating the novel view by computing a multiplicity of novel view locations each corresponding to dif ⁇ ferent first and second corresponding locations in the first and second reference views respectively based on the first and second corresponding locations and the trilinear tensor.
  • the step of providing includes scanning in the first and second reference images.
  • the step of employing includes the step of generating a set of first, second and third corresponding locations in the first reference view, second reference view and novel view, respectively.
  • a 3D scene reconstruction method for generating a 3D representation of a 3D scene from first, second and third views thereof, the method including providing first, second and third views of a 3D scene, employing geometric information regarding the first, second and third views to generate a trilinear tensor representing the geometric relationship between the first, second and third views, and generating a 3D representation of the 3D scene from the trilinear tensor.
  • the step of generating a 3D representation includes computing an epipolar geometric representation of the first and second views from the trilinear tensor, and generating the 3D representation from the epipolar geometric representation.
  • image transfer apparatus for generating a novel view of a 3D scene from first and second reference views thereof, the apparatus including apparatus for providing first and second reference views of a 3D scene, a trilinear tensor generator operative to employ geometric informa ⁇ tion regarding the first reference view, second reference view and novel view, respectively, to generate a trilinear tensor representing the geometric relationship between the first, second and novel views, and a novel view generator operative to generate the novel view by computing a multiplicity of novel view loca ⁇ tions each corresponding to different first and second corre ⁇ sponding locations in the first and second reference views re ⁇ spectively based on the first and second corresponding locations and the trilinear tensor.
  • 3D scene reconstruction apparatus for generating a 3D representation of a 3D scene from first, second and third views thereof, the apparatus including appparatus for providing first, second and third views of a 3D scene, a trilinear tensor generator operative to employ geometric information regarding the first, second and third views to generate a trilinear tensor representing the geometric relation ⁇ ship between the first, second and third views, and a 3D scene representation generator operative to generate a 3D representa ⁇ tion of the 3D scene from the trilinear tensor.
  • a visual recognition method including providing three perspective views of a 3D object be ⁇ tween which a trilinear relationships exists, and employing the trilinear relationship between the views in order to perform visual recognition by alignment.
  • the method also includes reprojecting the 3D object.
  • the information regarding the 3 D object includes a reconstruction of the 3D object.
  • the information regarding the 3D object includes at least one new view of the 3D object generated without reconstructing the 3D object.
  • the at least one and preferably three 2D projections includes at least one aerial photograph.
  • the at least one and preferably three 2D projections includes at least one satellite photograph.
  • the information regarding the 3D object comprises at least one coordinate of the 3D object.
  • the 3D object includes an aerospace ob ⁇ ject.
  • the 3D object includes a large object such as a ship.
  • the 3D object includes a nonexistent object.
  • V j _' and v ⁇ " are elements of vectors v' and v" respectively, wherein the matrices and vectors together describe camera parameters of three views of the 3D object, and an array analyzer employing the array to generate information regarding the 3D object.
  • apparatus for reconstructing a 3D object from at least one and preferably three 2D projections thereof including apparatus for providing at least one and preferably three 2D projections of a 3D object, an array generator operative to generate an array of numbers described by:
  • visual recognition appa ⁇ ratus including apparatus for providing three perspective views of a 3D object between which a trilinear relationships exists, and apparatus for employing the trilinear relationship between the views in order to perform visual recognition by alignment.
  • At least one result of performing the above method is employed in order to perform at least one of th follow i ng applications: map making from aerial and satem Photographs and coordinate measurements in aerospace and shipyard assembly plants, coordinate measurements of industrial parts (CMM) , automated optical based inspection of industrial parts robot i c cell alignment, robotic trajectory identification 3D robot i c feedback, 3D modelling of scenes, 3D modelling of' ob- Dects, reverse engineering, and 3D digitizing.
  • CCMM coordinate measurements of industrial parts
  • Fig. 1 is an illustration of two graphs comparing the performance of an epipolar intersection method, shown in dotted line, with the performance of a trilinear functions method, shown in dashed line, in the presence of image noise;
  • Fig. 2 is a pictorial illustration of two model views of a three-dimensional scene as well as a third reprojected view thereof;
  • Fig. 3 is a pictorial illustration of a reprojection using a trilinear result
  • Fig. 4 is a pictorial illustration of a reprojection using intersection of epipolar lines
  • Fig. 5 is a pictorial illustration of a reprojection using a linear combination of views method
  • Fig. 6 is a simplified functional block diagram of 3D scene reconstruction apparatus, constructed and operative in accordance with a preferred embodiment of the present invention, which is operative to generate a 3D representation of a 3D scene from at least three views thereof;
  • Fig. 7 is a simplified flowchart illustration of a preferred 3D scene reconstruction method, operative in accordance with a preferred embodiment of the present invention, which is useful in conjunction with the apparatus of Fig. 6;
  • Fig. 8 is a simplified functional block diagram of image transfer apparatus, constructed and operative in accordance with a preferred embodiment of the present invention, which is operative to generate a novel view of a 3D scene from at least two reference views thereof;
  • Fig. 9 is a simplified flowchart illustration of a preferred image transfer method, operative in accordance with a preferred embodiment of the present invention, which is useful in
  • Fig. 10 is an illustration of a 3D reconstruction of a shoe which is also illustrated in Fig. 2;
  • Fig. 11 is a simplified block diagram illustration of a preferred method and apparatus for quality assurance o f workpieces.
  • Fig. 12 is a simplified block diagram illustration of a preferred method and apparatus for generating a digital terrain map.
  • Appendix A which is a listing, in » c» language, of a preferred implementation of trilinear computation unit unit 5 0 and epipolar geometry generation unit 60 of Fig. 6;
  • Appendix B which is a listing of a preferred software i mplementation of 3D reconstruction apparatus constructed and operative in accordance with a preferred embodiment of the present invention.
  • a pin-hole camera like 35mm still camera or Video recorder, produces a two-dimensional projection (2D) of the viewed three-dimensional (3D) world.
  • the resulting image can be analyzed on a geometric and photometric level.
  • the geometric level means the geometric relation between the locations of features (points, lines) in 3D and their respective location in the 2D image.
  • the photometric level means the radiometric (reflectance properties of the surface, the spectral properties of the light sources illuminating the scene, etc.) relation between the scene and the luminosity (pixel grey values) in the image.
  • the 3D world is modeled as a cloud of points and so is the 2D world.
  • the camera moves in 3D we get more images of the same 3D world (i.e., of the same cloud of 3D points) and the question of interest is the geometrical relation between the corresponding set of 2D image points and the set of 3D points.
  • P denotes a set of 3D points
  • Pi, p' ⁇ ,p" are three sets of 2D points (across three images) arranged such that points with same index _ correspond to the same 3D point P;, then the image sets alone reveal much information about the 3D set.
  • 3D-from-2D is an active area in Computer Vision (structure-from- motion and stereopsis), and is the sole engine in the industry of photogrammetry (map making from areal and satellite photographs, coordinate measurements in aerospace and shipyard assembly plants, etc.).
  • 3D points and 2D points can be represented as a 3 x 4 transfor ⁇ mation matrix:
  • the sign denotes equality up to scale.
  • P (x, y, l, k) r
  • p' (x', y', l) ⁇
  • p" ⁇ x", y", l) ⁇ .
  • a and B are some 3 x 3 matrices (not independent of each other) and v', v" are some 3-vectors together describing the camera parameters of the three views.
  • These numbers are invariant to the particular projective representation of the 3D and 2D worlds, i.e., they are intrinsic to the three views (this is one of the general properties of tensors that they do not depend on the choice of basis for representation).
  • a corresponding triplet p, p', p" satisfies a number of trilinear relationships.
  • First notation We identify vectors and matrices by fixing some of the indices while varying others.
  • ,j_ is a set of scalars
  • a, j . is a set of 9 vectors (k varies while i, j remain fixed):
  • ⁇ ... is a set of 3 matrices (c-i.., 2 .. and 3 ..), and so forth.
  • the rank-4-constraint implies that the four largest principle components of the concate ⁇ nation matrix represents the geometry between views ⁇ _ and ⁇ 2 , 'n a way that is statically optimal.
  • a tensor of three views, or the four largest principle components of the concatenation matrix of m > 3 views can be used to recover the epipolar geometry between two views, and from there to be used for reconstructing a 3D model of the scene.
  • the epipolar point v' can be recovered from ..., A _, the four largest principle components concatenation matrix, as follows, let ⁇ !, ⁇ 2 , ⁇ 3 be the the columns of matrix A ⁇ , and by b ⁇ , b , b 3 the columns of _4 2 , then
  • ⁇ . x b j — 6, x _ is an epipolar line for ⁇ ⁇ j
  • ⁇ , x b provides another set of three epipolar lines.
  • the projective model of the 3D scene follows from F and v', for example: the 3 x 4 matrix [[v']F, v' ⁇ is a camera transformation from 3D coordinates to the image coordinates of the second view.
  • the 3 x 4 matrix [[v']F, v' ⁇ is a camera transformation from 3D coordinates to the image coordinates of the second view.
  • One of the important steps in manipulating 3D scenes from 2D imagery is the step of obtaining a reliable set (as dense as possible) of matching points across the views at hand.
  • the current state-of-the-art use correlation techniques across the image intensities of two views together with geometric information, such as the epipolar geometry.
  • the tensor With the tensor one can extend the correspondence method to take advantage of three views together and without assuming camera calibration. Given an initial set of corre ⁇ sponding points across three views one recovers the trilinear tensor. Then, the trilinear tensor provides a constraint for using correlation methods to accurately locate the matching points. This can be done as follows.
  • E! , E 2 , E 3 denote the arrangement a.,, of the tensor and W_, W , W 3 denote the ar ⁇ rangement a,., of the tensor.
  • the epipolar lines can be derived from the tensor as presented herein (Section II. B in detailed description of preferred embodiment).
  • w a, ⁇ are quantities that change with the location
  • FIG. 6 is a simplified functional block diagram of 3D scene reconstruction apparatus, constructed and operative in accordance with a preferred embodi ⁇ ment of the present invention, which is operative to generate a 3D representation of a 3D scene from at least three views there ⁇ of.
  • the apparatus of Fig. 6 includes apparatus for provid ⁇ ing at least 3 digital images of a 3D scene or object from at least 3 respective viewpoints, such as a CCD camera 10 which operates from 3 different viewpoints, or such as a film camera 20, which may be airborne, associated with a scanner 30 (such as a Zeiss PS-1 which is operative to digitize the images generated by film camera 20.
  • a CCD camera 10 which operates from 3 different viewpoints, or such as a film camera 20, which may be airborne
  • a scanner 30 such as a Zeiss PS-1 which is operative to digitize the images generated by film camera 20.
  • the at least 3 digital views are fed to a matching point finder 40 which is operative to identify at least 7 and preferably a multiplicity of triplets of matching points from the 3 digital views.
  • "Matching points" from different views are points or locations which correspond to a single location in the real, 3D world. These points may, for example, be identified manually.
  • commercially available matching point software may be employed such as the Match-T package mar ⁇ keted by Inpho in Stuttgart, which performs a correspondence function.
  • a trilinear tensor computation unit 50 receives the matching point triplets and computes a trilinear tensor representing the geometric relationship between the three views.
  • the trilinear tensor is then employed to generate a 3D representation of a 3D scene.
  • the trilinear tensor is employed by an epipo ⁇ lar geometry generation unit 60 to compute an epipolar geometric representation of two of the three views.
  • 3D representation generation unit 70 generates the 3D representation of the scene or object from the epipolar geometric representation output of unit 60, as described in more detail in: a. Faugeras, 0. D. "What can be seen in 3 dimensions with an uncalibrated stereo rig?", Proceedings of the European Conference on Computer Vision, pages 563 - 578, Santa Margherita Ligure, Italy, June 1992. b. Hartley, R.
  • a preferred implementation of units 50 and 60 is de ⁇ scribed, in "C" computer language form, in Appendix A.
  • a preferred implementation of unit 50 is described from page 1 of Appendix A until toward the end of page 4 thereof.
  • a preferred implementation of unit 60 is described from the end of page 4 of Appendix A until the middle of page 8. Subroutines and statisti ⁇ cal procedures which are useful in understanding the above mate ⁇ rial appear from page 8 of Appendix A onward.
  • the 3D representation including 3D information representating at least a portion or an aspect of the 3D scene or object, may be employed by a laser computer printer to generate a new view of the object or scene.
  • conventional CAD (computer aided design) software in conjunction with a conventional plotter may be employed to generate a new view of the object or scene.
  • the CAD software may also be operative to compare 2 CAD files in quality assurance applications.
  • Fig. 7 is a simplified flowchart illustration of a preferred 3D scene reconstruction method, operative in accordance with a preferred embodiment of the present invention, which is useful in conjunction with the apparatus of Fig. 6.
  • Fig. 7 is generally self-explanatory.
  • the 3 views of the scene or object can be digital, if they are accessed from a digital archive or generated, e.g. by a digital camera. If they are not digital, they are scanned or otherwise digitized.
  • SUBSTITUTESHEET( ULE ⁇ Any suitable conventional or other formats may be employed.
  • the Silicon Graphics's Inventor format may initially be employed for the 3D representation.
  • the Inventor format may be converted into Postscript in order to print a new view of the 3D representation.
  • the 3D representation of the scene is useful in performing a wide range of activities such as 3D measurements of the scene or object, generation, e.g. as a printout, of a new view of the scene or object, and quality assurance comparisons in which the generated 3D representation of the object or scene is compared to a desired object or scene or a desired 3D representation thereof, using conventional methods.
  • Fig. 8 is a simplified functional block diagram of image transfer apparatus, constructed and operative in accordance with a preferred embodiment of the present invention, which is operative to generate a novel view of a 3D scene from at least two reference views thereof.
  • the apparatus of Fig. 8 is similar to the apparatus of Fig. 6. Howev ⁇ er, in Fig. 8, a novel view of a 3D scene is directly generated from only at least two reference views thereof, preferably with ⁇ out generating a 3D representation intermediate.
  • Fig. 8 geometric information regarding the two reference views and the desired novel view is employed to gener ⁇ ate a trilinear tensor representing the geometric relationship between the three views.
  • at least 7 triplets of loca ⁇ tions in the novel view may be identified, e.g. manually, which correspond to 7 locations respectively in each of the at least two reference views.
  • at least some infromation regard ⁇ ing the novel view is available. For example, if it is desired to update a GIS (geographic information system) year-old view, based on at least two new reference views of the same area. It is typically possible to identify at least 7 locations in the year- old view which correspond to 7 locations in the two reference views and which can be assumed will still exist in the soon-to-be-generated current version of the year-old view.
  • GIS geo information system
  • the novel view is typically generated by computing a multiplicity of novel view locations each corresponding to dif ⁇ ferent first and second corresponding locations in said first and second reference views respectively based on said first and second corresponding locations and said trilinear tensor.
  • the matching point finder 140 may generate a multiplicity of pairs of matching points from the two reference views, say 1000 such pairs. For the first 7 pairs, a user may manually indicate 7 matching points in the novel view.
  • the coordinates of Matching Point Pairs 8 - 1000 may be plugged into the trilinear tensor, as shown in Fig. 9, in order to generate coordinates of matching points 8 - 1000 in the novel view.
  • the novel view thus generated may, for example, be compared to the same view as seen a year before, in order to identify differences in the scene which took place in the course of the year.
  • FIG. 9 is a simplified flowchart illustration of a preferred image transfer method, operative in accordance with a preferred embodiment of the present invention, which is useful in conjunction with the appa ⁇ ratus of Fig. 8.
  • Fig. 9 is generally self-explanatory.
  • intermediate tensors may be computed for each 3 views. Then, a representative tensor may be computed based on the relationships between these "intermediate" tensors.
  • Fig. 10 is an illustration of a 3D reconstruction of a shoe which is also illustrated in Fig. 2.
  • Fig. 10 was generated by finding matching points and reconstructing their 3D locations. Next the coordinates are processed by CAD software to generate the surface shown in Fig. 10.
  • Fig. 11 is a simplified block diagram illustration of a preferred method and apparatus for quality assurance of workpieces.
  • Fig. 11 includes an array 200 of 3 CCD cameras 210 which are aimed at a single location, so as to yield three perspectives of that location.
  • the CCD cameras are attached to a robot arm 212 and therefore can move relative to a workpiece 2 1 4 arriving along a conveyor belt 220 in accordance with suitable instructions from a controller 224.
  • the conveyor belt 220 When a workpiece enters the field of view of the C CD cameras 210, the conveyor belt 220 typically pauses to allow substantially the entirety of the surface area of the workpiece to be imaged by the cameras 210.
  • the controller 224 via robot arm 212, moves the camera array 200 around the object such that, typically, almost its entire surface area is imaged.
  • the camera array 200 may be moved through 10 different positions around the object, and at each position, each of the 3 CCD cameras images the workpiece. The number of positions employed depends on the complexity of the workpiece.
  • This process yields a plurality of image triplets, each image triplet including three digital images of the same portion of the workpiece, from 3 respective perspectives.
  • the corresponding position of the array 200 and of each of the cameras 210 may be computed, based on the robot arm's location, which is known, and using hand-eye calibration.
  • Each image triplet is processed by units 240, 250, 260, 270 and 290 which may be similar to units 40, 50, 60, 70 and 90, respectively, of Fig. 6.
  • the CAD model information generated by CAD S/W 290 from each image triplet is stored in a suitable memory 300.
  • a computation unit 310 is operative to integrate the multiplicity of probe locations corresponding to the multiplicity of positions of CCD camera array 200, into a single coordinate system.
  • the necessary coordinate transformations are computed by inverting the transformations which define the CCD camera array's motion.
  • a computational unit 320 compares the output of unit 310 to a reference CAD model and computes differences therebetween. These differences are compared, in a computational unit 330, to accepted tolerances.
  • the apparatus of Fig. 11 is also suitable for 3D digi ⁇ tization applications for reverse engineering or CAD (computer
  • Fig. 12 is a simplified block diagram illustration of a preferred method and apparatus for generating a digital terrain map, e.g. in order to update municipal maps, to detect illegal structures, to serve a car navigation system, or even to map a microscopic object such as an integrated circuit.
  • An airborne CCD camera 334 is flown over a scene for which it is desired to generate a digital terrain map.
  • the camera 334 generates 3 digital images of the scene from 3 respective perspectives.
  • the 3 images are processed by units 340, 350, 360 and 370 which may be similar to units 40, 50, 60 and 70 of Fig. 6.
  • a surface interpolation procedure is performed by a surface interpolation unit 380, on the output of 3D representa ⁇ tion generation unit 370.
  • a suitable surface interpolation method is described in Grimson, W. E. L, "A computational theory of visual surface interpolation", Proc. of the Royal Soc. of London
  • a camera array such as array 200 of Fig. 11 circles around an object to be visualized and images substantially the entire surface area which it is desired to display to a user.
  • the camera array may image the object from each of 200 positions surrounding the object.
  • Synthetic images may then be generated for positions other than the 200 above-mentioned positions.
  • a desired position may be indicated by a user, e.g. by means of a joystick.
  • the apparatus of the present invention may be used to generate a synthetic image for that position.
  • SUBSTITUTE SHEET (RULE 26 ⁇ Conventional driving simulation games employ synthetic backgrounds, however, the present invention may be employed to provide a driving simulation game with a real background.
  • an array of at least 3, and preferably 5 - 10 cameras is moved within a desired scene such that substantially the entirety of the scenes can be captured by the camera array from at least 3 and preferably more different perspectives.
  • the scene may be captured from each of approximately 1000 positions of the camera array.
  • New views are then generated, in accordance with the present invention, in order to accomodate a user's need for new views as indicated, e.g. by a joystick.
  • Algebraic functions useful for recognition are now described, based on an article entitled “Algebraic functions for recognition”, to be published in IEEE, Transactions on PAMI.
  • the central results are contained in Theorems 1, 2 and 3.
  • the coefficients of F can be recovered linearly without establishing first the epipolar geometry, 3D structure of the object, or camera motion.
  • the auxiliary Lemmas required for the proof of Theorem 1 may be of interest on their own as they ..establish certain regularities across projective transformations of the plane and introduce new view invariants (Lemma 4).
  • Theorem 2 addresses the problem of recovering the coefficients of the trilinear functions in the most economical way. It is shown that among all possible trilinear functions across three views, there exists at most four linearly independent such functions. As a consequence, the coefficients of these functions can be recovered linearly from seven corresponding points across three views.
  • Theorem 3 is an obvious corollary of Theorem 1 but contains a significant practical aspect. It i . shown that if the views v x . (_' 2 are obtained by parallel projection, then F reduces to a special bilinear form — or. equivalently, that any perspective view ⁇ can be obtained by a rational linear function of two orthographic views. The reduction to a bilinear form implies that simpler recognition schemes are possible if the two reference views (model views) stored in memory are orthographic.
  • the problem of re-projection can in principal be dealt with via 3D reconstruction of shape and camera motion.
  • the classic approaches for perspective views are known to be unstable under errors in image measurements, narrow field of view, and internal camera calibration [3, 9, 12] , and therefore, are unlikely to be of practical use for purposes of re-projection.
  • the non-metric approaches, as a general concept have not been fully tested on real images, but the methods proposed so far rely on recovering first the epipolar geometry — a process that is also known to be unstable in the presence of noise.
  • object space to be the three-dimensional projective space V 3
  • image space to be the two-dimensional projective space V 2
  • ⁇ C V s be a set of points standing for a 3D object
  • ⁇ , C V 2 denote views (arbitrary), indexed by ⁇ , of ⁇ .
  • the epipoles are defined to be at the intersection of the line 00' with both image planes. Because the image plane is finite, we can assign, without loss of generality, the value 1 as the third homogeneous coordinate to every observed image point.
  • image coordinates will denote the non-homogeneous coordinate representation of V 2 , e.g., (x, y), (x', y'), i ⁇ ", y") for the three corresponding points.
  • Planes will be denoted by 7r,, indexed by i, and just ⁇ if only one plane is discussed. All planes are assumed to be arbitrary and distinct from one another.
  • the symbol denotes equality up to a scale
  • GL n stands for the group of n x n matrices
  • PGL n is the group defined up to a scale.
  • ⁇ ⁇ , ⁇ 2 , ⁇ z be three arbitrary perspective views of some ob ⁇ ject, modeled by a set of points in 3D.
  • the image coordinates (x, y) € ⁇ ⁇ , ⁇ x', y') _ ⁇ and (x", y") £ ib 3 of three corresponding points across three views satisfy a pair of trilinear equations of the following form:
  • the coeffic i ent k is i ndependent of ⁇ , .e., is invariant to the choice of the second v t eu.
  • the scalar k is an affine invar i ant w i th i n a projective framework, and is called a relative affine invariant.
  • Homographies A, _ PGL 3 from ⁇ . ⁇ ⁇ x due to the same plane are sa i d to be scale-compatible if they are scaled to satisfy Lemma 1, i.e., for any point P € pro j ect i ng onto p e ⁇ . and ? 6 i t there exists a scalar k that satisfies
  • a - sA' [ v', ⁇ v', ⁇ fv' ⁇ , for some coefficients a, ⁇ , - .
  • Hp v' for all p _ 1 and s q is a fixed scalar s.
  • H is a matrix whose columns are multiples of v'. ⁇ ⁇
  • Lemma 4 (Auxiliary — Uniqueness)
  • the scalars _s, ⁇ , /3, of Lemma 2 are invariants indexed by ⁇ x , ⁇ , ⁇ 2 . That is, given an arbitrary third view ⁇ 3 .
  • B,B' be the homographies from ⁇ x ⁇ ⁇ 3 due to ⁇ x , ⁇ 2 , respectively.
  • B be scale-compatible with A
  • B' be scale-compatible with A 1 . Then,
  • B - sBCTC-' BCA- ⁇ HC ⁇ .
  • B' BCTC ⁇
  • the matrix A ⁇ H has columns which are multiples ot v (because A V S . y ), CA ⁇ l H is a matrix whose columns are multiple of ⁇ , and BCA ⁇ l H i s a matrix whose columns are multiples of v".
  • Pre-multiplying BCA ⁇ l H by C" 1 does not change its form because every column of BCA ⁇ HC ⁇ is simply a linear combination of the columns of BCA ⁇ H.
  • B - sB > is a matrix whose columns are multiples of v".
  • the direct implication of the theorem is that one can generate a novel view ( ⁇ 3 ) by simply combining two model views ( ⁇ , ⁇ 2 )-
  • the coefficients cx_. and ⁇ 3 of the combination can be recovered together as a solution of a linear system of 17 equations (24 — 6 — 1 ) given nine corresponding points across the three views (more than nine points can be used for a least-squares solution).
  • Theorem 2 There exists nine distinct trilinear forms of the type described in Theorem 1. of which at most four are linearly independent. The coefficients of the four trilinear forms can be recovered linearly with seven corresponding points across the three views.
  • the first four functions on the list produce a 4 x 27 matrix.
  • the rank of the matrix is four because it contains four orthogonal columns (columns associated with ⁇ ⁇ , c. 12 , ⁇ 2 i and 22 ), therefore these functions are linearly independent. Since we have 27 coefficients, and each triplet p, p', p" contributes four linear equations, then seven corresponding points across the three views provide a sufficient number of equations for a linear solution for the coefficients (given that the system is determined up to a common scale, seven points produce two extra equations which can be used for consistency checking or for obtaining a least squares solution).
  • both theorems provide a constructive means for solving for the positions x" , y" in a novel view given the correspondences p, p' across two model views.
  • This process of generating a novel view can be easily accomplished without the need to explicitly recover structure, camera transformation, or even just the epipolar geometry — and requires fewer corresponding points than any other known alternative.
  • Equation (12) is also a trilinear form, but not of the type introduced in Theorem 1.
  • the differences include (i) epipolar intersection requires the correspondences coming from eight points, rather than seven, (ii) the position of p" is solved by a line intersection process which is singular in the case the three camera centers are collinear; in the trilinearity result the components of p" are solved separately and the situation of three collinear cameras is admissible, (iii) the epipolar intersection process is decomposable, i.e., only two views are used at a time; whereas the epipolar geometries in the trilinearity result are intertwined and are not recoverable separately.
  • each point contributes four equations, but here there is no ad ⁇ vantage for using all four of them to recover the coefficients, therefore we may use only two out of the four equations, and require four corresponding points to recover the coefficients.
  • x" (y) is expressed as a linear combination of image coordinates of the two other views — as discovered by [38].
  • a bilinear function of three views has two advantages over the general trilinear function. First, as mentioned above, only five corresponding points (instead of seven) across three views are required for solving for the coefficients. Second, the lower the degree of the algebraic function, the less sensitive the solution may be in the presence of errors in measuring correspondences. In other words, it is likely (though not necessary) that the higher order terms, such as the term x"x'x in Equation 3, will have a higher contribution to the overall error sensitivity of the system.
  • the epipolar intersection method was implemented as described in Section III by recover ⁇ ing first the fundamental matrices. Although eight corresponding points are sufficient for a linear solution, in practice one would use more than eight points for recovering the fundamen ⁇ tal matrices in a linear or non-linear squares method. Since linear least squares methods are still sensitive to image noise, we used the implementation of a non-linear method described in [20] which was kindly provided by T. Luong and L. Quan (these were two implementations of the method proposed in [20] — in each case, the implementation that provided the better results was adopted).
  • the first experiment is with simulation data showing that even when the epipolar geometry is recovered accurately, it is still significantly better to use the trilinear result which avoids the process of line intersection.
  • the second experiment is done on a real set of images, comparing the performance of the various methods and the number of corresponding points that are needed in practice to achieve reasonable re-projection results.
  • Focal len g th was of 50 units and the first view was obtained by fx/z, fy/ .
  • the second view ( ⁇ , 2 ) was generated by a rotation around the point (0, 0, 100) with axis (0.14, 0.7, 0.7) and by an angle of 0.3 radians.
  • the third view ( ⁇ 3 ) was generated by a rotation around an axis (0, 1, 0) with the same translation and angle.
  • Figure 2A-2C shows three views of the object we selected for the experiment.
  • the object is a sports shoe with added texture to facilitate the correspondence process.
  • This object was chosen because of its complexity, i.e., it has a shape of a natural object and cannot easilv be described parametrically (as a collection of planes or algebraic surfaces). Note that the situation depicted here is challenging because the re-projected view is not in-between the two model views, i.e.. one should expect a larger sensitivity to image noise than in-between situations.
  • a set of 34 points were manually selected on one of the frames. ⁇ , and their correspondences were automatically obtained along all other frames used in this experiment.
  • the correspondence process is based on an implementation of a coarse-to-fine optical-flow algorithm described in [7]. To achieve accurate correspondences across distant views, in ⁇ termediate in-between frames were taken and the displacements across consecutive frames were added. The overall displacement field was then used to push ( "warp") the first frame towards the target frame and thus create a synthetic image. Optical-flow was applied again between the synthetic frame and the target frame and the resulting displacement was added to the overall displacement obtained earlier. This process provides a dense displacement field which is then sampled to obtain the correspondences of the 34 points initially chosen in the first frame. The results of this process are shown in Figure 2A-2C by displaying squares centered around the computed locations of the corresponding points.
  • the trilinear method requires at least seven corresponding points across the three views (we need 26 equation, and seven points provide 28 equations), whereas epipolar intersection can be done (in principle) with eight points.
  • the question we are about to address is what is t he number of points that are required in practice (due to errors in correspondence, lens distortions and other effects that are not adequately modeled by the pin-hole camera model ) to achieve reasonable performance?
  • the trilinear result was first applied with the minimal number of points (seven) for solving for the coefficients, and then applied with 8,9, and 10 points using a linear least-squares solution (note that in general, better solutions may be obtained by using SVD or Jacobi methods instead of linear least-squares, but that was not attempted here).
  • the results are shown in Figure 3A-3B.
  • Seven points provide a re-projection with maximal error of 3.3 pixels and average error of 0.98 pixels.
  • the solution using 10 points provided an improvement with maximal error of 1.44 and average error of 0.44 pixels.
  • the performance using eight and nine points was reasonably in-between the performances above. Using more points did not improve significantly the results; for example, when all 34 points were used the maximal error went down to 1.14 pixels and average error stayed at 0.42 pixels.
  • any view of a fixed 3D object can be expressed as a trilinear function with two reference views in the general case, or as a bilinear function when the reference views are created by means of parallel projection.
  • These functions provide alternative, much simpler, means for manipulating views of a scene than other methods.
  • thev require fewer corresponding points in theory, and much fewer in practice.
  • Experimental results show that the trilinear functions are also useful in practice yielding performance that is significantly better than epipolar intersection or the linear combination method.
  • the present invention has very broad applications and specifically is applicable in all fields in which 3D from 2D techniques are known to be useful.
  • Applications of the present invention include at least the following: photogrammetry applica ⁇ tions comprising map making from aerial and satellite photographs and coordinate measurements in aerospace and shipyard assembly plants, coordinate measurements of industrial parts (CMM) , automated optical based inspection of industrial parts, robotic cell alignment, robotic trajectory identification, 3D robotic feedback, 3D modelling of scenes, 3D modelling of objects, re ⁇ verse engineering, and 3D digitizing.
  • Appendix B is a listing of a preferred software imple ⁇ mentation of 3D reconstruction apparatus constructed and opera ⁇ tive in accordance with a preferred embodiment of the present invention.
  • Maple2 software commercially available from Math- Soft, may be employed in conjunction with an IBM PCT or SUN workstation.
  • the following procedure may be employed: a. Programs generated from the listing of Appendix B may be loaded into Maple2 using the OPEN FILE command. b.
  • the subroutines may be run by bringing the cursor to the row on which appears the WITH (LINALG) command.
  • the RETURN key is then processed until the cursor reaches the end of the file.
  • the cursor is then returned to the beginning of the file.
  • the simulation is run by pressing RETURN until the following row is reached, inclusive:
  • EVALN PCOMP
  • Appendix B is also useful in implementing the image transfer apparatus and method shown and described herein.
  • the image transfer embodiment may, for example, be based on the listing of Appendix B used in conjunction with Equations 3 and 4 of the above section entitled Algebraic Function for Recognition.
  • Eqn dmatrix(0, 3*sizeJE_basis*(size_E_basis-l) + 6*size_E_basis*size_E_basis - 1, 0, 2);
  • Eqn[n_Eq][l] E_basis[rf][i][l][0][k]* E_basis[rf][i]D][2][k] - E_basis[rfJ[i][l][2][k]* E_basis[rfJ[i]D][0][k];
  • Eqn[n_Eq][2] E_basis[rf][i]0][0][k]* E_basis[rf][i][l][l][k] -
  • each of the 3 4-by-3 matrices E_basis[*][*][j] is of */ /* rank 2.
  • svd dmatrix(l,4,l,3)
  • v_tr dmatrix( 1 ,3 , 1 ,3 )
  • w dvector(l,3);
  • BOOL removeEqOutliers /* remove outliers for homogeneous system */
  • subMat dmatrix(0, minRows+maxSize-1 - 1, 0, nCols - 1);
  • 3D points are generated by a pseudo random generator and the views are generated by choosing Camera parameters:
  • the first view camera transformation is the 3x4 matrix [I 0] where I is the identity matrix.
  • the second view camera transformation i.s the 3x4 matrix [R vp] where R is a rotation matrix and ⁇ ⁇ ; -s translation.
  • Warning new definition for norm Warning: new definition for trace
  • Test[3.i+63: col(M1 ,3)[l];
  • Test: submatrlx(Test,1..18,1..8):
  • Wc need at least 5 points.

Abstract

A method for generating information regarding a 3D object from at least one 2D projection thereof. The method includes providing at least one 2D projection (40) of a 3D object, generating an array of numbers (50, 60) described by: aijk = vi'bik - vj''ajk (i,j,k = 1,2,3), where ajk and bjk are elements of matrices A and B respectively and vi' and vi'' are elements of vectors v' and v'' respectively, wherein the matrices (50) and vectors (60) together describe camera parameters of three views (102) of the 3D object and employing the array to generate information regarding the 3D object (70).

Description

APPARATUS AND METHOD FOR RECREATING AND MANIPULATING A 3D OBJECT BASED ON A 2D PROJECTION THEREOF
FIELD OF THE INVENTION
The present invention relates to apparatus and methods for processing 2D projections of 3D objects and particularly to apparatus and methods for geometric analysis of a 2D projection image.
BACKGROUND OF THE INVENTION
Publications describing state of the art methods for processing 2D projections of 3D objects, and technologies relevant thereto, are described in references cited hereinbelo .
The disclosures of all publications mentioned in the specification and of the publications cited therein are hereby incorporated by reference.
SUMMARY OF THE INVENTION
The present invention seeks to provide image transfer apparatus and methods which are useful for generating a novel view of a 3D scene from first and second reference views thereof. The present invention also seeks to provide 3D scene reconstruction methods and apparatus for generating a 3D representation of a 3D scene from first, second and third views thereof.
The present invention also seeks to provide improved apparatus and methods for processing 2D projections of 3D ob¬ jects.
The present invention also seeks to provide methods for reconstruction of a 3D object based on a trilinear tensor defined on three views of the 3D object.
The present invention additionally seeks to provide methods for image transfer for a 3D object based on a trilinear tensor defined on three views of the 3D object.
The present invention also seeks to provide an image transfer method for generating a novel view of a 3D scene from first and second reference views thereof, the method including providing first and second reference views of a 3D scene, employ¬ ing geometric information regarding the first reference view, second reference view and novel view, respectively, to generate a trilinear tensor representing the geometric relationship between the first, second and novel views and generating the novel view by computing a multiplicity of novel view locations each corre¬ sponding to different first and second corresponding locations in the first and second reference views respectively based on the first and second corresponding locations and the trilinear ten¬ sor.
The step of providing may, for example, comprise scan¬ ning in the first and second reference images. The step of employing may, for example, include the step of generating a set of first, second and third corresponding locations in the first reference view, second reference view and novel view, respectively.
There is thus provided, in accordance with a preferred embodiment of the present invention, an image transfer method for generating a novel view of a 3D scene from first and second reference views thereof, the method including providing first and second reference views of a 3D scene, employing geometric infor¬ mation regarding the first reference view, second reference view and novel view, respectively, to generate a trilinear tensor representing the geometric relationship between the first, second and novel views, and generating the novel view by computing a multiplicity of novel view locations each corresponding to dif¬ ferent first and second corresponding locations in the first and second reference views respectively based on the first and second corresponding locations and the trilinear tensor.
Further in accordance with a preferred embodiment of the present invention, the step of providing includes scanning in the first and second reference images.
Still further in accordance with a preferred embodiment of the present invention, the step of employing includes the step of generating a set of first, second and third corresponding locations in the first reference view, second reference view and novel view, respectively.
There is also provided, in accordance with a preferred embodiment of the present invention, a 3D scene reconstruction method for generating a 3D representation of a 3D scene from first, second and third views thereof, the method including providing first, second and third views of a 3D scene, employing geometric information regarding the first, second and third views to generate a trilinear tensor representing the geometric relationship between the first, second and third views, and generating a 3D representation of the 3D scene from the trilinear tensor.
Further in accordance with a preferred embodiment of the present invention, the step of generating a 3D representation includes computing an epipolar geometric representation of the first and second views from the trilinear tensor, and generating the 3D representation from the epipolar geometric representation. Also provided, in accordance with another preferred embodiment of the present invention, is image transfer apparatus for generating a novel view of a 3D scene from first and second reference views thereof, the apparatus including apparatus for providing first and second reference views of a 3D scene, a trilinear tensor generator operative to employ geometric informa¬ tion regarding the first reference view, second reference view and novel view, respectively, to generate a trilinear tensor representing the geometric relationship between the first, second and novel views, and a novel view generator operative to generate the novel view by computing a multiplicity of novel view loca¬ tions each corresponding to different first and second corre¬ sponding locations in the first and second reference views re¬ spectively based on the first and second corresponding locations and the trilinear tensor.
Further provided, in accordance with another preferred embodiment of the present invention, is 3D scene reconstruction apparatus for generating a 3D representation of a 3D scene from first, second and third views thereof, the apparatus including appparatus for providing first, second and third views of a 3D scene, a trilinear tensor generator operative to employ geometric information regarding the first, second and third views to generate a trilinear tensor representing the geometric relation¬ ship between the first, second and third views, and a 3D scene representation generator operative to generate a 3D representa¬ tion of the 3D scene from the trilinear tensor.
Also provided, in accordance with another preferred embodiment of the present invention, is a method for generating information regarding a 3D object from at least one and prefera¬ bly three 2D projections thereof, the method including providing at least one and preferably three 2D projections of a 3D object, generating an array of numbers described by: αijk = vi' bjk " vj" aik (i'J'k = 1,2,3), where a^-; and b.;k are elements of matrices A and B respectively and VJ 1 and Vj_" are elements of vectors v1 and v" respectively, wherein the matrices and vectors together describe camera parameters of three views of the 3D object, and employing the array to generate information regarding the 3D object.
Also provided, in accordance with a preferred embodi¬ ment of the present invention, is a method for generating a new view of a 3D object from first and second existing views thereof having n corresponding points p^ and p^' (i = 1 ... n) , the method including generating a tensor a^jk, and plugging the values of points p^ and p^' (i = 1 ... n) into trilinear forms, thereby to extract an x", y" value representing a location in the new view, and generating the new view on the basis of the result of the plugging in step.
Further provided, in accordance with a preferred embodiment of the present invention, is a method for reconstruct¬ ing a 3D object from at least one and preferably three 2D projec¬ tions thereof, the method including providing at least one and preferably three 2D projections of a 3D object, generating an array of numbers described by: αijk = vi' bjk " vj" aik (i 'k = 1,2,3), where a^.; and b-jk are elements of matrices A and B respectively and v; 1 and v^" are elements of vectors v' and v" respectively, wherein the matrices and vectors together describe camera parameters of three views of the 3D object, permuting the array into three homography matrices associated with three corre¬ sponding planes in 3D space, and employing the three homography matrices to reconstruct the 3D object.
Also provided, in accordance with a preferred embodi¬ ment of the present invention, is a visual recognition method including providing three perspective views of a 3D object be¬ tween which a trilinear relationships exists, and employing the trilinear relationship between the views in order to perform visual recognition by alignment.
Further in accordance with a preferred embodiment of the present invention, the method also includes reprojecting the 3D object.
Still further in accordance with a preferred embodi¬ ment of the present invention, the information regarding the 3D object includes a reconstruction of the 3D object.
Additionally in accordance with a preferred embodiment of the present invention, the information regarding the 3D object includes at least one new view of the 3D object generated without reconstructing the 3D object.
Further in accordance with a preferred embodiment of the present invention, the at least one and preferably three 2D projections includes at least one aerial photograph.
Still further in accordance with a preferred embodi¬ ment of the present invention, the at least one and preferably three 2D projections includes at least one satellite photograph.
Further in accordance with a preferred embodiment of the present invention, the information regarding the 3D object comprises at least one coordinate of the 3D object.
Further in accordance with a preferred embodiment of the present invention, the 3D object includes an aerospace ob¬ ject.
Still further in accordance with a preferred embodi¬ ment of the present invention, the 3D object includes a large object such as a ship.
Further in accordance with a preferred embodiment of the present invention, the 3D object includes a nonexistent object.
Also provided, in accordance with a preferred embodi¬ ment of the present invention, is apparatus for generating infor¬ mation regarding a 3D object from at least one and preferably three 2D projections thereof, the apparatus including apparatus for providing at least one and preferably three 2D projections of a 3D object, an array generator operative to generate an.array of numbers described by: αijk = vi' bjk " vj" aik (i' 'k = 1,2,3) , where aj_-; and b-jk are elements of matrices A and B
SUBSTITUTESHEET(RULE2B. respectively and Vj_' and v^" are elements of vectors v' and v" respectively, wherein the matrices and vectors together describe camera parameters of three views of the 3D object, and an array analyzer employing the array to generate information regarding the 3D object.
Also provided, in accordance with a preferred embodi¬ ment of the present invention, is apparatus for generating a new view of a 3D object from first and second existing views thereof having n corresponding points p^ and p^1 (i = 1 ... n) , the apparatus including apparatus for generating a tensor -^^k, and apparatus for plugging the values of points p.^ and p^' (i = 1 ... n) into trilinear forms, thereby to extract an x", y" value representing a location in the new view, and apparatus for gener¬ ating the new view on the basis of the result of the plugging in step.
Also provided, in accordance with a preferred embodi¬ ment of the present invention, is apparatus for reconstructing a 3D object from at least one and preferably three 2D projections thereof, the apparatus including apparatus for providing at least one and preferably three 2D projections of a 3D object, an array generator operative to generate an array of numbers described by:
Qijk = vi' bjk " vj" aik (i'J** = 1,2,3) , where a^.: and b-:^ are elements of matrices A and B respectively and v^' and v^" are elements of vectors v' and v" respectively, wherein the matrices and vectors together describe camera parameters of three views of the 3D object, an array permutator operative to permute the array into three homography matrices associated with three corresponding planes in 3D space, and a 3D object reconstructor operative to employ the three homography matrices to reconstruct the 3D object.
Further provided, in accordance with a preferred embodiment of the present invention, is visual recognition appa¬ ratus including apparatus for providing three perspective views of a 3D object between which a trilinear relationships exists, and apparatus for employing the trilinear relationship between the views in order to perform visual recognition by alignment. Further in accordance with a preferred embodiment of the preSent invention, at least one result of performing the above method is employed in order to perform at least one of th following applications: map making from aerial and satem Photographs and coordinate measurements in aerospace and shipyard assembly plants, coordinate measurements of industrial parts (CMM) , automated optical based inspection of industrial parts robotic cell alignment, robotic trajectory identification 3D robotic feedback, 3D modelling of scenes, 3D modelling of' ob- Dects, reverse engineering, and 3D digitizing.
Algebraic computation processes useful for recognition are described below with reference to Figs. 1 - 5.
BRIEF DESCRIPTION OF THE DRAWINGS AND APPENDICES
The present invention will be understood and appreciat¬ ed from the following detailed description, taken in conjunction with the drawings in which:
Fig. 1 is an illustration of two graphs comparing the performance of an epipolar intersection method, shown in dotted line, with the performance of a trilinear functions method, shown in dashed line, in the presence of image noise;
Fig. 2 is a pictorial illustration of two model views of a three-dimensional scene as well as a third reprojected view thereof;
Fig. 3 is a pictorial illustration of a reprojection using a trilinear result;
Fig. 4 is a pictorial illustration of a reprojection using intersection of epipolar lines;
Fig. 5 is a pictorial illustration of a reprojection using a linear combination of views method;
Fig. 6 is a simplified functional block diagram of 3D scene reconstruction apparatus, constructed and operative in accordance with a preferred embodiment of the present invention, which is operative to generate a 3D representation of a 3D scene from at least three views thereof;
Fig. 7 is a simplified flowchart illustration of a preferred 3D scene reconstruction method, operative in accordance with a preferred embodiment of the present invention, which is useful in conjunction with the apparatus of Fig. 6;
Fig. 8 is a simplified functional block diagram of image transfer apparatus, constructed and operative in accordance with a preferred embodiment of the present invention, which is operative to generate a novel view of a 3D scene from at least two reference views thereof; and
Fig. 9 is a simplified flowchart illustration of a preferred image transfer method, operative in accordance with a preferred embodiment of the present invention, which is useful in
9 SUBSTITUTESHEET(RULE26 conjunction with the apparatus of Fig. 8;
Fig. 10 is an illustration of a 3D reconstruction of a shoe which is also illustrated in Fig. 2;
Fig. 11 is a simplified block diagram illustration of a preferred method and apparatus for quality assurance of workpieces; and
Fig. 12 is a simplified block diagram illustration of a preferred method and apparatus for generating a digital terrain map.
Also attached herewith are appendices which aid in the understanding and appreciation of one preferred embodiment of the invention shown and described herein, namely:
Appendix A, which is a listing, in »c» language, of a preferred implementation of trilinear computation unit unit 50 and epipolar geometry generation unit 60 of Fig. 6; and
Appendix B, which is a listing of a preferred software implementation of 3D reconstruction apparatus constructed and operative in accordance with a preferred embodiment of the present invention.
Detailed Description of preferred Embodiments
A pin-hole camera, like 35mm still camera or Video recorder, produces a two-dimensional projection (2D) of the viewed three-dimensional (3D) world. The resulting image can be analyzed on a geometric and photometric level. The geometric level means the geometric relation between the locations of features (points, lines) in 3D and their respective location in the 2D image. The photometric level means the radiometric (reflectance properties of the surface, the spectral properties of the light sources illuminating the scene, etc.) relation between the scene and the luminosity (pixel grey values) in the image.
The 3D world is modeled as a cloud of points and so is the 2D world. In other words, we identify point features in the image that correspond to point features in 3D. There is a geometric relation between the two sets of points. When the camera moves in 3D we get more images of the same 3D world (i.e., of the same cloud of 3D points) and the question of interest is the geometrical relation between the corresponding set of 2D image points and the set of 3D points. In other words, if P, denotes a set of 3D points, and Pi, p'ψ,p" are three sets of 2D points (across three images) arranged such that points with same index _ correspond to the same 3D point P;, then the image sets alone reveal much information about the 3D set.
There are two questions of interest: (i) what is necessary to measure in the 2D images (how many points across how many images) in order to recover the location of the 3D points (reconstruction), (ii) given the image correspondences can we synthesis new vieλvs (the problem of image transfer) without explicitly reconstructing the object. The patent deals with both questions.
The general material of 3D-from-2D is an active area in Computer Vision (structure-from- motion and stereopsis), and is the sole engine in the industry of photogrammetry (map making from areal and satellite photographs, coordinate measurements in aerospace and shipyard assembly plants, etc.).
The relationship between 3D points and 2D points can be represented as a 3 x 4 transfor¬ mation matrix:
p ^ MP, where p = (_ , y, l )τ represents the 2D point in homogeneous coordinates (i.e., the image plane is the 2D projective plane), where x, y are the horizontal and vertical components of the point's location in the image (with respect to some arbitrary origin - say the geometric center of the image plane); P is represented by a 4-vector, which means that we view the object space as embedded in the three-dimensional projective space. The matrix M is 3 x 4 and represents the camera parameters (for example, the vector Q defined by MQ=0 is the camera's center of projection — the point where all the optical rays converge to). The sign = denotes equality up to scale.
Because we are embedding the image and the object into projective spaces (which means that the world can undergo rigid motion and be stretched and sheared) we can transform the object space such that the camera transformation associated with the first view is [/, 0], where / is the identity matrix. For three views we have:
P = [/, 0] P' = [A, V- P p" = [B, v"]P where P = (x, y, l, k)r, and the image points are p = (x, y, l)τ,p' = (x', y', l)τ and p" = {x", y", l)τ . Note: what we measure in the images are the locations of p,p', p", the location of P (i.e., the value of k) is not known. A and B are some 3 x 3 matrices (not independent of each other) and v', v" are some 3-vectors together describing the camera parameters of the three views. Note: the camera parameters A, B, v', v" are unknown. In other words, given three views with corresponding points Pi,p'i,p", i = l, ..., n, there exists a set of numbers /_, and parameters A, B, υ', v" such that the equations above hold for all i.
I. TRILINEAR TENSOR The array of 27 numbers described by: otijk = v'ibjk - v"aik. i,j, k — 1, 2, 3 where ars. brs are the elements of A, B, and vr' , v" are the elements of v', υ", is referred to as the "trilinear tensor" . These numbers are invariant to the particular projective representation of the 3D and 2D worlds, i.e., they are intrinsic to the three views (this is one of the general properties of tensors that they do not depend on the choice of basis for representation).
NOTE: the array tjk provides the foundation for numerous applications. A preferred method for producing the array from image measurements p,p', p" will be described next.
12
SUBSTITUTE SHEET (RULE 2S A . recovering α,^ from, image measurements
A corresponding triplet p, p', p" satisfies a number of trilinear relationships. First notation: We identify vectors and matrices by fixing some of the indices while varying others. For example, ,j_ is a set of scalars, a,j. is a set of 9 vectors (k varies while i, j remain fixed): α... is a set of 3 matrices (c-i.., 2.. and 3..), and so forth.
For a given order (i.e., we do not permute the views), there are 9 trilinear functions, that come in sets of 4 linearly independent forms. The coefficients of these forms are elements of the array aijk. For example, the following four forms are linearly independent invariant functions of three views:
x II a Tl3.p - x IIx i 3 T3.p + i x / a3 Tλ.p - a Tu.p = 0 r_, y"ai3.p - y"χl(χ z.p + χ'<χ_~2.p - <χj2.p = o, χ"a23-p - χ"y'a33.p + y'aIi.p -
Figure imgf000015_0001
= o, y"<*_~z.p - y"y'oi__jf + y'oc 2.p - a22.P = o.
The remaining functions are described in equations 5-11 presented herein.
Smce every corresponding triplet p, p', p" contributes four linearly independent equations, then seven corresponding points across the three views uniquely determine (up to scale) the tensor ,-^.
NOTE 1: what this means is that if we identify 7 triplets of corresponding points across the three views — say the object is a human face and we track the corner of the nose across three views: its location in the first view will be denoted by (x, y), the location of the nose in the second view will be denoted by (x', y') and the location of the nose in the third view is i ", y") — then each such triplet provides 4 linearly independent equations for the 27 parameters we seek to recover. Thus, we obtain 28 equations which is more than sufficient to solve for α,-^ .
NOTE 2: There are many ways to solve for linear systems, especially when more equations than unknowns are provided (i.e., when more than 7 corresponding triplets are given), for example least-squares technique, and robust methods for removing outliers within least- squares techniques.
13
SUBSTITUTE SHEET (RULE 16) II. RANK 4 CONSTRAINT: TENSORIAL CONNECTIONS ACROSS MORE THAN 3 VIEWS
Given a set of m > 3 views, the various trilinear tensors of subsets of 3 views out of entire set of views are not independent, and in fact are linearly related to each other. This relation, described below, we call "the rank 4 constraint" .
We arrange a tensor of three views a tensor in a 9 x 3 matrix denoted by G, where column j = 1, 2.3 contains the elements α.lιj_1, c_ι _2, ..., 0.3.^.3.
Let the the arbitrary set of m views be arranged as φι, φ2, φ_ , j = 3, ..., m. Consider the m — 2 tensors of views < .£ _., _-_> Φ_ > and let the correspondence G matrices be denoted by (?_,. Then the concatenation of the Gj matrices into a 9 3(m— 2) matrix has rank 4 regardless of m (in general, if the G_ matrices were independent, the rank of the concatenation would have been the smaller dimension, i.e., 9 when m > 5).
The rank-4-constraint implies that the four largest principle components of the concate¬ nation matrix represents the geometry between views φ_ and ψ2, 'n a way that is statically optimal.
A. Application 1: Image Transfer
Say we have two views with corresponding points Pi,p[, i = 1, ..., n. We wish to create a new view of the same object. Given the tensor _tJjt we plug into the trilinear forms the values of pι, p[ and extract the value of x", y", regardless of the manner in which tjk is obtained.
The tensor ι:k can be obtained, for example, if we know the locations of p" for 7 points, e.g., i = 1, ..., 7. Then for every 8th point we can recover p" from the pair p, p' and the tensor.
B. Application 2: Reconstruction
A tensor of three views, or the four largest principle components of the concatenation matrix of m > 3 views (see Section II) can be used to recover the epipolar geometry between two views, and from there to be used for reconstructing a 3D model of the scene.
Let E: = a.}. be three 3 x 3 matrices, i.e., E_ = αij,ι, αij,2, —, a3j,3- Then this particular permutation of the array of 27 numbers into three matrices yields the following result:
The three matrices Eχ , E , E3 are projective transformations (homography matrices) as¬ sociated with some three planes in 3D space What this means is that there exists three planes T J in space, such that if P is coplanar with 71"! (for example) and p, p' are the image locations in the first two views, then Exp = p'. Similarly for j = 2, 3.
The "fundamental matrix" F (or "coplanarity constraint" as used in photogrammetrical literature) between the first two views satisfies:
Figure imgf000017_0001
a relation that provides a linear system of equations for solving for F.
Similarly, the four largest principle components of the concatenation matrix (Section II) for m > 3 views can be used to solve for F, as follows. Let Ak, k = 1, ..., 4 be the four largest principle components of the concatenation matrix, each arranged in a 3 x 3 matrix (row- wise). Then,
A~F + FτAk = 0 k = 1, 2, 3, 4
which provides a statically optimal way to linearly recover F from m > 3 views.
The "epipolar" point v' can be recovered, either by the relation Fτv' = 0, or directly from the tensor as follows:
The cross-product of corresponding columns of each two of the matrices out of the three matrices E: , E2, E3 provides an epipolar line, i.e., a line that is coincident with υ'. Similarly, if we denote by αi , α2, α3 the columns of one E matrix, and by 6l 5 62, 63 the columns of another E matrix, then α. x bj — b. x tij
is an epipolar line for i φ j. Taken together the tensor gives rise to 18 epipolar lines whose intersection defines the epipole v'. The point of intersection could be found by, for example, a least-squares method.
Similarly, the epipolar point v' can be recovered from
Figure imgf000017_0002
..., A _, the four largest principle components concatenation matrix, as follows, let α!, α2, α3 be the the columns of matrix A\, and by bχ , b , b3 the columns of _42, then
α. x bj — 6, x _, is an epipolar line for ι Φ j , and α, x b, provides another set of three epipolar lines. By selecting in this way two out of the four matrices Λ1 , ..., Λ4 we get 24 epipolar lines which provide a redundant set of equations for solving for v'.
The projective model of the 3D scene follows from F and v', for example: the 3 x 4 matrix [[v']F, v'} is a camera transformation from 3D coordinates to the image coordinates of the second view. In other words, for every matching pair p, p' of points in first and second view, respectively, we have: p' ^ [v~Fp + kυ', thus the coordinate vector [x, y, 1 , k] provides a 3D (projective) model of scene.
NOTE 1: the permutation ,.. yields also three homography matrices Wx , W2, W3 of three other planes. The homography matrices are from first two third view, i.e., W\ = p" for p, p" coming from P coplanar with the plane associated with W_. Thus, analogous to the use of the matrices E1, E2, E3, we can recover the fundamental matrix F" and epipolar point υ" associated with the epipolar geometry between the first and third view.
16
SUBSTITUT ACHIEVING IMAGE CORRESPONDENCES ACROSS 3 VIEWS
One of the important steps in manipulating 3D scenes from 2D imagery is the step of obtaining a reliable set (as dense as possible) of matching points across the views at hand. The current state-of-the-art use correlation techniques across the image intensities of two views together with geometric information, such as the epipolar geometry.
With the tensor one can extend the correspondence method to take advantage of three views together and without assuming camera calibration. Given an initial set of corre¬ sponding points across three views one recovers the trilinear tensor. Then, the trilinear tensor provides a constraint for using correlation methods to accurately locate the matching points. This can be done as follows.
Let E! , E2, E3 denote the arrangement a.,, of the tensor and W_, W , W3 denote the ar¬ rangement a,., of the tensor. We first align all three images such that the displacements between matching points become only along epipolar lines. This is done as follows.
Let E = _ j pjEj where the coefficients pj are the solution of the linear least-squares con¬ dition Epk = p'k for all initial matching points ph,p'k in first and second image, respectively. Similarly, let W = ∑, μtW. where the coefficients μ,- are the solution of the linear least- squares condition Wpk = pk for all initial matching points Pk, pk' in first and third image, respectively. The process of "warping" image two with the transformation E~l and image three with the transformation W~λ produces two new images denote by 2' and 3', such that the matching points across images 1, 2' are along epipolar lines and the matching points across images 1 , 3' are along epipolar lines.
The epipolar lines can be derived from the tensor as presented herein (Section II. B in detailed description of preferred embodiment). Let w' and w" be the direction vectors of the epipolar lines between 1, 2' and 1, 3', respectively. Hence, if p = (x, y),p' = (x', y'),p" = (_", y") are matching points in images 1, 2', 3' respectively, then p' = p+aw' and p" = p+βw" for some coefficients , β. Note that to', w", a, β are quantities that change with the location
(*. _/)•
The trilinear equations, (3)-(6) presented herein, provide a geometric constraint on the coefficients , β, i.e., for each arbitrary value of α, the value of β is then uniquely determined, and vice versa. In order to recover both _, β we use the image intensities of 1.2', 3' as follows. The coefficients , β satisfy the "constant brightness equation":
a(Ix, Iy)Tw' + It 12' = 0 β{I*Jy)Tw" + I? = 0
where (/,, /„) is the gradient vector at location p = (χ, y) in the first image and /»' is the temporal derivative between images 1, 2', and /«' is the tempQral ^.^ ^ images 1, 3' (all derivatives are discrete approximations). From the trilinear equations (3)- (6) presented herein, we can obtain /(«, β) = 0 where the coefficients of the function /() are elements of the tensor. Thus one obtains a least-squares solution for a and β that satisfies both geometric an photometric constraints.
^ The entire procedure can be done hierarchically along a Laplacian Pyramid as described in "J.R. Bergen and R. Hingorani, Hierarchical motion-based frame rate conversion"; Technical report. David Sarnoff Research Center, 1990.
Reference is now made to Fig. 6 which is a simplified functional block diagram of 3D scene reconstruction apparatus, constructed and operative in accordance with a preferred embodi¬ ment of the present invention, which is operative to generate a 3D representation of a 3D scene from at least three views there¬ of.
The apparatus of Fig. 6 includes apparatus for provid¬ ing at least 3 digital images of a 3D scene or object from at least 3 respective viewpoints, such as a CCD camera 10 which operates from 3 different viewpoints, or such as a film camera 20, which may be airborne, associated with a scanner 30 (such as a Zeiss PS-1 which is operative to digitize the images generated by film camera 20.
The at least 3 digital views are fed to a matching point finder 40 which is operative to identify at least 7 and preferably a multiplicity of triplets of matching points from the 3 digital views. "Matching points" from different views are points or locations which correspond to a single location in the real, 3D world. These points may, for example, be identified manually. For aerial photographs, commercially available matching point software may be employed such as the Match-T package mar¬ keted by Inpho in Stuttgart, which performs a correspondence function.
Conventional methods for finding matching points are described in Wang, H. and Brady, J. M, "Corner detection: some new results", IEEE colloquium Digest of Systems Aspects of Machine Perception and vision", Londone 1992, pp. 1.1 - 1.4.
A trilinear tensor computation unit 50 receives the matching point triplets and computes a trilinear tensor representing the geometric relationship between the three views.
The trilinear tensor is then employed to generate a 3D representation of a 3D scene. According to one embodiment of the present invention, the trilinear tensor is employed by an epipo¬ lar geometry generation unit 60 to compute an epipolar geometric representation of two of the three views. Subsequently, 3D representation generation unit 70 generates the 3D representation of the scene or object from the epipolar geometric representation output of unit 60, as described in more detail in: a. Faugeras, 0. D. "What can be seen in 3 dimensions with an uncalibrated stereo rig?", Proceedings of the European Conference on Computer Vision, pages 563 - 578, Santa Margherita Ligure, Italy, June 1992. b. Hartley, R. et al, "Stereo from uncalibrated cameras." Proceedings IEEE Conf. on Computer Vision and Pattern Recognition, 761 - 764, Champaign, IL, June 1992. c. Shashua, A. "Projective structure from uncalibrated images: structure from motion and recognition. IEEE Transactions on PAMI, 16(8) : 778 - 790, 1994.
A preferred implementation of units 50 and 60 is de¬ scribed, in "C" computer language form, in Appendix A. A preferred implementation of unit 50 is described from page 1 of Appendix A until toward the end of page 4 thereof. A preferred implementation of unit 60 is described from the end of page 4 of Appendix A until the middle of page 8. Subroutines and statisti¬ cal procedures which are useful in understanding the above mate¬ rial appear from page 8 of Appendix A onward.
The 3D representation, including 3D information representating at least a portion or an aspect of the 3D scene or object, may be employed by a laser computer printer to generate a new view of the object or scene. Alternatively, conventional CAD (computer aided design) software in conjunction with a conventional plotter may be employed to generate a new view of the object or scene. The CAD software may also be operative to compare 2 CAD files in quality assurance applications.
Reference is now made of Fig. 7 which is a simplified flowchart illustration of a preferred 3D scene reconstruction method, operative in accordance with a preferred embodiment of the present invention, which is useful in conjunction with the apparatus of Fig. 6. Fig. 7 is generally self-explanatory. The 3 views of the scene or object can be digital, if they are accessed from a digital archive or generated, e.g. by a digital camera. If they are not digital, they are scanned or otherwise digitized.
20
SUBSTITUTESHEET( ULE^ Any suitable conventional or other formats may be employed. For example, the Silicon Graphics's Inventor format may initially be employed for the 3D representation. The Inventor format may be converted into Postscript in order to print a new view of the 3D representation.
As shown, the 3D representation of the scene is useful in performing a wide range of activities such as 3D measurements of the scene or object, generation, e.g. as a printout, of a new view of the scene or object, and quality assurance comparisons in which the generated 3D representation of the object or scene is compared to a desired object or scene or a desired 3D representation thereof, using conventional methods.
Reference is now made to Fig. 8 which is a simplified functional block diagram of image transfer apparatus, constructed and operative in accordance with a preferred embodiment of the present invention, which is operative to generate a novel view of a 3D scene from at least two reference views thereof. The apparatus of Fig. 8 is similar to the apparatus of Fig. 6. Howev¬ er, in Fig. 8, a novel view of a 3D scene is directly generated from only at least two reference views thereof, preferably with¬ out generating a 3D representation intermediate.
In Fig. 8, geometric information regarding the two reference views and the desired novel view is employed to gener¬ ate a trilinear tensor representing the geometric relationship between the three views. Typically, at least 7 triplets of loca¬ tions in the novel view may be identified, e.g. manually, which correspond to 7 locations respectively in each of the at least two reference views. Typically, at least some infromation regard¬ ing the novel view is available. For example, if it is desired to update a GIS (geographic information system) year-old view, based on at least two new reference views of the same area. It is typically possible to identify at least 7 locations in the year- old view which correspond to 7 locations in the two reference views and which can be assumed will still exist in the soon-to-be-generated current version of the year-old view.
The novel view is typically generated by computing a multiplicity of novel view locations each corresponding to dif¬ ferent first and second corresponding locations in said first and second reference views respectively based on said first and second corresponding locations and said trilinear tensor. For example, the matching point finder 140 may generate a multiplicity of pairs of matching points from the two reference views, say 1000 such pairs. For the first 7 pairs, a user may manually indicate 7 matching points in the novel view.
Once the trilinear tensor is generated by trilinear tensor computation unit 150, the coordinates of Matching Point Pairs 8 - 1000 may be plugged into the trilinear tensor, as shown in Fig. 9, in order to generate coordinates of matching points 8 - 1000 in the novel view.
The novel view thus generated may, for example, be compared to the same view as seen a year before, in order to identify differences in the scene which took place in the course of the year.
Reference is now made to Fig. 9 which is a simplified flowchart illustration of a preferred image transfer method, operative in accordance with a preferred embodiment of the present invention, which is useful in conjunction with the appa¬ ratus of Fig. 8. Fig. 9 is generally self-explanatory.
In any of the embodiments described herein, if more than 3 views are provided, "intermediate" tensors may be computed for each 3 views. Then, a representative tensor may be computed based on the relationships between these "intermediate" tensors.
Fig. 10 is an illustration of a 3D reconstruction of a shoe which is also illustrated in Fig. 2. Fig. 10 was generated by finding matching points and reconstructing their 3D locations. Next the coordinates are processed by CAD software to generate the surface shown in Fig. 10.
Fig. 11 is a simplified block diagram illustration of a preferred method and apparatus for quality assurance of workpieces. Fig. 11 includes an array 200 of 3 CCD cameras 210 which are aimed at a single location, so as to yield three perspectives of that location. The CCD cameras are attached to a robot arm 212 and therefore can move relative to a workpiece 214 arriving along a conveyor belt 220 in accordance with suitable instructions from a controller 224.
When a workpiece enters the field of view of the CCD cameras 210, the conveyor belt 220 typically pauses to allow substantially the entirety of the surface area of the workpiece to be imaged by the cameras 210. The controller 224, via robot arm 212, moves the camera array 200 around the object such that, typically, almost its entire surface area is imaged. For example, the camera array 200 may be moved through 10 different positions around the object, and at each position, each of the 3 CCD cameras images the workpiece. The number of positions employed depends on the complexity of the workpiece.
This process yields a plurality of image triplets, each image triplet including three digital images of the same portion of the workpiece, from 3 respective perspectives. For each image triplet, the corresponding position of the array 200 and of each of the cameras 210 may be computed, based on the robot arm's location, which is known, and using hand-eye calibration.
Each image triplet is processed by units 240, 250, 260, 270 and 290 which may be similar to units 40, 50, 60, 70 and 90, respectively, of Fig. 6.
The CAD model information generated by CAD S/W 290 from each image triplet (each probe location) is stored in a suitable memory 300. A computation unit 310 is operative to integrate the multiplicity of probe locations corresponding to the multiplicity of positions of CCD camera array 200, into a single coordinate system. The necessary coordinate transformations are computed by inverting the transformations which define the CCD camera array's motion.
A computational unit 320 compares the output of unit 310 to a reference CAD model and computes differences therebetween. These differences are compared, in a computational unit 330, to accepted tolerances.
The apparatus of Fig. 11 is also suitable for 3D digi¬ tization applications for reverse engineering or CAD (computer
23
SUBSUME SHEET (RULE 28) aided design purposes) .
Fig. 12 is a simplified block diagram illustration of a preferred method and apparatus for generating a digital terrain map, e.g. in order to update municipal maps, to detect illegal structures, to serve a car navigation system, or even to map a microscopic object such as an integrated circuit.
An airborne CCD camera 334 is flown over a scene for which it is desired to generate a digital terrain map. The camera 334 generates 3 digital images of the scene from 3 respective perspectives. The 3 images are processed by units 340, 350, 360 and 370 which may be similar to units 40, 50, 60 and 70 of Fig. 6.
A surface interpolation procedure is performed by a surface interpolation unit 380, on the output of 3D representa¬ tion generation unit 370. A suitable surface interpolation method is described in Grimson, W. E. L, "A computational theory of visual surface interpolation", Proc. of the Royal Soc. of London
B, 298, pp. 395 - 427, 1982.
Finally, perspective distortions are removed from the output of surface interpolation unit 380 by an orthophoto genera¬ tion unit 390, resulting in an orthophoto (planimetric map) . A suitable method of operation for unit 390 is described in Slama,
C. C, "Manual of Photogram etry", American Society of Photogram- metry, 1980.
The present invention is also useful for visualization of a 3D object, particularly for entertainment, animation or educational applications. A camera array such as array 200 of Fig. 11 circles around an object to be visualized and images substantially the entire surface area which it is desired to display to a user. For example, the camera array may image the object from each of 200 positions surrounding the object.
Synthetic images may then be generated for positions other than the 200 above-mentioned positions. A desired position may be indicated by a user, e.g. by means of a joystick. The apparatus of the present invention may be used to generate a synthetic image for that position.
24
SUBSTITUTE SHEET (RULE 26} Conventional driving simulation games employ synthetic backgrounds, however, the present invention may be employed to provide a driving simulation game with a real background. To do this, an array of at least 3, and preferably 5 - 10 cameras is moved within a desired scene such that substantially the entirety of the scenes can be captured by the camera array from at least 3 and preferably more different perspectives. For example, the scene may be captured from each of approximately 1000 positions of the camera array. New views are then generated, in accordance with the present invention, in order to accomodate a user's need for new views as indicated, e.g. by a joystick.
Algebraic functions useful for recognition are now described, based on an article entitled "Algebraic functions for recognition", to be published in IEEE, Transactions on PAMI.
Algebraic Functi ns For Recognition
I. INTRODUCTION
We establish a general result about algebraic connections across three perspective views of a 3D scene and demonstrate its application to visual recognition via alignment. We show- that, in general, any three perspective views of a scene satisfy a pair of trilinear functions of image coordinates. In the limiting case, when all three views are orthographic, these functions become linear and reduce to the form discovered by [38]. Using the trilinear result one can manipulate views of an object (such as generate novel views from two model vieλvs) without recovering scene structure (metric or non-metric), camera transformation, or even the epipolar geometry. Moreover, the trilinear functions can be recovered by linear methods with a minimal configuration of seven points. The latter is shown to be new lower bound on the minimal configuration that is required for a general linear solution to the problem of re-projecting a 3D scene onto an arbitrary novel view given corresponding points across two reference views. Previous solutions rely on recovering the epipolar geometry which, in turn, requires a minimal configuration of eight points for a linear solution.
The central results are contained in Theorems 1, 2 and 3. The first theorem states that the variety of views φ of a fixed 3D object obtained by an uncalibrated pin-hole camera satisfy a relation of the sort F(φ, u^ 'ώ_) = 0, where φ\, Φτ are two arbitrary views of the object, and F has a special trilinear form. The coefficients of F can be recovered linearly without establishing first the epipolar geometry, 3D structure of the object, or camera motion. The auxiliary Lemmas required for the proof of Theorem 1 may be of interest on their own as they ..establish certain regularities across projective transformations of the plane and introduce new view invariants (Lemma 4).
Theorem 2 addresses the problem of recovering the coefficients of the trilinear functions in the most economical way. It is shown that among all possible trilinear functions across three views, there exists at most four linearly independent such functions. As a consequence, the coefficients of these functions can be recovered linearly from seven corresponding points across three views.
Theorem 3 is an obvious corollary of Theorem 1 but contains a significant practical aspect. It i . shown that if the views vx . (_'2 are obtained by parallel projection, then F reduces to a special bilinear form — or. equivalently, that any perspective view ψ can be obtained by a rational linear function of two orthographic views. The reduction to a bilinear form implies that simpler recognition schemes are possible if the two reference views (model views) stored in memory are orthographic.
These results may have several applications (discussed in Section VI), but the one empha¬ sized throughout this manuscript is for the task of recognition of 3D objects via alignment. The alignment approach for recognition ([37, 16], and references therein) is based on the result that the equivalence class of views of an object (ignoring self occlusions) undergoing 3D rigid, affine or projective transformations can be captured by storing a 3D model of the object, or simply by storing at least two arbitrary "model" views of the object — assum¬ ing that the correspondence problem between the model views can somehow be solved (cf. [27, 5. 33]). During recognition a small number of corresponding points between the novel input view and the model views of a particular candidate object are sufficient to "re-project" the model onto the novel viewing position. Recognition is achieved if the re-projected image is successfully matched against the input image. We refer to the problem of predicting a novel view from a set of model views using a limited number of corresponding points, as the problem of re-projection.
The problem of re-projection can in principal be dealt with via 3D reconstruction of shape and camera motion. This includes classical structure from motion methods for recovering rigid camera motion parameters and metric shape [36, 18, 35, 14, 15], and more recent methods for recovering non-metric structure, i.e., assuming the objects undergo 3D affine or projective transformations, or equivalently, that the cameras are uncalibrated [17, 25, 39, 10, 13. 30]. The classic approaches for perspective views are known to be unstable under errors in image measurements, narrow field of view, and internal camera calibration [3, 9, 12] , and therefore, are unlikely to be of practical use for purposes of re-projection. The non-metric approaches, as a general concept, have not been fully tested on real images, but the methods proposed so far rely on recovering first the epipolar geometry — a process that is also known to be unstable in the presence of noise.
It is also known that the epipolar geometry alone is sufficient to achieve re-projection by means of intersecting epipolar lines [24, 6, 8, 26, 23, 11] using at least eight corresponding points across the three views. This, however, is possible only if the centers of the three cameras are non-collinear — which can lead to numerical instability unless the centers are far from collinear — and any object point on the tri-focal plane cannot be re-projected as
27 well. Furthermore, as with the non-metric reconstruction methods, obtaining the epipolar geometry is at best a sensitive process even when dozens of corresponding points are used and with the state of the art methods (see Section V for more details and comparative analysis with simulated and real images).
For purposes of stability, therefore, it is worthwhile exploring more direct tools for achiev¬ ing re-projection. For instance, instead of reconstruction of shape and invariants we would like to establish a direct connection between views expressed as a functions of image coor¬ dinates alone — which we call "algebraic functions of views" . Such a result was established in the orthographic case by [38]. There it was shown that any three orthographic views of an object satisfy a linear function of the corresponding image coordinates — this we will show here is simply a limiting case of larger set of algebraic functions, that in general have a trilinear form. With these functions one can manipulate views of an object, such as create new views, without the need to recover shape or camera geometry as an intermediate step — all what is needed is to appropriately combine the image coordinates of two reference views. Also, with these functions, the epipolar geometries are intertwined, leading not only to absence of singularities, and a lower bound on the minimal configuration of points, but as we shall see in the experimental section to more accurate performance in the presence of errors in image measurements. Part of this work (Theorem 1 only) was presented in concise form in [31].
II. NOTATIONS
We consider object space to be the three-dimensional projective space V3, and image space to be the two-dimensional projective space V2. Let Φ C Vs be a set of points standing for a 3D object, and let φ, C V2 denote views (arbitrary), indexed by ι, of Φ. Given two cameras with centers located at 0, 0' _ "P3, respectively, the epipoles are defined to be at the intersection of the line 00' with both image planes. Because the image plane is finite, we can assign, without loss of generality, the value 1 as the third homogeneous coordinate to every observed image point. That is, if (x, y) are the observed image coordinates of some point (with respect to some arbitrary origin — say the geometric center of the image), then p = (x. y. 1) denotes the homogeneous coordinates of the image plane. Note that this convention ignores special views in which a point in Φ is at infinity in those views — these singular cases are not modeled here. Since we will be working with at most three views at a time, we denote the relevant epipoles as follows: let v _ φ\ and v' £ φ2 be the corresponding epipoles between views φ_ φ2, and let ϋ G xi>_ and v" Φ the corresponding epipoles between views φ\, φ_. Likewise, corre¬ sponding image points across three views will be denoted by p = (x, y, l), p' = (x', y', 1) and p" = [x", y", 1). The term "image coordinates" will denote the non-homogeneous coordinate representation of V2, e.g., (x, y), (x', y'), iχ", y") for the three corresponding points.
Planes will be denoted by 7r,, indexed by i, and just π if only one plane is discussed. All planes are assumed to be arbitrary and distinct from one another. The symbol = denotes equality up to a scale, GLn stands for the group of n x n matrices, and PGLn is the group defined up to a scale.
III. THE TRILINEAR FORM
The central results of this manuscript are presented in the following two theorems. The remaining of the section is devoted to the proof of this result and its implications.
Theorem 1 (Trilinearity) Let φ\, φ2, Φz be three arbitrary perspective views of some ob¬ ject, modeled by a set of points in 3D. The image coordinates (x, y) € φ\ , {x', y') _ Φ and (x", y") £ ib3 of three corresponding points across three views satisfy a pair of trilinear equations of the following form:
x"(a x + 2y + 3) + x"x'{a x + a5y + 6) + x'(a7x + a8y + α. ) + awx + auy + 12 = 0,
and
y"(β_x + β2y + β3) + y" '(β4 + βsy + β.) + x 3-x + βsy + β9) + β.0x + βny + βχ = 0,
where the coefficients ctj, βj, j = 1, ..., 12, are fixed for all points, are uniquely defined up to an overall scale, and ctj = βj, j = 1, ..., 6.
The following auxiliary propositions are used as part of the proof.
Lemma 1 (Auxiliary - Existence) Let A € PGL3 be the projective mapping (homogra¬ phy) rb 1 r→ ιi-2 due to some plane π. Let A be scaled to satisfy p'o = Ap0 + v' . where p0 _ φx and p'0 _ _-2 are corresponding points coming from an arbitrary point P0 . -. Then, for any corresponding pair p € φ, and p> € φ2 coming from an arbitrary point P g V we ha
p' ^ Ap + kv'.
The coefficient k is independent of φ , .e., is invariant to the choice of the second vteu.
The lemma, its proof and its theoretical and practical implications are discussed in detail m [2 . 32]. Note that the particular case where the homography A is affine, and the epipole o is on the line at infinity, corresponds to the construction of affine structure from two orthographic views [17]. In a nutshell, a representation H, of V3 (tetrad of coordinates) can always be chosen such that an arbitrary plane * is the plane at infinity. Then, a general uncalibrated camera motion generates representations 71 which can be shown to'be related to H_ by an element of the affine group. Thus, the scalar k is an affine invariant within a projective framework, and is called a relative affine invariant. A ratio of two such invariants, each corresponding to a different reference plane, is a projective invariant [3o] For our purposes, there is no need to discuss the methods for recovering * - all we need is to use the existence of a relative affine invariant k associated with some arbitrary reference plane TΓ which, in turn, gives rise to a homography A.
Definition 1 Homographies A, _ PGL3 from φ. → φx due to the same plane ,, are said to be scale-compatible if they are scaled to satisfy Lemma 1, i.e., for any point P € projecting onto p e φ. and ? 6 i t there exists a scalar k that satisfies
_■ ~ A,p + kv
for any view φt, where v e φ. is the epipole with φ_ (scaled arbitrarily). Lemma 2 (Auxiliary - Uniqueness) Let A, A' e PGL3 be two homographies of Xx „ v_ due to planes π,, π2 ; respectively. Then, there exists a scalar s, that satisfies the equation:
A - sA' = [ v', βv', ~fv'}, for some coefficients a, β, - .
Proof: Let , _ φx be any point in the first view. There exists a scalar sq that satisfies i - Aq - SqA'q. Let H = A - sqA>, and we have Hq 2. „'. But, as shown in [29], Av ≥ _>' tor any homography φx >→ _,2 due to any plane. Therefore, Hv ≥ «,' as well. The mapping of two distinct points q, v onto the same point „' could happen only if H is the homographv due
30
SϋBS ϋJϊ S to the meridian plane (coplanar with the projection center O), thus Hp = v' for all p _ 1 and sq is a fixed scalar s. The latter, in turn, implies that H is a matrix whose columns are multiples of v'. ~\
Lemma 3 (Auxiliary for Lemma 4) Let A, A' £ PGLz be homographies from φx \→ 2 due to distinct planes π ,ιτ2, respectively, and B,B' _ PGL3 be homographies from φ >→ φ3 due to πx,π2, respectively. Then, A' = AT for some T € PGL3, and B = BCTC'1, where Cυ^v.
Proof: Let A = A2 rAι, where AX,A2 are homographies from φι,φ onto π , respectively. Similarly B = B2 lBx, where B_,B2 are homographies from φx,φ3 onto π , respectively. Let Ax - = (cι,C2,c3)τ, and let C = Aϊ1 diag(cx, c2, c3)Ax. Then, B_ = A_C~l , and thus, we have B = B2 1A C~Ϊ. Note that the only difference between A and B is due to the different location of the epipoles v,v, which is compensated by C (Cυ = ϋ). Let Ei G PGL3 be the homography from φx to π2, and E2 £ PGL the homography from π2 to ττ . Then with proper scaling of Ex and E2 we have
A' = A2 1E2EX = AAX 1E2EX = AT,
and with proper scaling of C we have,
B' = B2 lE2ExC~l = BCA-lE2ExC~l = BCTC-1.
D
Lemma 4 (Auxiliary — Uniqueness) For scale-compatible homographies, the scalars _s,α, /3, of Lemma 2 are invariants indexed by φx,π ,π2. That is, given an arbitrary third view ψ3. let B,B' be the homographies from φx ι→ φ3 due to πx,π2, respectively. Let B be scale-compatible with A, and B' be scale-compatible with A1. Then,
B-sB' = [av",βv",7v").
Proof: We show first that 5 is invariant, i.e., that B — sB' is a matrix whose columns are multiples of v". From Lemma 2, and Lemma 3 there exists a matrix H, whose columns are multiples of v', a matrix T that satisfies A' = AT, and a scalar s such that I — sT = A~lH. After multiplying both sides by BC, and then pre-multiplying by C~l we obtain
B - sBCTC-' = BCA-λHC~ . From Lemma 3, we have B' = BCTC~ The matrix A^ H has columns which are multiples ot v (because A V S. y), CA~lH is a matrix whose columns are multiple of ϋ, and BCA~lH is a matrix whose columns are multiples of v". Pre-multiplying BCA~lH by C"1 does not change its form because every column of BCA→HC^ is simply a linear combination of the columns of BCA^H. As a result, B - sB> is a matrix whose columns are multiples of v". Let H = A s A' and H = B sB'. Since the homographies are scale compatible, we have from Lemma 1 the existence of invariants k, k' associated with an arbitrary p _ φ where k is due to TTJ , and k' is due to τr2: p' ≥ Ap + kv' ≥. A'p + k'v' and p" S_ Bp + kv" ^ B'p + k'v". Then from Lemma 2 we have Hp = (sk' - k)v> and Hp = (sk' - k)v" . Since p is arbitrary this could happen only if the coefficients of the multiples of v' in H and the coefficients of the multiples of v" in H, coincide. ~\
LemmVf SΗ ^ L?nm I pr°VideS, the existence Part of theorem, as follows. Since
Figure imgf000034_0001
(ι'os-α,)r P (y'o,-o_)rp (x ___y _, )7-p . (j)
where bx, b2, b3 and ax , a2, a3 are the row vectors of A and B and υ' = (v[ . v' . v' ) v" = W, t _\ ι _"). Because of the invariance of k we can equate terms of (1) with terms of (2) and obtain trilinear functions of image coordinates across three views. For example, by equating the first two terms in each of the equations, we obtain:
x ;&3 - vi' a )τp - x"x v_b_ - vi' a3)τp + x'(ι _&ι - v'(a3)τp - ( _•;&! - v'lax)τp = 0, (3)
In a similar fashion, after equating the first term of (1) with the second term of (2), we obtain:
y"(v[b3 - v3''ax)τp - y"x'(v3' b3 - υ^' a3)τ P + x'(v3' b2 - vi' a3)τp - (v[b2 - υ~a_)τp = 0.
(4)
Both equations are of the desired form, with the first six coefficients identical across both equations. The question of uniqueness arises because Lemma 1 holds for any plane. If we choose a different plane, say π2, with homographies A', B', then we must show that the new homo¬ graphies give rise to the same coefficients (up to an overall scale). The parenthesized terms in (3) and (4) have the general form: τ>'6_, — v"at, for some i and j . Thus, we need to show that there exists a scalar s that satisfies (α. - ._;) = (6, - ^).
This, however, follows directly from Lemmas 2 and 4. _~
The direct implication of the theorem is that one can generate a novel view (φ3) by simply combining two model views (φ , φ2)- The coefficients cx_. and β3 of the combination can be recovered together as a solution of a linear system of 17 equations (24 — 6 — 1 ) given nine corresponding points across the three views (more than nine points can be used for a least-squares solution).
In the next theorem we obtain the lower bound on the number of points required for solving for the coefficients of the trilinear functions. The existence part of the proof of Theorem 1 indicates that there exists nine trilinear functions of that type, with coefficients having the general form v b — v"at. Thus, we have at most 27 distinct coefficients (up to a uniform scale), and thus, if more than two of the nine trilinear functions are linearly independent, we may solve for the coefficients using less than nine points. The next theorem shows that at most four of the trilinear functions are linearly independent and consequently seven points are sufficient to solve for the coefficients.
Theorem 2 There exists nine distinct trilinear forms of the type described in Theorem 1. of which at most four are linearly independent. The coefficients of the four trilinear forms can be recovered linearly with seven corresponding points across the three views.
Proof: The existence of nine trilinear forms follow directly from (1) and (2). Let aX] = υ[b3 — υ"at. The nine forms are given below (the first two are (3) and (4) repeated for convenience):
x"cx 3p - x"x'ocJ3p +
Figure imgf000035_0001
- a lP = 0, y"ct 3p - y"x'a_~ 3p + x'αJ2P - a 2p = 0, x"a 3p - x"y'o 3p + y'a^p - α^p = 0, (ό) y"a]3p - y"y'θL3 T 3p + y'o 3 τ 2p - ot]_p = 0. ( 6)
SUBSπTUlfSHEET (RULE 26) y"x'a3lP - x"x'a32p + x" 2p - y"a[_p = 0, (7) y"y'<χJιP - χ"y'az-2P + χ"cχ 22p - y"a _p = o, (s) x"y'a 3p - x"x'a23p + x'a^p - y'a~_p = 0, (9) y'V «i3P - y"χ'a23p + χ'<χlτP - y'auP = o, '. ( 10) x"_/,α«P - "x' 3" 2p + y"x'<*21p - y"y'a _p = 0, (H)
For a given triplet p, p',p" the first four functions on the list produce a 4 x 27 matrix. The rank of the matrix is four because it contains four orthogonal columns (columns associated with απ, c.12, α2i and 22), therefore these functions are linearly independent. Since we have 27 coefficients, and each triplet p, p', p" contributes four linear equations, then seven corresponding points across the three views provide a sufficient number of equations for a linear solution for the coefficients (given that the system is determined up to a common scale, seven points produce two extra equations which can be used for consistency checking or for obtaining a least squares solution).
The remaining trilinear forms are linearly spanned by the first four, as follows:
(7) = y"(3) - x"(4)
(8) = y"(5) - x"(6)
(9) = y'(3) - x'(5)
(10) = y'(4) - x'(6)
(11)
Figure imgf000036_0001
where the numbers in parenthesis represent the equation numbers of the various trilinear functions. ~
Taken together, both theorems provide a constructive means for solving for the positions x" , y" in a novel view given the correspondences p, p' across two model views. This process of generating a novel view can be easily accomplished without the need to explicitly recover structure, camera transformation, or even just the epipolar geometry — and requires fewer corresponding points than any other known alternative.
The solution for x", y" is unique without constraints on the allowed camera transforma¬ tions. There are, however, certain camera configurations that require a different set of four trilinear functions from the one suggested in the proof of Theorem 2. For example, the set of equations (5), (6), (9) and (10) are also linearly independent. Thus, for example, in case (■', and 1-3 vanish simultaneously, i.e., v' = (0, 1. 0). then that set should be used instead. Similarly, equations (3), (4). (9) and (10) are linearly independent, and should be used in case v' ≥ (1, 0, 0). Similar situations arise with v" ≥ ( 1, 0, 0) and v" ≥ (0. 1, 0) which can be dealt by choosing the appropriate basis of four functions from the six discussed above. Note that we have not addressed the problem of singular configurations of seven points. For example, its clear that if the seven points are coplanar, then their correspondences across the three views could not possibly yield a unique solution to the problem of recovering the coefficients. The matter of singular surfaces has been studied for the eight-point case nec¬ essary for recovering the epipolar geometry [19, 14, 22]. The same matter concerning the results presented in this manuscript is an open problem.
Moving away from the need to recover the epipolar geometry carries distinct and significant advantages. To get a better idea of these advantages, we consider briefly the process of re¬ projection using epipolar geometry. The epipolar intersection method can be described succinctly (see [11]) as follows. Let F 3 and E23 be the matrices ("fundamental" matrices in recent terminology [10]) that satisfy p"Fx3p = 0, and p"F23p' = 0. Then, by incidence of p" with its epipolar line, we have:
Figure imgf000037_0001
Therefore, eight corresponding points across the three views are sufficient for a linear solution of the two fundamental matrices, and then all other object points can be re-projected onto the third view. Equation (12) is also a trilinear form, but not of the type introduced in Theorem 1. The differences include (i) epipolar intersection requires the correspondences coming from eight points, rather than seven, (ii) the position of p" is solved by a line intersection process which is singular in the case the three camera centers are collinear; in the trilinearity result the components of p" are solved separately and the situation of three collinear cameras is admissible, (iii) the epipolar intersection process is decomposable, i.e., only two views are used at a time; whereas the epipolar geometries in the trilinearity result are intertwined and are not recoverable separately. The latter implies a better numerically behaved method in the presence of noise as well, and as will be shown later, the performance, even using the minimal number of required points, far exceeds the performance of epipolar intersection using many more points. In other words, by avoiding the need to recover the epipolar geometry we obtain a significant practical advantage as well, since the epipolar geometry is the most error-sensitive component when working with perspective views. The connection between tl^ general result of trilinear functions of views and the "linear combination of views" result [38] for orthographic views, can easily be seen by setting A and B to be affine in V2, and v3 = v3 = 0. For example, (3) reduces to v,x — vx x + υ_ a\ — υ_b\) p = 0, which is of the form a x" + 2x' + x + 4y + a_ — 0.
As in the perspective case, each point contributes four equations, but here there is no ad¬ vantage for using all four of them to recover the coefficients, therefore we may use only two out of the four equations, and require four corresponding points to recover the coefficients. Thus, in the case where all three views are orthographic, x" (y") is expressed as a linear combination of image coordinates of the two other views — as discovered by [38].
IN. THE BILINEAR FORM
Consider the case for which the two reference (model) views of an object are taken or- thographically (using a tele lens would provide a reasonable approximation), but during recognition any perspective view of the object is allowed. It can easily be shown that the three views are then connected via bilinear functions (instead of trilinear):
Theorem 3 (Bilinearity) Within the conditions of Theorem 1, in case the views Φ and w are obtained by parallel projection, then the pair of trilinear forms of Theorem 1 reduce to the following pair of bilinear equations: x"(axx + 2y + 3) + a4x"x' + 5x' + 6x + 7y + _ = 0, and y"{βxx + β2y + β3) + β4y"x + βsχ' + β6x + βry + = 0. where α, = J, . ' = 1, ..., 4.
Proof: Under these conditions we have from Lemma 1 that A is affine in V2 and v3' = 0, therefore (3) reduces to: ;63 - v" ai)Tp + v^' x"x' - __' + «αι - v[bx)τp = 0.
Similarly. (4) reduces to: y"(υ[b3 - v a_)τp + υi' x' - .£_' + {v{' a_ - v', b2)T P = 0.
36 RULE 26 Both equations are of the desired form, with the first four coefficients identical across both equations. ~\
The remaining trilinear forms undergo a similar reduction, and Theorem 2 still holds, i.e., we still have four linearly independent bilinear forms. Consequently, we have 21 coefficients up to a common scale (instead of 27) and four equations per point, thus five corresponding points (instead of seven) are sufficient for a linear solution.
A bilinear function of three views has two advantages over the general trilinear function. First, as mentioned above, only five corresponding points (instead of seven) across three views are required for solving for the coefficients. Second, the lower the degree of the algebraic function, the less sensitive the solution may be in the presence of errors in measuring correspondences. In other words, it is likely (though not necessary) that the higher order terms, such as the term x"x'x in Equation 3, will have a higher contribution to the overall error sensitivity of the system.
Compared to the case when all views are assumed orthographic, this case is much less of an approximation. Since the model views are taken only once, it is not unreasonable to require that they be taken in a special way, namely, with a tele lens (assuming we are dealing with object recognition, rather than scene recognition). If this requirement is satisfied, then the recognition task is general since we allow any perspective view to be taken during the recognition process.
V. EXPERIMENTAL DATA
The experiments described in this section were done in order to evaluate the practical aspect of using the trilinear result for re-projection compared to using epipolar intersection and the linear combination result of [38] (the latter we have shown is simply a limiting case of the trilinear result).
The epipolar intersection method was implemented as described in Section III by recover¬ ing first the fundamental matrices. Although eight corresponding points are sufficient for a linear solution, in practice one would use more than eight points for recovering the fundamen¬ tal matrices in a linear or non-linear squares method. Since linear least squares methods are still sensitive to image noise, we used the implementation of a non-linear method described in [20] which was kindly provided by T. Luong and L. Quan (these were two implementations of the method proposed in [20] — in each case, the implementation that provided the better results was adopted).
The first experiment is with simulation data showing that even when the epipolar geometry is recovered accurately, it is still significantly better to use the trilinear result which avoids the process of line intersection. The second experiment is done on a real set of images, comparing the performance of the various methods and the number of corresponding points that are needed in practice to achieve reasonable re-projection results.
. Computer Simulations
We used an object of 46 points placed randomly with z coordinates between 100 units and 120 units, and x, y coordinates ranging randomly between -125 and +125. Focal length was of 50 units and the first view was obtained by fx/z, fy/ . The second view (φ,2) was generated by a rotation around the point (0, 0, 100) with axis (0.14, 0.7, 0.7) and by an angle of 0.3 radians. The third view (φ3) was generated by a rotation around an axis (0, 1, 0) with the same translation and angle. Various amounts of random noise was applied to all points that were to be re-projected onto a third view, but not to the eight or seven points that were used for recovering the parameters (fundamental matrices, or trilinear coefficients). The noise was random, added separately to each coordinate and with varying levels from 0.5 to 2.5 pixel error. We have done 1000 trials as follows: 20 random objects were created, and for each degree of error the simulation was ran 10 times per object. We collected the maximal re-projection error (in pixels) and the average re-projection error (averaged of all the points that were re-projected). averaging over all trials
Figure imgf000040_0001
no error were added to the eight or seven points that were used to determine the epipolar geometry and the trilinear coefficients, we simply solved the associated linear systems of equations required to obtain the fundamental matrices or the trilinear coefficients.
The results are shown in Figure 1A-1B. The graph on the left shows the performance of both algorithms for each level of image noise by measuring the maximal re-projection error. We see that under all noise levels, the trilinear method is significantly better and also has a smaller standard deviation. Similarly for the average re-projection error shown in the graph on the right.
This difference in performance is expected, as the trilinear method takes all three views together, rather than every pair separately, and thus avoids line intersections. B. Experiments On Real Images
Figure 2A-2C shows three views of the object we selected for the experiment. The object is a sports shoe with added texture to facilitate the correspondence process. This object was chosen because of its complexity, i.e., it has a shape of a natural object and cannot easilv be described parametrically (as a collection of planes or algebraic surfaces). Note that the situation depicted here is challenging because the re-projected view is not in-between the two model views, i.e.. one should expect a larger sensitivity to image noise than in-between situations. A set of 34 points were manually selected on one of the frames. φ , and their correspondences were automatically obtained along all other frames used in this experiment. The correspondence process is based on an implementation of a coarse-to-fine optical-flow algorithm described in [7]. To achieve accurate correspondences across distant views, in¬ termediate in-between frames were taken and the displacements across consecutive frames were added. The overall displacement field was then used to push ( "warp") the first frame towards the target frame and thus create a synthetic image. Optical-flow was applied again between the synthetic frame and the target frame and the resulting displacement was added to the overall displacement obtained earlier. This process provides a dense displacement field which is then sampled to obtain the correspondences of the 34 points initially chosen in the first frame. The results of this process are shown in Figure 2A-2C by displaying squares centered around the computed locations of the corresponding points. One can see that the correspondences obtained in this manner are reasonable, and in most cases to sub-pixel ac¬ curacy. One can readily automate further this process by selecting points in the first frame for which the Hessian matrix of spatial derivatives is well conditioned — similar to the con¬ fidence values suggested in the implementations of [4, 7, 34] — however, the intention here was not so much as to build a complete system but to test the performance of the trilinear re-projection method and compare it to the performance of epipolar intersection and the linear combination methods.
The trilinear method requires at least seven corresponding points across the three views (we need 26 equation, and seven points provide 28 equations), whereas epipolar intersection can be done (in principle) with eight points. The question we are about to address is what is t he number of points that are required in practice (due to errors in correspondence, lens distortions and other effects that are not adequately modeled by the pin-hole camera model ) to achieve reasonable performance? The trilinear result was first applied with the minimal number of points (seven) for solving for the coefficients, and then applied with 8,9, and 10 points using a linear least-squares solution (note that in general, better solutions may be obtained by using SVD or Jacobi methods instead of linear least-squares, but that was not attempted here). The results are shown in Figure 3A-3B. Seven points provide a re-projection with maximal error of 3.3 pixels and average error of 0.98 pixels. The solution using 10 points provided an improvement with maximal error of 1.44 and average error of 0.44 pixels. The performance using eight and nine points was reasonably in-between the performances above. Using more points did not improve significantly the results; for example, when all 34 points were used the maximal error went down to 1.14 pixels and average error stayed at 0.42 pixels.
Next the epipolar intersection method was applied. We used two methods for recovering the fundamental matrices. One method is by using the implementation of [20], and the other is by taking advantage that four of the corresponding points are coming from a plane (the ground plane). In the former case, much more than eight points were required in order to achieve reasonable results. For example, when using all the 34 points, the maximal error was 43.4 pixels and the average error was 9.58 pixels. In the latter case, we recovered first the homography B due to the ground plane and then the epipole v" using two additional points (those on the film cartridges). It is then known (see [28, 21 , 32]) that E13 = [υ"]E, where [v" is the anti-symmetric matrix of v". A similar procedure was used to recover E23' Therefore, only six points were used for re-projection, but nevertheless, the results were slightly better: maximal error of 25.7 pixels and average error of 7.7 pixels. Figure 4A-4B shows these results.
Finally, we tested the performance of re-projection using the linear combination method. Since the linear combination method holds only for orthographic views, we are actually testing the orthographic assumption under a perspective situation, or in other words, whether the higher (bilinear and trilinear) order terms of the trilinear equations are significant or not. The linear combination method requires at least four corresponding points across the three views. We applied the method with four, 10 (for comparison with the trilinear case shown in Figure 3A-3B), and all 34 points (the latter two using linear least squares). The results are displayed in Figure 5A-5C. The performance in all cases are significantly poorer than when using the trilinear functions, but better than the epipolar intersection method. VI. DISCUSSION
We have seen that any view of a fixed 3D object can be expressed as a trilinear function with two reference views in the general case, or as a bilinear function when the reference views are created by means of parallel projection. These functions provide alternative, much simpler, means for manipulating views of a scene than other methods. Moreover, thev require fewer corresponding points in theory, and much fewer in practice. Experimental results show that the trilinear functions are also useful in practice yielding performance that is significantly better than epipolar intersection or the linear combination method.
In general two views admit a "fundamental" matrix (cf. [10]) representing the epipolar geometry between the two views, and whose elements are subject to a cubic constraint (rank of the matrix is 2). The trilinearity results (Theorems 1,2) imply that three views admit a tensor with 27 distinct elements. We have seen that the tensor does not fail in cases where the epipolar constraint fails, such as when the three cameras are along a straight line (not an uncommon situation). The issue of singular configurations of 7 points (besides the obvious singular configuration of 7 coplanar points) was not addressed in this manuscript. However, the robustness of the re-projection results may indicate that either such configurations are very rare or do not exist. It would be, thus, important to investigate this issue as it is widely believed that the numerical instability of the epipolar constraint lies in the existence of such critical surfaces. The notion of the "trilinear" tensor, its properties, relation to the geometry of three views, and applications to 3D reconstruction from multiple views, constitutes an important future direction.
The application that was emphasized throughout the manuscript is visual recognition via alignment. Reasonable performance was obtained with the minimal number of required points (seven) with the novel view (φ3) — which may be too many if the image to model matching is done by trying all possible combinations of point matches. The existence of bilinear functions in the special case where the model is orthographic, but the novel view is perspective, is more encouraging from the standpoint of counting points. Here we have the result that only five corresponding points are required to obtain recognition of perspective views (provided we can satisfy the requirement that the model is orthographic). We have not experimented with bilinear functions to see how many points would be needed in practice, but plan to do that in the future. Because of their simplicity, one may speculate that these algebraic functions will find uses in tasks other than visual recognition — some of those are discussed below.
There may exist other applications where simplicity is of major importance, whereas the number of points is less of a concern. Consider for example, the application of model- based compression. With the trilinear functions we need 17 parameters to represent a view as a function of two reference views in full correspondence (recall, 27 coefficients were used in order to reduce the number of corresponding points from nine to seven). Assume both the sender and the receiver have the two reference views and apply the same algorithm for obtaining correspondences between the two views. To send a third view (ignoring problems of self occlusions that may be dealt with separately) the sender can solve for the 17 parameters using many points, but eventually send only the 17 parameters. The receiver then simply combines the two reference views in a "trilinear way" given the received parameters. This is clearly a domain where the number of points is not a major concern, whereas simplicity, and robustness (as shown above) due to the short-cut in the computations, is of great importance. Related to image coding, an approach of image decomposition into "layers" was recently proposed by [1, 2]. In this approach, a sequence of views is divided up into regions, whose motion of each is described approximately by a 2D affine transformation. The sender sends the first image followed only by the six affine parameters for each region for each subsequent frame. The use of algebraic functions of views can potentially make this approach more powerful because instead of dividing up the scene into planes one can attempt to divide the scene into objects, each carries the 17 parameters describing its displacement onto the subsequent frame.
Another area of application may be in computer graphics. Re-projection techniques pro¬ vide a short-cut for image rendering. Given two fully rendered views of some 3D object, other views (again ignoring self-occlusions) can be rendered by simply "combining" the reference views. Again, the number of corresponding points is less of a concern here.
REFERENCES
[1] E.H. Adelson. Layered representations for image coding. Technical Report 181, Media Laboratory, Massachusetts Institute of Technology, 1991.
[2] E.H. Adelson and J.Y.A. Wang. Layered representation for motion analysis. In Proceed¬ ings IEEE Conf. on Computer Vision and Pattern Recognition, pages 361-366, New York. XV, June 1993. [3] G Adiv. Inherent ambiguities in recovering 3D motion and structure from a noisy flow field. IEEE Transactions on Pattern Analysis and Machine Intelligence. PAMI-
11(5):477-4S9, 1989. [4] P. Anandan. A unified perspective on computational techniques for the measurement of visual motion. In Proceedings Image Understanding Workshop, pages 219-230, Los
Angeles, CA, February 1987. Morgan Kaufmann, San Mateo, CA. [5] LA. Bachelder and S. Ullman. Contour matching using local affine transformations. In
Proceedings Image Understanding Workshop. Morgan Kaufmann, San Mateo, CA, 1992. [6] E.B. Barrett, M.H. Brill, N.N. Haag, and P.M. Payton. Invariant linear methods in photogrammetry and model-matching. In J.L. Mundy and A. Zisserman. editors. .4p- phcations of invariances in computer vision. MIT Press, 1992. [7] J.R. Bergen and R. Hingorani. Hierarchical motion-based frame rate conversion. Tech¬ nical report, David Sarnoff Research Center, 1990. [S] S. Demey, A. Zisserman, and P. Beardsley. Affine and projective structure from motion.
In Proceedings of the British Machine Vision Conference, October 1992. [9] R. Dutta and M.A. Synder. Robustness of correspondence based structure from motion.
In Proceedings of the International Conference on Computer Vision, pages 106-110,
Osaka, Japan, December 1990. [10] O.D. Faugeras. What can be seen in three dimensions with an uncalibrated stereo rig?
In Proceedings of the European Conference on Computer Vision, pages 563-57S. Santa
Margherita Ligure. Italy. June 1992. [11] O.D. Faugeras and L. Robert. What can two images tell us about a third one? Technical
Report INRIA, France. 1993. [12] W.E.L. Grimson. Why stereo vision is not always about 3D reconstruction. A.I. Memo
Xo. 1435, Artificial Intelligence Laboratory, Massachusetts Institute of Technology. July
1993. [13] R. Hartley, R. Gupta, and T. Chang. Stereo from uncalibrated cameras. In Proceedings
IEEE Conf. on Computer Vision and Pattern Recognition, pages 761-764, Champaign,
IL., June 1992. [14] B.K.P. Horn. Relative orientation. International Journal of Computer Vision, 4:59-78,
1990. [15] B.K.P. Horn. Relative orientation revisited. Journal of the Optical Society of America, WO 96/34365 __.-_,-,
PCT/US96/05697
8:1630-1638, 1991. [16] D.P. Huttenlocher and S. Ullman. Recognizing solid objects by alignment with an image.
International Journal of Computer Vision, 5(2):195— 212, 1990. [17] J.J. Koenderink and A.J. Van Doom. Affine structure from motion. Journal of the
Optical Society of America, 8:377-3S5, 1991. [IS] H.C. Longuet-Higgins. A computer algorithm for reconstructing a scene from two pro¬ jections. Nature, 293:133-135, 1981. [19] H.C. Longuet-Higgins. the reconstruction of a scene from two projections - configura¬ tions that defeat the 8-point algorithm. In Proc. of the 1st Conf. Al applications, pages 395-397, Denver, December 1984. [20] Q.T. Luong, R. Deriche, O.D. Faugeras, and T. Papadopoulo. On determining the fundamental matrix: Analysis of different methods and experimental results. Technical Report INRIA, France, 1993. [21] Q.T. Luong and T. Vieville. Canonical representations for the geometries of multiple projective views. Technical Report INRIA, France, fall 1993. [22] S.J. Maybank. The projective geometry of ambiguous surfaces. Proceedings of the Royal
Society of London, 332:1-47, 1990. [23] J. Mundy and A. Zisserman. Appendix — projective geometry for machine vision. In J. Mundy and A. Zisserman, editors, Geometric invariances in computer vision. MIT Press, Cambridge, 1992. [24] J.L. Mundy, R.P. Welty, M.H. Brill, P.M. Payton, and E.B. Barrett. 3-D model align¬ ment without computing pose. In Proceedings Image Understanding Workshop, pages 727-735. Morgan Kaufmann, San Mateo, CA, January 1992. [25] A. Shashua. Correspondence and affine shape from two orthographic views: Motion and Recognition. A.I. Memo No. 1327, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, December 1991. [26] A. Shashua. Geometry and Photometry in 3D visual recognition. PhD thesis. M.I.T
Artificial Intelligence Laboratory, AI-TR-1401, November 1992. [27] A. Shashua. Illumination and view position in 3D visual recognition. In S.J. Hanson J.E. Moody and R.P. Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 404-411. San Mateo, CA: Morgan Kaufmann Publishers. 1992. Pro¬ ceedings of the fourth annual conference NIPS, Dec. 1991 , Denver, CO.
44 [2S] A. Shashua. On geometric and algebraic aspects of 3D affine and projective structures from perspective 2D views. In The 2nd European Workshop on Invariants. Ponta Dela- gada. Azores, October 1993. Also in MIT Al memo No. 1405, July 1993.
[29] A. Shashua. Projective depth: A geometric invariant for 3D reconstruction from two perspective/orthographic views and for visual recognition. In Proceedings of the Inter¬ national Conference on Computer Vision, pages 583-590, Berlin, Germany. May 1993.
[30] A. Shashua. Projective structure from uncalibrated images: structure from motion and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1994. in press.
[31] A. Shashua. Trilinearity in visual recognition by alignment. In Proceedings of the Euro¬ pean Conference on Computer Vision, Stockholm, Sweden, May 1994.
[32] A. Shashua and N. Navab. Relative affine structure: Theory and application to 3d reconstruction from perspective views. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition, Seattle, Washington, 1994.
[33] A. Shashua and S. Toelg. The quadric reference surface: Applications in registering views of complex 3d objects. In Proceedings of the European Conference on Computer Vision, Stockholm, Sweden, May 1994.
[34] C. Tomasi and T. Kanade. Factoring image sequences into shape and motion. In IEEE Workshop on Visual Motion, pages 21-29, Princeton, NJ, September 1991.
[35] R.Y. Tsai and T.S. Huang. Uniqueness and estimation of three-dimensional motion parameters of rigid objects with curved surface. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-6: 13-26, 1984.
[36] S. Ullman. The Interpretation of Visual Motion. MIT Press, Cambridge and London, 1979.
[37] S. Ullman. Aligning pictorial descriptions: an approach to object recognition. Cognition, 32:193-254, 1989. Also: in MIT Al Memo 931, Dec. 19S6.
[3S] S. Ullman and R. Basri. Recognition by linear combination of models. IEEE Transac¬ tions on Pattern Analysis and Machine Intelligence, PAMI-13:992 — 1006. 1991. Also in M.I.T Al Memo 1052, 1989.
[39] D. Weinshall. Model based invariants for 3-D vision. International Journal of Computer Vision, 10( l ):27-42, 1993. It is appreciated that some or all of the image proc¬ essing operations shown and described herein may be implemented in software. Alternatively, some or all of the image processing operations shown and described herein may be implemented in ROM (read-only memory) form. The software components may, alterna¬ tively, be implemented in hardware, if desired, using convention¬ al techniques.
The present invention has very broad applications and specifically is applicable in all fields in which 3D from 2D techniques are known to be useful. Applications of the present invention include at least the following: photogrammetry applica¬ tions comprising map making from aerial and satellite photographs and coordinate measurements in aerospace and shipyard assembly plants, coordinate measurements of industrial parts (CMM) , automated optical based inspection of industrial parts, robotic cell alignment, robotic trajectory identification, 3D robotic feedback, 3D modelling of scenes, 3D modelling of objects, re¬ verse engineering, and 3D digitizing.
Appendix B is a listing of a preferred software imple¬ mentation of 3D reconstruction apparatus constructed and opera¬ tive in accordance with a preferred embodiment of the present invention. To implement the 3D reconstruction apparatus based on Appendix B, Maple2 software, commercially available from Math- Soft, may be employed in conjunction with an IBM PCT or SUN workstation. The following procedure may be employed: a. Programs generated from the listing of Appendix B may be loaded into Maple2 using the OPEN FILE command. b. The subroutines may be run by bringing the cursor to the row on which appears the WITH (LINALG) command. c. The RETURN key is then processed until the cursor reaches the end of the file. d. The cursor is then returned to the beginning of the file. e. The simulation is run by pressing RETURN until the following row is reached, inclusive:
EVALN (PCOMP) The system then prints on the screen a comparison between the given 3D world and the reconstructed 3D world.
It is appreciated that the listing of Appendix B is also useful in implementing the image transfer apparatus and method shown and described herein. The image transfer embodiment may, for example, be based on the listing of Appendix B used in conjunction with Equations 3 and 4 of the above section entitled Algebraic Function for Recognition.
It is appreciated that the particular embodiments de¬ scribed in the Appendices is intended only to provide an extreme¬ ly detailed disclosure of the present invention and is not in¬ tended to be limiting.
It is appreciated that various features of the inven¬ tion which are, for clarity, described in the contexts of sepa¬ rate embodiments may also be provided in combination in a single embodiment. Conversely, various features of the invention which are, for brevity, described in the context of a single embodiment may also be provided separately or in any suitable subcombina- tion.
It will be appreciated by persons skilled in the art that the present invention is not limited to what has been par¬ ticularly shown and described hereinabove. Rather, the scope of the present invention is defined only by the claims that follow:
APPENDIX A
48 BOOL ComputeAllTensors( IN int nTrials,
TENS Tensors [MAXJFRAMES] [MAX_FRAMES]
[MAX_FRAMES], OUT DMAT3 E_basis[_vIAX_FRAMES]
[MAX_FRAMES][4], OUT int *size_E_basis)
{ int rf, bf, i,j, k, 1, m, n; int num_frames_for_i; int legalFrame[MAX_FRAMES];
double **rank4mat, *w, **v_tr; double thresh.4; BOOL foundThresh; double sum;
int Mummy; /* for the call of findTensor */
/* check the the number of frames is not too large */ if (NumberOfFrames > MAX_FRAMES) { return(FALSE); }
/* loop on the reference frame */ for (rf=0; rf<NumberOfFrames; rf++) { /* consider only marked frame */ if (((specifιed_rf < 0) || (rf = specifιed_rf)) && Images_Flags[rf]) { /* loop on the i'th frame */ for (i=0; i<NumberOfFrames; i++) { /* consider only marked frame different of rf */ (Images_Flags[i] && (i != rf)) {
/* initialize num_frames_for_i */ (*size_E_basis) = 0; num_frames_for_i = 0;
/* loop on the base frame */ /*
* There is dependence between
* Tensors[rf][i][bf] and Tensors [rf][bf][i]
* so we avoid the computation of Tensors [rf][i][bf]
* for bf<i */ for (bf=(i+l); bf<NumberOfFrames; bf++) { /* consider only marked frame different of rf */ if (Images_Flags[bfj && (bf != rf)) { fιndTensor(Tensors[rf][i][bf], rf, i, bf, 0, dummy, 0, nTrials); legalFrame[num_frames_for_i] = bf; num_frames_for_i++;
}
} for (bfM); bf<i; bf++) { if (Images_Flags[bf] && (bf != rf)) { for (j=0; j<3; j++) { for (k=0; k<3; k++) { for (l=0; K3; l++) {
Tensors[rf][i][bf]D][k][l] = Tensors[rf][bf][i][k]Q][l]; } }
} legalFrame[num_frames_for_i] = bf;
50
SUBSTITUTE SHEET (RULE 2^ num_frames_for_i++; }
} if (num_frames_for_i > 1) { /* there are num_frames_for_i Tensors for these */ /* rf and i and when ordering the E0,E1,E2 to a */ /* 9-by-(3*num_frames_for_i) matrix its rank=4 */ /* Note that E[j] = Tensors[rf][i][bf][*][j][*] */
(*size_E_basis) = 4;
/* if num_frames_for_i>2 we consider the transposed */ /* matrix for the svdcmp which requires rows >= cols */
if (num_frames_for_i = 2) { rank4mat = dmatrix( 1 ,9, 1 ,6); w = dvector(l,6); v_tr = dmatrix( 1 ,6, 1 ,6); for (bf=0; bf<num_frames_for_i; bf++) { for (j=0; j<3; j++) { for (k=0; k<3; k++) { for (1=0; 1<3 ; _++) { rank4mat[l+3*k+l][l+ 3*bf+j] = Tensors[rf][i][legalFrame[bf]][k][j][l];
}
}
}
} if (!svdcmp(rank4mat, 9, 6, w, v_tr)) { free_dmatrix(rank4mat, 1 ,9, 1 ,6); free_dvector(w, 1 ,6); free_dmatrix(v_tr, 1 ,6, 1 ,6); return(FALSE);
51
TITUTE SHEET RULE 2β }
/* look for threshold for the largest 4 values */ foundThresh = FALSE; thresh4 = 0.02; while (! foundThresh) { n = 0; for (k=l; k<=6; k++) { if(w[k] > thresh4) { n++;
}
} if (n>4) { thresh4*=2;
} else if(n<4) { thresh4/=2;
} else { foundThresh = TRUE;
} }
/* the 4 columns Of rank4mat which correspond to */ /* the 4 largest singular values form E_basis */ /* Note that E_basis is [4] [9] i.e. goes by rows */ n = 0; for (m=l; m<=6; m++) { if(w[m] > thresh4) { for (k=0; k<3; k++) { for (1=0; 1<3; 1++) { E_basis[rf][i][n][k][l] = rank4mat[l+3*k+l][m]; } } n-
} }
if (!ImproveE_basis(E_basis[rf][i])) { free_dmatrix(rank4mat, 1 ,9, 1 ,6); free_dvector(w, 1 ,6); free_dmatrix(v_tr, 1 ,6, 1 ,6); return(FALSE);
}
/* improve the tensors with the svd results */ for (bf=0; bf<num_frames_for_i; bf++) { forG=0;j<3;j++){ for(k=0;k<3;k++){ for(l=0;K3;l++){ sum = 0.0; n = 0; for (m=l; m<=6; m++) { if(w[m]>thresh4) { sum += E_basis[rf][i][n][k][l]* w[m]*v_tr[l+ 3*bf+j][m]; n++;
}
}
Tensors[rfJ[i][legalFrame[bf]][k][j][l] = sum;
} } } }
53
SUBSTITUTE SHEET {RULE 26) free_dmatrix(rank4mat, 1 ,9, 1 ,6); free_dvector(w, 1 ,6); free_dmatrix(v_tr, 1,6,1,6);
else { rank4mat = dmatrix( 1 ,3 *num_frames_for_i, 1 ,9); w = dvector(l,9); v_tr = dmatrix(l,9,l,9); or (bf=0; bf<num_frames_for_i; bf++) {
Figure imgf000056_0001
for(k=0;k<3;k++){ for(l=0;K3;l++){ rank4mat[l+ 3*bf+j][l+3*k+l] = Tensors[rf] [i] [legalFrame[bf]][k] [j] [1] ; } } } } if (!svdcmp(rank4mat, 3*num_frames_for_i, 9, w, v_tr)) { free_dmatrix(rank4mat, 1 ,3 *num_frames_for_i, 1 ,9); free_dvector(w, 1 ,9); free_dmatrix(v_tr, 1 ,9, 1 ,9); retum(FALSE); }
/* look for threshold for the largest 4 values */ foundThresh = FALSE; thresh4 = 0.02; while (! foundThresh) { n = 0; for(k=l;k<=9;k++){ if(w[k]>thresh4){ n++; } } if (n>4) { thresh4*=2;
} elseif(n<4){ thresh4/=2;
} else { foundThresh = TRUE;
} }
/* the 4 rows of v which correspond to the */ /* 4 largest singular values form E_basis */ n = 0; for(m=l;m<=9;m++) { if(w[m]>thresh4) { for(k=0;k<3;k++){ for(l=0;K3;l++){
E_basis[rf][i][n][k][l] = v_tr[l+3*k+l][m]; }
} n++;
} }
if(!ImproveE_basis(E_basis[rf][i])) { free_dmatrix(rank4mat, 1 ,3 *num_frames_for_i, 1 ,9); free_dvector(w, 1 ,9); free_dmatrix(v_tr, 1 ,9, 1 ,9); retum(FALSE);
}
55
SUBSTITUTE SHEET (RULE 26^ /* improve the tensors with the svd results */ for (bf=0; bf<num_frames_for_i; bf++) { for(j=0;j<3;j++){ for(k=0;k<3;k++){ for(l=0;K3;l++){ sum = 0.0; n = 0; for(m=l;m<= ;m++){ if(w[m]>thresh4){ sum += rank4mat[l+3*bf+j][m]*w[m]*
E_basis[rfJ[i][n][k][l]; n++;
} } Tensors[rf][i][legalFrame[bf]][k][j][l] = sum;
} } } }
free_dmatrix(rank4mat, 1 ,3 *num_frames_for_i, 1 ,9); free_dvector(w, 1 ,9); free_dmatrix(v_tr, 1 ,9, 1 ,9);
} } else if (num_frames_for_i = 1) { (*size_E__basis) = 3;
/* keep the tensor as the first 3 of E_basis[rf][i] */ for(j=0;j<3;j++){ for(k=0;k<3;k++){ for (1=0; 1<3; _■++){ E_basis[rf][i]D][k][l] = 7
Tensors[rfj [i] [legalFrame[0]] [k] [j] [1] ;
}
} return(TRUE);
}
ComρuteAllEpipoles(IN DMAT3 E_basis[MAX_FRAMES]
[MAX_FRAMES][4], IN int size_E_basis,
IN BOOL removeOutliers,
OUT DXYZ Epipoles[MAX_FRAMES]
[MAX_FRAMES])
Figure imgf000059_0001
/* check the the number of frames is not too large */ if (NumberOfFrames > MAX_FRAMES) { return(FALSE); }
/* given a pair of images (rf,i) : */
/* there are 3*size_E_basis*(size_E_basis-l) + */
/* 6*size_E_basis*size_E_basis equations */
Eqn = dmatrix(0, 3*sizeJE_basis*(size_E_basis-l) + 6*size_E_basis*size_E_basis - 1, 0, 2);
/* loop on the reference frames */ for (rf=0; rf<NumberOfFrames; rf++) { /* consider only marked frames */ if (((specifιed_rf < 0) || (rf = specified_rf)) && Images_Flags[rfJ) {
/* loop on the i'th frame */ for (i=0; i<NumberOfFrames; i++) {
/* consider only marked frames that differ from rf */ if (Images_Flags[i] && (i != rf)) {
/* accumulate equations on Epipole */
n_Eq = 0;
/* contribution of Tensors[rf][i][bfJ [*]_]][*] */
/* denoting E[j]=Tensors[r_][i][bfl [*][]][*] and */ /* E ][k]=Tensors[rfj[i][bf][*][)][k] then */
/* [v'] E[0] ~=~ [v'j E[l] ~=~ [V] E[2] — F */ /* the fundamental matrix. */
/* In particular, transposed(E[j]) * [v1]* E[l] */
/* is a skew symmetric matrix and therefore */
/* V CROSS E[j][k] dot E[l][k] = 0 for j<l */
/* and also V CROSS E[j][k] dot E[l][m] + */ /* v' CROSS E[j][m] dot E[l][k] = 0 for j<l, k<m */
/* Equivalently (being the same "volume") */ /* E[j][k] CROSS E[l][k] dot v'=0 for j<l k=0,l,2*/ /* and also E[j][k] CROSS E[l][m] dot V + */
/* E[j][m] CROSS E[l][k] dot v'=0 for j<l k<m */
for (row=0; row<(3 *size_E_basis*(size_E_basis- 1 ) +
6*size_E_basis*size_E_basis); row++) { for (l=0; K3; l-H-) { Eqn[n_Eq+row][l] = 0.0;
} }
/* loop on E_basis[j] */ for (j=0; j<(size_E_basis-l); j++) { /* loop on E_basis[l] */ for (Hj+1); Ksize_E_basis; 1++) { /* loop on the column */ for (k=0; k<3; k++) { Eqn[n_Eq][0] = E_basis[rf][i][j][l][k]* E_basis[rfj[i][l][2][k] - E_basis[rf][i]D][2][k]* E_basis[rf][i][l][l][k];
Eqn[n_Eq][l] = E_basis[rf][i][l][0][k]* E_basis[rf][i]D][2][k] - E_basis[rfJ[i][l][2][k]* E_basis[rfJ[i]D][0][k];
Eqn[n_Eq][2] = E_basis[rf][i]0][0][k]* E_basis[rf][i][l][l][k] -
59 E_basis[r_][i][j][l_[k]* E_basis[rf][i][l][0][k]; /* normalize the equation */ row_sum = Eqn[n_Eq][0]*Eqn[n_Eq][0] + Eqn[n_Eq][l]*Eqn[n_Eq][l] + Eqn[n_Eq][2]*Eqn[n_Eq][2]; if (row_sum > 0.0) { row_sum = sqrt(row_sum); Eqn[n_Eq] [0]/=row_sum; Eqn[n_Eq] [ 1 ]/=row_sum; Eqn[n_Eq] [2]/=row_sum; n_Eq++;
} } for (k=0; k<2; k++) { for (m=(k+l); m<3; m++) { Eqn[n_Eq][0] = E_basis[rf|[i][j][l][k]* E_basis[rf][i][l][2][m] - E_basis[rf][i][j][2][k]* E_basis[rf][i][l][l][m] + E_basis[rf][i][j][l][m]* E_basis[rfj[i][l][2][k] - E_basis[rf][i]β][2][m]* E_basis[rf][i][l][l][k];
Eqn[n_Eq][l] = E_basis[rf][i][l][0][m]* E_basis[rfj[i]D][2][k] - E_basis[rf][i][l][2][m]* E_basis[rfj[i] ][0][k] + E_basis[rf][i][l][0][k]* E_basis[rf][i] ][2][m] - E_basis[rf][i][l][2][k]* E_basis[rf][i][j][0][m]; Eqn[n_Eq][2] = E_basis[rfj[i][j][0][k]* E_basis[rf][i][l][l][m] - E_basis[rf][i]D][l][k]* E_basis[r_][i][l][0][m] + E_basis[rf][i] ][O][m]* E_basis[rfJ[i][l][l][k] - E_basis[rf][i]D][l][m]* E_basis[rfj[i][l][0][k]; /* normalize the equation */ row_sum = Eqn[n_Eq][0]*Eqn[n_Eq][0] + Eqn[n_Eq][l]*Eqn[n_Eq][l] + Eqn[n_Eq] [2] *Eqn[n_Eq] [2] ; if (row_sum > 0.0) { row_sum = sqrt(row_sum); Eqn[n_Eq] [0]/=row_sum; Eqn[n_Eq] [ 1 ]/=row_sum; Eqn[n_Eq] [2]/=row_sum; n_Eq++;
} } } } }
/* additional size_E_basis*size_E_basis*6 eqns /* come from [v']*E[rf][i][j]*E[i][rf][k] being */ /* skew symmetric for any j,k. */
if (specifιed_rf < 0) { /* loop on E_basis[rf][i][j] */ for (j=0; j<size_E_basis; j++) { /* loop on E_basis[i][rf][l] */
for (1=0; l<size_E_basis; 1++) { matmulmat3 (mat, E_basis [rf] [{][}], E_basis[i][rf][l]);
/* ignore mat if it is a scalar matrix */ if ((mat[0][l] > mm_epsilon) || (mat[0][l] < -mm_epsilon) (mat[0][2] > mm_epsilon) || (mat[0][2] < -mm_epsilon) (mat[l][2] > mm_epsilon) || (mat[l][2] < -mm_epsilon) | (mat[l][0] > mm_epsilon) || (mat[l][0] < -mm_epsilon) | (mat[2][0] > mm_epsilon) || (mat[2][0] < -mm_epsilon) | (mat[2][l] > mm_epsilon) || (mat[2][l] < -mm_epsilon)) {
Eqn[n_Eq][0] = 0.0; Eqn[n_Eq][l] = mat[2][0]; Eqn[n_Eq][2] = -mat[l][0]; /* normalize the equation */ row_sum = Eqn[n_Eq][0]*Eqn[n_Eq][0] + Eqn[n_Eq][l]*Eqn[n_Eq][l] + Eqn[n_Eq][2]*Eqn[n _q][2]; if (row_sum > 0.0) { row_sum = sqrt(row_sum);
Eqn[n_Eq] [0]/=row_sum;
Eqn[n_Eq] [ 1 ]/=row_sum;
Eqn[n_Eq] [2]/=row_sum; n_Eq++;
}
62 Eqn[n_Eq][0] = -mat[2][l]; Eqn[n_Eq][l] = 0.0; Eqn[n_Eq][2] = mat[0][l]; /* normalize the equation */ row_sum = Eqn[n_Eq][0]*Eqn[n_Eq][0] + Eqn[n_Eq][l]*Eqn[n_Eq][l] + Eqn[n_Eq] [2] *Eqn[n_Eq] [2] ; if (row_sum > 0.0) { row_sum = sqrt(row_sum);
Eqn[n_Eq] [0]/=row_sum;
Eqn[n_Eq] [ 1 ]/=row_sum;
Eqn[n_Eq] [2]/=row_sum; n_Eq++;
}
Eqn[n_Eq][0] = mat[l][2]; Eqn[n_Eq][l] = -mat[0][2]; Eqn[n_Eq][2] = 0.0; /* normalize the equation */ row_sum = Eqn[n_Eq][0]*Eqn[n_Eq][0] + Eqn[n_Eq][l]*Eqn[n_Eq][l] + Eqn[n_Eq][2]*Eqn[n_Eq][2]; if (row_sum > 0.0) { row_sum = sqrt(row_sum);
Eqn[n_Eq] [0]/=row_sum;
Eqn[n_Eq] [ 1 ]/=row_sum;
Eqn[n_Eq] [2]/=row_sum; n_Eq++;
}
Eqn[n_Eq][0] = -mat[2][0];
Eqn[n_Eq][l] = mat[2][l];
Eqn[n_Eq][2] = mat[0][0]-mat[l][l];
/* normalize the equation */ row_sum = Eqn[n_Eq][0]*Eqn[n_Eq][0] + Eqn[n_Eq][l]*Eqn[n_Eq][l] +
Eqn[n_Eq][2]*Eqn[nJEq][2]; if (row_sum > 0.0) { row_sum = sqrt(row_sum); Eqn[n_Eq] [0]/=row_sum; Eqn[n_Eq] [ 1 ]/=row_sum; Eqn[n_Eq] [2]/=row_sum; n_Eq++;
} Eqn[n_Eq][0] = mat[l][0];
Eqn[n_Eq][l] = mat[2][2]-mat[0][0]; Eqn[n_Eq][2] = -mat[l][2]; /* normalize the equation */ row_sum = Eqn[n_Eq][0]*Eqn[n_Eq][0] + Eqn[n_Eq][l]*Eqn[n_Eq][l] + Eqn[nJEq][2]*Eqn[nJEq][2]; if (row_sum > 0.0) { row_sum = sqrt(row_sum);
Eqn[n_Eq] [0]/=row_sum;
Eqn[n_Eq] [ 1 ]/=row_sum;
Eqn[n_Eq] [2]/=row_sum; n_Eq++;
}
Eqn[n_Eq][0] = mat[l][l]-mat[2][2]; Eqn[n_Eq][l] = -mat[0][l]; Eqn[n_Eq][2] = mat[0][2]; /* normalize the equation */ row_sum = Eqn[n_Eq][0]*Eqn[n_Eq][0] + Eqn[n_Eq][l]*Eqn[n_Eq][l] + Eqn[n_Eq][2]*Eqn[n_Eq][2]; if (row_sum > 0.0) { row_sum = sqrt(row_sum);
Eqn[n_Eq] [0]/=ro w_sum; Eqn[n_Eq] [ 1 ]/=row_sum; Eqn[n_Eq] [2]/=row_sum; n_Eq++;
} } } } }
if (removeOutliers) {
/* remove outliers */ nGroups[0] = n_Eq; Sizes[0] = l;
if (!removeEqOutliers(Eqn, nCols, nSizes, nGroups, Sizes, minRows, nTrials, SquaredThreshold, &n_Eq_new)) { n_Eq = n_Eq_new;
}
}
if(n_Eq > 0) {
/* solve F by Jacobi */ fιnd_solJacobi(Eqn, n_Eq, 3, Epipoles [rf][i], &error);
} } } } } free_dmatrix(Eqn, 0, 3*size_E_basis*(size_E_basis-l) +
6*size_E_basis*size_E_basis - 1, 0, 2); return(TRUE);
}
BOOL ImproveE_basis(INOUT DMAT3 E_basis[4])
{ int i, j, k, 1, m; double **u, **v_tr, *w; double min_sv, sum;
/* Note that each of the 3 4-by-3 matrices E_basis[*][*][j] is of */ /* rank 2. After applying svd to each of them, ignore the smallest */ /* singular value (which should be 0) in order to improve them */ u = dmatrix(l,4,l,3); v_tr = dmatrix( 1 ,3 , 1 ,3 ); w = dvector(l,3);
for (j=0; j<3; j++) { for (k=0; k<4; k++) { for (l=0; K3; B-r) { u[l+k][l+l] = E_basis[k][l]D]; } }
if (!svdcmp(u, 4, 3, w, v_tr)) { free_dmatrix(u, 1 ,4, 1 ,3); free_dmatrix(v_tr,l,3,l,3); free_dvector(w, 1 ,3); return(FALSE); }
/* find smallest singular value */
66
SUBSTITUTE SHEET (RULE 26? 17US96/05697
min_sv = w[l]; i=l; for(l=2;K=3;l++){ if (w[l]<min_sv) { min_sv = w[l];
} }
/* replace the E_basis by the product ignoring w[i] */ for (k=0; k<4; k++) { for(l=0;K3;l++){ sum = 0.0; for (m=l ; m<=3; m++) { if(m!=i){ sum += u[l+k][m]*w[m]*v_tr[l+l][m];
} } E_basis[k][l][j] = sum;
} } }
free_drnatrix(u, 1 ,4, 1 ,3); free_dmatrix(v_tr, 1 ,3 , 1 ,3); free_dvector(w, 1 ,3); return(TRUE); }
FUNCTION
BOOL removeEqOutliers /* remove outliers for homogeneous system */
67 (
INOUT double **Mat, /* the initial equations */ IN int nCols, /* there are nCols columns (unknowns) */ IN int nSizes, /*
* it is assumed that the equations are
* organized in groups. These groups
* have nSizes different sizes. */
IN int nGroups [], /*
* there are nSizes nGroups which mark the
* number of groups of each size */
IN int Sizes[], /*
* there are nSizes Sizes which mark the
* size of each group.
* e.g. : the first nGroups[0] are groups
* each of size Sizes [0], and so on */
IN int minRows, /*
* each iteration, try at least
* minRows number of rows */
IN int nTrials, /* number of random trials to perform */
IN double SquaredThreshold,
/* for boundary determination */
OUT int *outMatRows /*
* The number of rows in Mat to be
* considered. These rows are the first
* rows in the output Mat and do not
* keep any grouping order */
68
SUBSTITUTE SHEET (RULE 2W DESCRIPTION
Given the initial equations in Mat, get rid of the outliers and keep the fitting equations as the leading rows of the output Mat.
Perform nTrials random trials, each time randomly choose groups so that there are at least minRows equations. Solve this set of n_current_equations by Jacobi and check the solution against ALL equations. Keep track of the best solution (for which the norm of the Mat* sol is minimal).
Use the best solution to determine outliers (keeping the initial partition of the equations into groups). Put the groups which fit the best solution as the leading *outMatRows rows of Mat
ASSUMPTIONS
The equations are normalized so that there is no need to consider the equation's norm when performing the outliers removal
AUTHOR
Tamir Shalom (April 10 1995)
*/
Figure imgf000071_0001
Figure imgf000072_0001
/*
* Let maxSize be the maximal size of groups, then
* the maximal number of rows that will be used is
* minRows+maxSize-1 .
*/
/* find maxSize, initialRows and nTotalGroups */ initialRows = nGroups[0]*Sizes[0]; nTotalGroups = nGroups[0]; maxSize = Sizes [0]; for (i=l; i<nSizes; i++) { initialRows += nGroups[i]*Sizes[i]; nTotalGroups += nGroups [i]; if (Sizes[i] > maxSize) { maxSize = Sizes[i];
}
}
70 if (initialRows < minRows) { return(FALSE);
}
/* allocate subMatrix */ subMat = dmatrix(0, minRows+maxSize-1 - 1, 0, nCols - 1);
/* allocate solution and best_solution */ solution = dvector(0, nCols-1); best_solution = dvector(0, nCols-1);
/* allocate from_row and to_row_plusl */ from_row = ivector(0, nTotalGroups- 1); to_row_plusl = ivector(0, nTotalGroups- 1);
/* perform the trials */ for (iTrial=0; iTrial < nTrials; iTrial++) {
/* initialize from_row and to_row_plus 1 */ iRows = 0; iGroup = 0; nAvailGroups = nTotalGroups; for (i=0; i<nSizes; i++) { for G=0; j<nGroups[i]; j-H-) { from_row[iGroup] = iRows; iRows += Sizes [i]; to_row_plus 1 [iGroup] = iRows; iGroup++;
} }
/* iRows denotes the number of already chosen equations */ iRows = 0;
71
SUBSTITUTE SHEET (RULE 2B while (iRows < minRows) { nrandGroup = randomn(nAvailGroups); /* move the group to the subMat */ for (i=from_row[nrandGroup]; i < to_row__plusl [nrandGroup]; _++) { for (j=0; j < nCols; j++) { subMat[iRows][j] = Mat[i]0];
} iRows++;
}
/* update nAvailGroups, from_row and to_row_plusl */ nAvailGroups—; for (i=nrandGroup; i < nAvailGroups; i++) { from_row[i] = from_row[i+l]; to_row_plus 1 [i] = to_row_plus 1 [i+ 1 ] ; } }
/* solve the equations of subMat */ fιnd_solJacobi(subMat, iRows, nCols, solution, &error);
/* estimate the solution against all equations */ squared_norm = 0.0; for (i=0; i<initialRows; i++) { /*
* compute the i'th entry of the resulting product of
* Mat by solution */ entry = 0.0; for (j=0; j<nCols; j-H-) { entry += Mat[i][j]*solution[j]; } squaredjnorm += entry*entry; }
if ((iTrial = 0) || (squared_norm < min_error)) { min_error = squared_norm; /* printf("iTrial %d error = %g\n", iTrial, squared_norm); */
/* keep the best solution so far */ for G=0; j<nCols; j++) { best_solution[j] = solution!]]; } } }
/* determine boundary for rejection of outliers */ boundary = min_error*SquaredThreshold/(initialRows-l);
/* remove outlying groups of equations from Mat */ (*outMatRows) = 0; iRows = 0; iGroup = 0; for (i=0; i<nSizes; i++) { for G=0; j<nGroups[i]; j++) {
/* accumulate the norm of the whole group */ squared_norm =0.0; for (k=0; k<Sizes[i]; k++) { entry = 0.0; for (1=0; KnCols; 1++) { entry += MatfiRows + k][l]*best_solution[l];
} squared_norm += entry* entry;
} iGroupH if (squared_norm < Sizes [i]* boundary) { /* add Sizes[i] equations */
/* xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx printf (" °/od of kind %d is taken\n", j, i); xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx */ if ((*outMatRows) = iRows) {
/* there is no need to copy the equations */
/* yet (*outMatRows) should be updated */
(*outMatRows) += Sizesfi];
} else { for (k=0; k<Sizes[i]; k++) { for (1=0; KnCols; 1++) {
Mat[(*outMatRows)][l] = Mat[iRows + k][l];
} (*outMatRows)++;
} } } iRows += Sizes[i];
} } free_dmatrix(subMat, 0, minRows+maxSize-1 - 1, 0, nCols - 1); free_dvector(solution, 0, nCols-1); free_dvector(best_solution, 0, nCols-1); free_ivector(from_row, 0, nTotalGroups- 1); free_ivector(to_row_plusl, 0, nTotalGroups- 1); printf ("after outliers removal only %d out of %d equations\n",
(*outMatRows), initialRows); return(TRUE); }
APPENDIX B
simulation of using 'I rihnear Tensυ. T r scene rccon.str.ic.iυn f:om three view^
3D points are generated by a pseudo random generator and the views are generated by choosing Camera parameters:
The first view camera transformation is the 3x4 matrix [I 0] where I is the identity matrix.
The second view camera transformation i.s the 3x4 matrix [R vp] where R is a rotation matrix and ^ ■; -s translation.
The third view i yenerat • '• by [Rpp vpp] where Rpp is a lotatson matrix and vpp ii a translation. define space points and image points of the three images gel back space points in P and image points in p.pp.ppp
> get. lmagβ_points():
> solve for trilinear tensor from p.pp.ppp and get back El ,E2,E3 (the three matrices representing the tensor)
> fnd_tensor(p,pp,ppp):
Given the tensor (represented by the matrices E1.E2.E3) we compute the epipolar geometric representation of the views one and two. The epipolar geometry consists of a 3x3 matrix F satisfying the constraint transposc(Ej)F + F transpose(Ej)=0 and the epipole v' satisfying the constraint FΛtv'=Q.
> fnd_matrlx(E1,E2,E3):
> vp_from_F(F):
The projective coordinates of the scene satisfy the constraint: pp = Elp + kv' and die resulting coordinates are (x.y.l .k) another projective representation is by (x.y.l .s) where s satisfies
PP = (v']Fp + sV where [v'] is the skew-symmetric matrix associatex with vector product.
We implement here oι?ly the irst representation.
> proj :=matrix(4,10):
> for j from 1 to 10 do
> ProjP[1,|]:= p[1 ,j]:
> projP[2,J]:_ p[2,j]:
> prolPpjJr- pPJ]:
> projP[4fjj:=getk(col(pp,j)revalm(E1 &* col(p.j)), vp_est):
> od:
To go from projective to Euclidean coordinates we use live control points. There are other ways to find the piojectivc-to-Euclidean transformation, but here we implement the simplest way by using control points, get back matrix T
> proj2E_c(proJP,P):
76 compare projective and Euclidean
Figure imgf000079_0001
> evalm(Pcomp);
Collection of subroutines (procedures).
>
> with(linalg):
Warning: new definition for norm Warning: new definition for trace
> fnd_teπsor:=proc(p,pp,ppp)
> TT:=matrix(40,27,i->0):
> j:=1 :
Figure imgf000079_0002
77
SUBSTTnπi: _HEET (RULE 26)
Figure imgf000080_0001
> b:=-co!(TT,27);
> TT:=submatrix(TT,1..40,1..26):
> sol:aleastsqrs(TT,b):
> X:=vector(27,i->1):
> for i from 1 to 26 do X[i]:=sol[i3; od:
> alpha:=array(1..3,1..3,1..3):
> for i from 1 to 3 do
> for j from 1 to 3 do
> for k from 1 to 3 do
> βlp a[i,jfk].=X[k+(M)*9 + (j-1)*33;
> od: od: od:
>
> E1 :=matrix(3,3):
> E2:s-matrix(3,3):
> E3:=matrix(3f3):
> for i from 1 to 3 do
> for k from 1 to 3 do
> E1[i,k]:sa!pha[i,1,k]
> E2[i,k]:=alpha[i,2,k]
> E3[i,k]:=alpha[i,3,k]
> od: od:
>
78 > W1 :=matrix(3,3)
> V/2:=matrlx(3,3)
> W3:=ma_rlx(3,3)
> for ] from l to 3 do
> for k from 1 to 3 do
> W1 [j,k]:=alpha[1 ,),k)
Figure imgf000081_0001
>
> end:
> fnd_matrix:=proc(M1,M2,M3)
> Teβt:=matrix(18,9,l->0):
> for I from 1 to 3 do
> Test[1 ,i3:=col(M1 ,1 )[i};Test[2,i+3J:=col(M1 ,2)[l].Test[3.i+63:=col(M1 ,3)[l];
> Test[4,l3:=col(M1 ,2)[i]; •
> Test[4,l+3]:=coI( 1,1)[il;
> Test[5,.3:sco.(M1 ,3)[i];
Figure imgf000081_0002
> Test[17,l3:=col(M3,3)[il; >
Figure imgf000081_0003
> Test:=submatrlx(Test,1..18,1..8):
> sol:=leastsqrs(Teot,b):
> X:=vector(9,i->1 ):
Figure imgf000082_0001
Figure imgf000082_0004
>
> skew..cross:-sproc(w)
> atack(evalm(crossprod([1 ,0,03,w)),stack(evalm(crossprod([0r1 ,Q],w)),evalm(cro
> s8prod([0,0,1],w))));
> end:
> gel epipole from FΛtv'=U
> vp_from_F:=proc(F)
> G:^transpose(F):
> b:=-col(G,3):
> G:=8ubmatrix(G,1..3,1..2):
> sol: eastsqrs(G,b): >
> vp_est:=vector(3,i->1):
> vp_est[1]:=sol[13: vp_est[2]:=sol[23:
> end:
Figure imgf000082_0002
> randn:=rand(1..100):
> R:=rot_mat(0.9f[0.3rQ.5,0.93):
> Rp:=rot_mat(0.9,[2.3,0.7,1.13):
> vpp:=vector(3,[1.1,3.3,1-0]):
> vp:=vactor(3, [0.5,-2.2,1.0]):
> P:=matrlx(4,10):
> for I from 1 to 3 do
> for j from 1 to 10 do
> P[i,j]:=randn()/10;
> od: od:
> for ) from 1 to 10 do
> P[4,j]:=1 ;
> od:
Figure imgf000082_0003
> B:=augmβnt(Rp,vpp):
> p:=evaim(8ubmatrix(P,1..3_1..10)):
> pp:=evatm(A &* P):
80 > ppp:=evfilm(B &*P.:
> for j from 1 to i " -
> for i from i _υ 2 do
> PPPt'.il- PPli.ll/PPPP.Jl;
>
Figure imgf000083_0001
> for j from 1 to 10 do
> p[3,j]:=1 : pp[3,j]:=1 : ppp[3,j]:=1 ;
> od: end:
> > > transform back lυ P.
Wc need at least 5 points.
> pro]2Euc:=-proc(projP,P) > T:=matrix(15,16,i->0): > for j from 1 to 5 do > i:=1 +(j-1)*3: > T[il1}:=col(P,j)[4}'col(projP,])[1] > T[.,2]:βcol(P,J)[4]'col(projP,l)[2] > T[i,33:=col(Ptj)[4]»col(pro]P,j)[3] > T[i,43:=col(P,J)[43'col(projP,j)[43 > T[M3]:=-col(P,j)[1]*col(projP,|)[1] > T[l,14}:_.col(Pfl)[1]*col(pro)P,|)[23 > T[ll15]:-col(PJ)[1]*col(pro)P,l)[3] > T[i,1
Figure imgf000083_0002
]*col(proJPf|)[4] > > > T[i+1 ,5]:acol(Pl|)[4l*col(projPlJ)[1] > T[i+1 ,6]:=col(P,j)[4]-col(projP,i)[2] > T[i+1 ,7]:=col(P,j)[4]*col(projP,i)[3] > T[i+1,8]:=col(P,J)[4]*col(projP,j)[4] > T[i+1,13]:=-col(P,j)[2]*col(projP,i)[1] > T[i+1l14]:=-col(PIj)[2l*col(projP,J)t2] > T[i+1 l .5]:=-col(Pf))[2l*col(projP,|)[3] > T[l+1116]:=-col(P,|)[2l*col(proiP,I)[4] > > T[i+2t9]:»col(Pt|)[4]-col(projPlJ)[1]: > T[i+2,103:=col(P,j)[43*col(projP,j)[23: > T[i+2)11J:=col(P,i)[4]'col(proiP,j)[3]: > T[l+2,123:=col(P,])[4]*col(projP,J)[43: > T[i+2,13]:=-col(P,i)[33*col(projP,j)[13 > T[i+2,14]:=-col(P,j)[33*col(proiP,|)[2] > T[l+2,15]:=-col(PIi)[3]'col(proiP,i)[3] > T[i+2,16]:=-col(PfJ)[3]*col(proj*PtJ)[4] > od: >
81 > b:=-col(T.16):
> T:_submatr.x(T,1..15,1..15):
> sol:=!insolve(T,b):
> X-_ve_._or( .5,ι,->1):
> for i from 1 to 15 do X[l]:-sol[i]; od:
> T:_matrix(4,4):
> for I from 1 to 4 do
> for j from 1 to 4 do
> T[i,Jl:-X[]+(|.ir4]:
> od: od:
> end:
>
> getk:=proc(a,b,c)
> evalm((transpose(cro88prod(a,b)) &*
> crossprod(c,a))/(transpose(crossprod(c,a)) &* crossprod(c,a)));
> end:
>
Figure imgf000084_0001
> w_unit:_evalm((1/norm(w,frobβnius))"w);
> WW:=evalrn(w._unit &* transpose(w_unit)):
> cos_mat:-evalm(cos(angle) * array(iden.lty,1..3,1..3));
> W_.antisym:=stack(evalm(crossprod([1,0f0], _unit)),stack(evalm(crossprod([0,
> 1 ,03, w_unit)),evalm(crossprod([0,0,1 ],w_unit))));
> evalm(cos_mat + sin(angle)*W_antisym + (1-cos(angle))*WW);
> end:
> cross_mat:=proc(w)
> evalm(8tack(evalm(crossprod([1,0,0], _unit)),stack(evalm(crossprod([0,1 ,0],w
> _unit)),evalm(crossprod([0,0,13, _unit)))));
> end:

Claims

1. A method for generating information regarding a 3D object from at least one 2D projection thereof, the method com¬ prising: providing at least one 2D projection of a 3D object; generating an array of numbers described by: αijk = vi' bjk ~ vj" aik (i J»k = 1/2,3), where aj_ and b-jjς are elements of matrices A and B respectively and VJ 1 and Vj_" are elements of vectors v1 and v" respectively, wherein said matrices and vectors together describe camera parameters of three views of the 3D object; and employing the array to generate information regarding the 3D object.
2. A method for generating a new view of a 3D object from first and second existing views thereof having n corresponding points ^ and p^' (i = 1 ... n) , the method comprising: generating a tensor c-ijk' and plugging the values of points p^ and p^ ' (i = l ... n) into trilinear forms, thereby to extract an x" , y" value- representing a location in the new view; and generating the new view on the basis of the result of the plugging in step.
3. A method for reconstructing a 3D object from at least one 2D projection thereof, the method comprising: providing at least one 2D projection of a 3D object; generating an array of numbers described by: αijk = vi' bjk " vj" aik (i. D #k = 1,2,3) , where aj_j and bj^ are elements of matrices A and B respectively and v^' and v^" are elements of vectors v1 and v" respectively, wherein said matrices and vectors together describe camera parameters of three views of the 3D object; permuting the array into three homography matrices associated with three corresponding planes in 3D space; and employing the three homography matrices to reconstruct the 3D object.
4. A visual recognition method comprising: providing three perspective views of a 3D object between which a trilinear relationships exists; and employing the trilinear relationship between the views in order to perform visual recognition by alignment.
5. A method according to claim 4 and also comprising reprojecting the 3D object.
6. A method according to claim 1 wherein said information regarding the 3D object comprises a reconstruction of the 3D object.
7. A method according to claim 1 wherein said information regarding the 3D object comprises at least one new view of the 3D object generated without reconstructing the 3D object.
8. A method according to claim 1 wherein said at least one 2D projection comprises at least one aerial photograph.
9. A method according to claim 1 wherein said at least one 2D projection comprises at least one satellite photograph.
10. A method according to claim 1 wherein said information regarding the 3D object comprises at least one coordinate of the 3D object.
11. A method according to claim 1 wherein the 3D object comprises an aerospace object.
12. A method according to claim 1 wherein the 3D object comprises a large object such as a ship.
13. A method according to claim 1 wherein the 3D object comprises a nonexistent object.
14. Apparatus for generating information regarding a 3D object from at least one 2D projection thereof, the apparatus comprising: apparatus for providing at least one 2D projection of a 3D object; an array generator operative to generate an array of numbers described by: αijk = vi' bjk ~ vj" aik (i/j,k = 1,2,3), where a^j and b k are elements of matrices A and B respectively and v^' and v^" are elements of vectors v' and v" respectively, wherein said matrices and vectors together describe camera parameters of three views of the 3D object; and an array analyzer employing the array to generate information regarding the 3D object.
15. Apparatus for generating a new view of a 3D object from first and second existing views thereof having n corresponding points p^ and p^' (i = 1 ... n), the apparatus comprising: apparatus for generating a tensor c-^jk; and apparatus for plugging the values of points p^ and p^* (i = 1 ... n) into trilinear forms, thereby to extract an x" , y" value representing a location in the new view; and apparatus for generating the new view on the basis of the result of the plugging in step.
16. Apparatus for reconstructing a 3D object from at least one 2D projection thereof, the apparatus comprising: apparatus for providing at least one 2D projection of a
3D object; an array generator operative to generate an array of numbers described by: αijk = vi' bjk " vj" aik (i»J,k = 1,2,3), where a_ and b_jk are elements of matrices A and B respectively and v- ' and v^" are elements of vectors V and v" respectively, wherein said matrices and vectors together describe camera parameters of three views of the 3D object; an array permutator operative to permute the array into three homography matrices associated with three corresponding planes in 3D space; and a 3D object reconstructor operative to employ the three homography matrices to reconstruct the 3D object.
17. Visual recognition apparatus comprising: apparatus for providing three perspective views of a 3D object between which a trilinear relationships exists; and apparatus for employing the trilinear relationship between the views in order to perform visual recognition by alignment.
18. A method according to claim 1 wherein said at least one 2D projection comprises three 2D projections.
19. Apparatus according to claim 14 wherein said at least one 2D projection comprises three 2D projections.
20. A method according to claim 1 and also comprising employing a result of said method in order to perform coordinate measurement of industrial parts.
21. A method according to claim 1 and also comprising employing a result of said method in order to perform automated optical based inspection of industrial parts.
22. A method according to claim 1 and also comprising employing a result of said method in order to perform robotic cell alignment.
23. A method according to claim 1 and also comprising employing _. result of said method in order to perform robotic trajectory identification.
24. A method according to claim 1 and also comprising employing a result of said method in order to perform 3D robotic feedback.
25. A method according to claim 1 and also comprising employing a result of said method in order to perform 3D modell¬ ing of scenes.
26. A method according to claim 1 and also comprising employing a result of said method in order to perform 3D modell¬ ing of objects.
27. A method according to claim 1 and also comprising employing a result of said method in order to perform reverse engineering.
28. A method according to claim 1 and also comprising employing a result of said method in order to perform 3D digitiz¬ ing.
29. A 3D scene reconstruction method for generating a 3D representation of a 3D scene from first, second and third views thereof, the method comprising: providing first, second and third views of a 3D scene; employing geometric information regarding said first, second and third views to generate a trilinear tensor represent¬ ing the geometric relationship between the first, second and third views; and generating a 3D representation of the 3D scene from said trilinear tensor.
30. A method according to claim 29 wherein said step of generating a 3D representation comprises: computing an epipolar geometric representation of the first and second views from the trilinear tensor; and generating said 3D representation from said epipolar geometric representation.
31. Image transfer apparatus for generating a novel view of a 3D scene from first and second reference views thereof, the apparatus comprising: apparatus for providing first and second reference views of a 3D scene; a trilinear tensor generator operative to employ geo¬ metric information regarding said first reference view, second reference view and novel view, respectively, to generate a tri¬ linear tensor representing the geometric relationship between the first, second and novel views; and a novel view generator operative to generate said novel view by computing a multiplicity of novel view locations each corresponding to different first and second corresponding loca¬ tions in said first and second reference views respectively based on said first and second corresponding locations and said trilin¬ ear tensor.
32. 3D scene reconstruction apparatus for generating a 3D representation of a 3D scene from first, second and third views thereof, the apparatus comprising: appparatus for providing first, second and third views of a 3D scene; a trilinear tensor generator operative to employ geo¬ metric information regarding said first, second and third views to generate a trilinear tensor representing the geometric rela¬ tionship between the first, second and third views; and a 3D scene representation generator operative to gener¬ ate a 3D representation of the 3D scene from said trilinear tensor.
33. A method according to claim 29 wherein said providing step comprises providing m > 3 views of the 3D scene and wherein said employing step is performed for each of a plurality of subsets of 3 views from among the m views, thereby to generate a plurality of intermediate tensors, the method also comprising, prior to said step of generating, combining the plurality of intermediate tensors into a final trlinear tensor and wherein said step of generating comprises generating a 3D representation of the 3D scene from said final trlinear tensor.
34. A method according to claim 33 wherein said employing step is performed for the first and second views in combination with each of the remaining views, thereby to generate m-2 intermediate tensors.
35. A method according to claim 33 wherein said step of combining comprises: arranging each of the intermediate tensors within a corresponding 9 x (3 (m-2)) matrix; and defining the final trilinear tensor as the four largest principal components of said matrix.
36. Matching point finding apparatus for finding a multiplicity of matching locations across three views, the apparatus comprising: providing an initial set of matching locations across three views; generating a trilinear tensor representing the rela¬ tionships between said three views, based on said initial set; and employing the trilinear tensor to generate a final set of matching locations.
37. Apparatus according to claim 36 wherein said step of employing comprises: for each of a multiplicity of locations within a first of the three views: generating first and second corresponding epipolar lines from the tensor which are associated with the remaining two views respectively; selecting a first candidate matching location along the first epipolar line and computing a second candidate matching location along the second epipolar line based on the first matching location; and repeating the selecting step until the two candidate matching locations and the first view location match.
38. A method according to claim 29 and also comprising: performing surface interpolation on the 3D representa¬ tion, thereby to generate a surface interpolated output; and generating an orthophoto from the surface interpolated output.
39. A method according to claim 29 wherein said step of providing comprises sequentially positioning at least three imaging devices at a sequence of positions within the scene and imaging at least a portion of the scene, using each of the imaging devices, at each of the positions within the sequence. 40• A method according to claim 29 and also comprising comparing the 3D representation of the 3D scene to a stored model of the 3D scene.
91
ET RULE 26
PCT/US1996/005697 1995-04-25 1996-04-24 Apparatus and method for recreating and manipulating a 3d object based on a 2d projection thereof WO1996034365A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP96913833A EP0832471A4 (en) 1995-04-25 1996-04-24 Apparatus and method for recreating and manipulating a 3d object based on a 2d projection thereof
JP8532665A JPH11504452A (en) 1995-04-25 1996-04-24 Apparatus and method for reproducing and handling a three-dimensional object based on a two-dimensional projection
AU56674/96A AU5667496A (en) 1995-04-25 1996-04-24 Apparatus and method for recreating and manipulating a 3D object based on a 2D projection thereof

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
IL113496 1995-04-25
IL11349695A IL113496A (en) 1995-04-25 1995-04-25 Apparatus and method for recreating and manipulating a 3d object based on a 2d projection thereof
US08/497,224 US5821943A (en) 1995-04-25 1995-06-30 Apparatus and method for recreating and manipulating a 3D object based on a 2D projection thereof
US08/497,224 1995-06-30

Publications (2)

Publication Number Publication Date
WO1996034365A1 WO1996034365A1 (en) 1996-10-31
WO1996034365A9 true WO1996034365A9 (en) 1996-12-27

Family

ID=26323038

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US1996/005697 WO1996034365A1 (en) 1995-04-25 1996-04-24 Apparatus and method for recreating and manipulating a 3d object based on a 2d projection thereof

Country Status (6)

Country Link
EP (1) EP0832471A4 (en)
JP (1) JPH11504452A (en)
CN (1) CN1198230A (en)
AU (1) AU5667496A (en)
CA (1) CA2219314A1 (en)
WO (1) WO1996034365A1 (en)

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5877897A (en) 1993-02-26 1999-03-02 Donnelly Corporation Automatic rearview mirror, vehicle lighting control and vehicle interior monitoring system using a photosensor array
US6822563B2 (en) 1997-09-22 2004-11-23 Donnelly Corporation Vehicle imaging system with accessory control
US7655894B2 (en) 1996-03-25 2010-02-02 Donnelly Corporation Vehicular image sensing system
IL119831A (en) * 1996-12-15 2002-12-01 Cognitens Ltd Apparatus and method for 3d surface geometry reconstruction
ES2167885T3 (en) 1997-04-04 2002-05-16 Isis Innovation APPARATUS AND METHOD OF IMAGE FORMATION BY MICROSCOPY.
US6201541B1 (en) * 1997-12-11 2001-03-13 Cognitens, Ltd. System and method for “Stitching” a plurality of reconstructions of three-dimensional surface features of object(s) in a scene defined relative to respective coordinate systems to relate them to a common coordinate system
DE19832974A1 (en) 1998-07-22 2000-01-27 Siemens Ag Arrangement for generating virtual industrial system model compares system component information with real system image data to identify components in image data
WO2000002162A1 (en) * 1998-06-30 2000-01-13 Siemens Aktiengesellschaft Device and method for constructing a virtual installation model
WO2003012368A1 (en) 2001-07-30 2003-02-13 Topcon Corporation Surface shape measurement apparatus, surface shape measurement method, surface state graphic apparatus
EP1504276B1 (en) 2002-05-03 2012-08-08 Donnelly Corporation Object detection system for vehicle
NZ544937A (en) * 2003-07-29 2009-03-31 Pixar Improved paint projection method and apparatus
US7526103B2 (en) 2004-04-15 2009-04-28 Donnelly Corporation Imaging system for vehicle
WO2008024639A2 (en) 2006-08-11 2008-02-28 Donnelly Corporation Automatic headlamp control system
CN104596484A (en) * 2015-01-30 2015-05-06 黄河水利委员会黄河水利科学研究院 Method of measuring drift ice density in ice flood season of Yellow River
KR101865112B1 (en) * 2017-03-07 2018-07-13 광주과학기술원 3D restoration apparatus and method including exterior material modeling
AU2021283398A1 (en) 2020-06-01 2023-01-05 Icares Medicus, Inc. Double-sided aspheric diffractive multifocal lens, manufacture, and uses thereof

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5344298A (en) * 1984-08-08 1994-09-06 3D Systems, Inc. Apparatus for making three-dimensional objects by stereolithography
JP2892423B2 (en) * 1990-02-28 1999-05-17 株式会社日立製作所 Image display device and image display method
JPH0454682A (en) * 1990-06-22 1992-02-21 Toshiba Corp Method and device for three-dimensional picture processing
JP3117097B2 (en) * 1992-01-28 2000-12-11 ソニー株式会社 Image conversion device
US5454371A (en) * 1993-11-29 1995-10-03 London Health Association Method and system for constructing and displaying three-dimensional images

Similar Documents

Publication Publication Date Title
US5821943A (en) Apparatus and method for recreating and manipulating a 3D object based on a 2D projection thereof
Shashua et al. Relative affine structure: Canonical model for 3D from 2D geometry and applications
Quan et al. Affine structure from line correspondences with uncalibrated affine cameras
Avidan et al. Novel view synthesis by cascading trilinear tensors
US6137491A (en) Method and apparatus for reconstructing geometry using geometrically constrained structure from motion with points on planes
Wilczkowiak et al. Using geometric constraints through parallelepipeds for calibration and 3D modeling
US6198852B1 (en) View synthesis from plural images using a trifocal tensor data structure in a multi-view parallax geometry
Benjemaa et al. Fast global registration of 3D sampled surfaces using a multi-z-buffer technique
Devernay et al. From projective to euclidean reconstruction
Rother et al. Linear multi view reconstruction and camera recovery using a reference plane
Shashua et al. Relative affine structure: Theory and application to 3D reconstruction from perspective views
Shashua Algebraic functions for recognition
Szeliski et al. Geometrically constrained structure from motion: Points on planes
Kumar et al. Registration of video to geo-referenced imagery
Pollefeys et al. From images to 3D models
Laveau et al. Oriented projective geometry for computer vision
WO1996034365A9 (en) Apparatus and method for recreating and manipulating a 3d object based on a 2d projection thereof
EP0832471A1 (en) Apparatus and method for recreating and manipulating a 3d object based on a 2d projection thereof
Wong et al. Reconstruction of sculpture from its profiles with unknown camera positions
Irani et al. What does the scene look like from a scene point?
Pagani et al. Dense 3D Point Cloud Generation from Multiple High-resolution Spherical Images.
Wilczkowiak et al. 3D modelling using geometric constraints: A parallelepiped based approach
Rousso et al. Robust recovery of camera rotation from three frames
Demirdjian et al. Stereo autocalibration from one plane
Bartelsen et al. Orientation and dense reconstruction from unordered wide baseline image sets