JP2002516443A - Method and apparatus for three-dimensional display - Google Patents

Abstract

(57) [Summary] An apparatus for generating a three-dimensional display of an object (3) is freely movable with respect to a further viewpoint (CM ′) and acquires an individual overlap image at each viewpoint. Equipped with a camera (CM). The orientation of the camera is kept approximately constant between the two viewpoints, and after correlating the images, an almost three-dimensional display can be obtained, for example, by Gruen's algorithm, in a computer (4), using a vector (V , FIG. 3). In another embodiment (FIGS. 13 and 15), the object is illuminated with a predetermined optical radiation, and a three-dimensional representation is derived from the correlation between the image of the object and a target acquired at a common viewpoint. A method for moving the camera (FIGS. 4 to 6) is also disclosed.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

TECHNICAL FIELD OF THE INVENTION

The present invention relates to a method and an apparatus for obtaining a three-dimensional representation from two or more two-dimensional images.

[0002]

[Prior art]

As is well known, generally, such a three-dimensional display can only be generated if the configuration (eg, points) of the images is correlated. The composition of each of the two overlapping images is the same composition of the object (eg, a point)
(Ie, if combined with the configuration). If the position and orientation of the camera from which the overlap image is obtained are known, the overlap (assuming the camera's geometry (specifically, its focal length)) is known, The three-dimensional coordinates of the object in the area can be determined.

[0003] Hu et al., "Matching Point Features with ordered Geometric, Rigidity an
d Disparity Constraints "IEEE Transactions on Pattern Analysis and Machi
ne Intelligence Vol 16 No 10, 1994 pp 1041-1049 (and references cited therein) disclose a suitable algorithm for correlating the composition of overlapping images. Such an algorithm can be used in the present invention and will not be described further. However, in some circumstances (e.g., where the object is relatively simple or one of the known algorithms allows one to accurately reconstruct the part of the object to be obtained When sufficient embodiments cannot be correlated, the configuration of the overlap image can be correlated visually with the method and apparatus embodiments of the present invention, and the correlation can be displayed on a screen. Note that it can be recorded.

[0004] Assuming that the correlated points of the image have been obtained, in principle, information about the position and orientation of the camera needed to reconstruct the object in three dimensions can be obtained directly. It can be obtained by any of a variety of advanced mathematical techniques, including measurement, or processing of all correlated sets of configurations. (Three-dimensional camera arrangements (where the camera is fixed) will not be considered for the purposes of the present invention.)

[0005] EP-A-782,100 discloses a method and apparatus in a first category, namely a photographic 3D acquisition arrangement, in which camera displacement signals are described. And lens position signals are used to enable a three-dimensional display of an object to be constructed from a two-dimensional image obtained by a digital camera. Both the position and orientation of the camera are monitored. The hardware needed to accomplish this appears to be expensive.

[0006] Numerous research articles have addressed the more difficult problem of obtaining the correct orientation from correlated configurations of images, such as: i) "A rational Algebraic Formulation of the problem" by EH Thompson
of Relative Orientation "(Photogrammetric Record Vol 3 No 14 (1959) pp15
2-159) describes a mathematical procedure for aligning two images of the same scene in the correct orientation to reconstruct a three-dimensional scene (iteratively calculates five simultaneous equations obtained from five sets of correlation points). To solve the problem). ii) "Stereo from uncalibrated cameras" by Richard Harley et al. (Proc. IEE
E Conf Computer Vision and Pattern Recognition (1992) pp761-763) teaches that two 3.times.4 camera matrices define camera orientation and position, as well as camera internal parameters such as focal length. ing. Once the camera has been calibrated (ie, if the intrinsic parameters are known), the camera matrix can be found from the matched points, and thus the true three-dimensional position for all points. If the camera is not calibrated, it is necessary to use known "ground control" points in order to obtain a camera matrix and thus a three-dimensional position with respect to the matched points . iii) "Estimation of Relative Camera Positions for U" by Richard Harley
ncalibrated Cameras "(Computer Vision-ECCV'92, LNCS-Series, Vol 588, 1992
pp579-587) is an extension of Harley's paper mentioned above, where if all other camera parameters are known, finding the focal length in addition to the camera position and orientation from the matched point, if known. Indicates that it can be done. iv) "Computing Matched-epipolar projections" by Hartley et al. (Computer V
ision and Pattern Recognition 1993, pp. 549-555) is an extension of the above paper, and is used to convert images into images obtained by cameras juxtaposed with their optical axes parallel. A conversion matrix has been introduced. The remaining points can then be more easily correlated. Furthermore, there is no need to calibrate these cameras.

[0007] A disadvantage of the above mathematical method is the intensive computation required to process all correlated points. Many of these calculations require determining the position and orientation of the camera from a large number of points and are very wasteful. The reason is that the position and orientation are significantly over-determined by a large number of processed points.

In general, all information from the camera's viewpoint (ie, azimuth and position)
It is shown that eight sets of correlated points can be found. However, there are special cases where this is not possible. An extreme example is when the eight points on the object corresponding to the set of correlated points are all colinear.

Significantly, another case where the position of the camera cannot be determined is when the orientation of the camera (in the reference frame of the object) is the same. As will be apparent from FIG. 3 (discussed below), separating the cameras in such a case is very indeterminate, no matter how many sets of points are correlated. In such a case, even if the focal length and other camera parameters are known, the size of the object cannot be determined from the two images, and the scaling (scaling
factor) must be increased.

The resolution of many digital cameras is severely limited (eg, 640 ×
Taking into account (up to 480 pixels), it is clear that a slight difference in the orientation of the camera leads to a considerable uncertainty in the position of the camera and thus also in the three-dimensional object reconstruction. It is. On the other hand, if the orientation of the camera is carefully selected to be very different (ie, converge sharply to the object), the overshoot present in the field of view when a certain image is obtained Hang area (over
Points on the object in the hanging region will not be present in the field of view if another image is obtained, hindering the three-dimensional configuration of the overhang region.

[0011]

[Problems to be solved by the invention]

It is an object of the present invention to overcome and mitigate at least some of the disadvantages of known methods and devices, especially when the resolution of the image in which the three-dimensional reconstruction is generated is limited.

[0012]

[Means for Solving the Problems]

Accordingly, in one aspect, the invention provides a method for obtaining a three-dimensional reconstruction of at least a portion of an object from correlated two-dimensional overlapping images of the object obtained from various viewpoints remote from the object; It is not known exactly how to separate the viewpoints, and the method uses a mutual offset and a scaling variabl between the respective two-dimensional image correspondences.
e) digitally processing the two-dimensional image to form a three-dimensional display extending in the simulated three-dimensional space in response to both; Represents the interval between.

The present invention further provides an image processing apparatus for obtaining a three-dimensional representation of at least a portion of an object from a correlated two-dimensional overlap image of the object obtained from various viewpoints separated from the object. However, the device was simulated as a function of both the mutual offset and magnification variables between the respective two-dimensional image correspondences.
Image processing means arranged to digitally process the two-dimensional image to form a three-dimensional display extending in the three-dimensional space, wherein the magnification variable represents an interval between viewpoints from which the two-dimensional image was obtained. .

Preferably, the magnification variable is input by a user during use.

[0015] For example, a virtual projector having an actual camera spacing a and having the same optical parameters as the camera and a 'spacing in a simulated three-dimensional space is partially If three-dimensional reconstruction occurs, a
The scaling factor for enlarging the partial three-dimensional reconstruction by the factor of / a 'is a / a', and this can cause a partial three-dimensional reconstruction having the same size as the object. it can. Such a partial three-dimensional reconstruction can be adapted to other full-scale partial three-dimensional reconstructions similarly generated from other sets of images. In these embodiments, any value for the magnification that allows the partial 3D reconstruction to be adapted together is satisfied and, for example, during the process of filling the partial 3D reconstruction together on the screen, the user May be applied.

In these embodiments, the orientation of the camera (in the reference frame of the object) in each of the two viewpoints is preferably the same or nearly the same (eg ± 10 °) and / or Is preferably generated by a virtual projector having the same orientation as one or more cameras and having an optical center on a line joining the optical centers of the cameras.

Preferably, the method comprises the step of obtaining a two-dimensional overlap image from a camera moved between different viewpoints on the object. The net movement of the camera between viewpoints is not completely constrained.

For example, for a camera to move in a direction transverse to its optical axis (its azimuth and position along two axes are restricted, but its movement along a third axis is not restricted). Can be mounted on a fixed slide or, for the camera, mounted on a tripod (so that the movement of the camera in the vertical direction and the rotation around the horizontal axis are restricted). .

However, in a preferred embodiment, the camera is held by hand.

Optionally, the orientation of the camera is described in, for example, an inertial sensor, such as a vibrating gyroscope, and in GB 2,292,605, which is incorporated herein by reference. It can be measured by appropriate filters and integrated circuits as disclosed. At present, Kalman
A) A filter is preferred for filtering the inertial sensor output signal.

Preferably, the orientation of the camera changes after the first image is obtained from one viewpoint and before the second image is obtained from another viewpoint, whereby the second image is obtained. Sometimes, the orientation of the camera with respect to the reference frame of the object is maintained.

One purely optical method of ensuring that the orientation of the camera does not change with respect to the orientation at the first viewpoint is the correlated points of the first image and the corresponding points of the second image. And changing the azimuth until the projections in the camera's image plane for (preferably displayed on the screen instantaneously) converge at a common point. In the special case where the distance from the image plane along the optical axis of the camera to the object does not change between the two viewpoints (if the common point is infinite), the projection is parallel Until then, the orientation in the second viewpoint can be adjusted.

The present invention further provides a method and apparatus for obtaining the shape of a three-dimensional representation of an object from an image of the projection of the predetermined optical radiation onto the object surface (also referred to herein as an object image). About. The term "predetermined optical radiation" is a generalization of the term "predetermined light" and includes not only predetermined light but also predetermined electromagnetic radiation of other wavelengths according to the laws of optics. Is intended for.

In principle, for a three-dimensional shape of a part of the object surface, projecting a predetermined light (eg a grid pattern onto the object surface) and obtaining an image of the illuminated area of the object surface And the composition of each of the images (eg, crossed lines)
Can be obtained by identifying a predetermined light element (e.g., an intersection line of a grid pattern) corresponding to, and assuming that the predetermined light spatial distribution is known.

One such arrangement is described by Hu & Stockman in “3-D Surface Solution Usi
ng Structured Light and Constraint Propagation "in IEEE Trans PAMI Vol 2
No 4 pp390-402, 1989, they discuss the advantages of their technology over stereoscopic imaging technology.

In a second aspect, the present invention provides a method for obtaining a three-dimensional representation of at least a portion of an object from a two-dimensional image thereof, the method comprising the steps of: Illuminating, obtaining a two-dimensional image of the illuminated object, correlating the two-dimensional image with a beam of predetermined optical radiation, and simulating 3 in response to both the correlation and magnification variables. Digitally processing a two-dimensional image to form a three-dimensional representation extending in a three-dimensional space, wherein the magnification variable comprises:
It represents an interval between a starting point position at which a predetermined optical radiation is projected and a viewpoint at which a two-dimensional image is obtained.

In this aspect, the present invention further provides an image processing apparatus for obtaining a three-dimensional display of at least a part of an object from a two-dimensional image of an illuminated object, wherein the object obtains a two-dimensional image. Illuminated by a predetermined optical radiation projected from a location remote from the viewpoint, a two-dimensional image is correlated with the predetermined radiation, and the device is simulated in response to both the correlation and magnification variables 3 extends in three-dimensional space
Digital processing means arranged to form a two-dimensional representation, wherein the magnification variable represents the distance between the starting position at which the predetermined optical radiation is projected and the viewpoint from which the two-dimensional image is obtained.

This feature can be achieved by using the camera perspective in the method and apparatus of the first aspect.
e) that the center spacing of e) does not need to be known, and that in the method and apparatus of the second aspect, the center spacing of the perspective of the camera and the center of the perspective of the projector need not be known. 1 is related to the first feature. There is a clear similarity between the camera-camera arrangement used in the first aspect and the camera-projector arrangement used in the second aspect. In each case, a three-dimensional representation can be obtained from the intersection of the two projections, in one case representing the respective luminous flux of the camera rays, and in the other case, of the projector rays and the camera rays. Each luminous flux is represented.

For example, the calibration image may be due to the projection of predetermined optical radiation onto the calibration surface, or project the predetermined optical radiation with the camera used to obtain the initial object image. Further object images obtained after the object has moved with respect to the projection means used for this purpose.

The first and second projections are baselines linking the origin of the predetermined optical radiation and the center of the perspective associated with the image (eg, the optical center of the camera lens used to obtain the image). Preferably, the reconstruction processing means is arranged to obtain said baseline from two or more sets of correlated configurations. This configuration is shown in FIGS. 3 and 13 described above.

In one embodiment, a correspondence between two or more calibration images is generated and the correspondence between the two or more calibration images and the input or stored associated with these calibration images The image processing means is arranged to determine a distance between the origin of the first projection and the origin of the second projection according to both of the metric information. This configuration is shown in FIG. 15, which will be described in detail later.

In another embodiment, the reconstruction processing means is arranged to change a distance between the origin of the first projection and the origin of the second projection according to a magnification function that can be input by a user. In this embodiment, no additional calibration images are needed. Preferably, the device comprises means for displaying a three-dimensional display with a relative magnification depending on the value of the magnification function.

In a further feature, from an object image relating to the projection of the predetermined optical radiation onto the object surface and at least one calibration image relating to the projection of the predetermined optical radiation onto the surface shifted from the object surface, Providing a method for generating a three-dimensional representation of an object, the method comprising: i) correlating at least one calibration image with an object image and, optionally, further with a further calibration image; ii) object image Simulating a first projection of the first optical projection and a second projection of the predetermined optical radiation; and iii) obtaining a three-dimensional reconstruction from the intersection of the first and second projections. .

In a related aspect, the invention relates to an object image for projecting predetermined optical radiation on an object surface and at least one object for projecting predetermined optical radiation on a surface shifted from the object surface. An image processing apparatus for generating a three-dimensional representation of at least a portion of an object from a calibration image, the apparatus comprising at least one calibration image, an object image, and (optionally, a further calibration image). Simulating image processing means arranged to generate a correspondence between: a first projection of the object image; a second projection linking a respective one of the at least two correlated images; and , Reconstruction processing means arranged to obtain said three-dimensional representation from the intersection of the first and second projections.

Preferred features of the invention are defined in the dependent claims.

In another aspect, the invention provides an image processing device for obtaining a three-dimensional representation of at least a portion of an object from a two-dimensional image of the object, wherein the object is spaced from a viewpoint from which the two-dimensional image is obtained. Illuminated by a predetermined optical radiation projected from a given location, a two-dimensional image is correlated with the predetermined radiation, and the apparatus is simulated in a three-dimensional space responsive to both the correlation and magnification variables. Digital processing means arranged to form a three-dimensional representation extending at a distance, wherein the magnification variable is the distance between the origin position at which the predetermined optical radiation is projected and the viewpoint at which the two-dimensional image is obtained. Represent.

This aspect of the invention is illustrated in FIG. According to one calibration procedure that does not require any information about the position of the camera or the projector with respect to the object, it is possible to generate a three-dimensional display. For a three-dimensional display, it can be selectively displayed and scaled, or, for example,
Can be distorted for special effects in graphics and animation.

[0038] The device is capable of obtaining three-dimensional images from various viewpoints of the object.
A combining means arranged to obtain a three-dimensional representation, wherein the combining means comprises a first three-dimensional representation and a further three-dimensional representation by operation in a simulated three-dimensional space with one or more rotations and transformations. Wherein the apparatus scales one 3D reconstruction along one axis with respect to the other 3D reconstruction so that any remaining defects between the 3D reconstructions Preferably, the apparatus further comprises scaling means arranged to reduce or eliminate the following.

The device is arranged to display both three-dimensional displays simultaneously and to operate these three-dimensional displays in a simulated three-dimensional space in response to commands entered by a user. Is done.

In one embodiment, the device is arranged to perform a three-dimensional reconstruction operation under the control of a computer pointing device.

The device comprises means for combining two or more three-dimensional reconstructions, and means for adjusting the relative magnification of these representations, in order to allow their representations to match each other. Is preferred.

In another embodiment, the variable is an incorrect positioning of the virtual projector (eg, a misalignment with respect to the camera viewpoint (misa), as shown in FIGS.
One or more distortions of the partial three-dimensional reconstruction that may result from the lignment are corrected either in the lateral direction or in the depth direction (field curvature).

In one embodiment, the partial three-dimensional reconstruction is distorted by intentionally displacing one or both of the virtual projectors with respect to the camera or the camera and the projector.

Such a configuration is useful in the fields of design, graphics, and animation.

Some differences in orientation can be tolerated, and the resulting distortion in the three-dimensional reconstruction can be corrected, as described below.

The smaller the difference in orientation, the smaller the distortion in three-dimensional reconstruction. Ideally, the orientation of the camera is kept constant between the two viewpoints.

Preferably, in the corresponding configuration of the object, the angle opposed by the set of correlated configurations is 90 ° ± 30 ° (more preferably ± 10 °). Ideally, the opposed angles are exactly 90 °. This configuration allows for more accurate correction of any distortions resulting from slight orientation changes.

According to the partial reconstruction according to the method described above, complementary three-dimensional reconstructions for various parts of the object, which are likewise obtained from a further set of overlapping images, can be filtered together.

As long as the reconstruction is distorted, simple corrections can be applied to the distortion parallel to the image plane and the curvature of the field in order to be able to adapt the three-dimensional reconstruction.

The representation of the reconstruction in the simulated three-dimensional space is preferably shown, for example, on a screen, and the variables are changed by the user in response to the representation shown on the screen .

[0051] The image processing means may include a local radiographic distribution of the image.
It is preferably arranged to generate the correspondence of the images by comparing the corresponding images.

In a related aspect, the present invention provides a method for obtaining a three-dimensional representation of at least a portion of an object from a two-dimensional image thereof, the method comprising illuminating the object with a predetermined projected optical radiation. Performing a two-dimensional image of the illuminated object;
Correlating the two-dimensional image with a beam of predetermined optical radiation and digitally processing the two-dimensional image to form a three-dimensional representation extending in a simulated three-dimensional space in response to both the correlation and magnification variables. And wherein the magnification variable represents a distance between a starting position at which a predetermined optical radiation is projected and a viewpoint at which a two-dimensional image is obtained.

For example, Gruen's algorithm (Gruen, AW “Adaptive least square”
s correlation: a powerful image matching technique "S Afr J of Photogram
metry, remote sensing and Cartography Vol 14 No 3 (1985) and Gruen, AW a
nd Baltsavias, EP "High precision image matching for digital terrain mo
del generation "Int Arch photogrammetry Vol 25 No 3 (1986) p254), suitable algorithms for correlating overlapping images (generating correspondence between overlapping images) are known, and Otto and Chau "Reg
ion-growing algorithm for matching terrain images "Image and Vision Comp
uting Vol 7 No 2 May 1989 “region-growing
It is also known to make modifications to the present invention, and these articles are all incorporated herein by reference.

In essence, Gruen's algorithm is a compatible least-squares correlation algorithm in which two image patches, usually 15 × 15 to 30 × 30 pixels, have coordinates in these images. To allow for affine geometric distortion between them (ie, stretching and compression, where the originally parallel lines remain parallel due to this transformation) and further radiation between the gray levels of the pixels in the image patch Are correlated by tolerating dynamic distortion, generating an overconstrained set of linear equations that represent inconsistencies between correlated pixels, and finding a least-squares solution that minimizes inconsistencies ( (I.e., selected from the larger left and right images in a way that gives the most consistent matching between patches)
.

The Gruen algorithm is essentially an iterative algorithm, and requires a reasonable approximation for the correlations to be provided to the algorithm before it converges to the correct solution. The region growing algorithm by Otto and Chau
Starting with approximate matching between points in one image and points in the other image, Gruen's algorithm to produce more accurate matches and generate geometric and radial distribution parameters And using the distribution parameters to predict approximate parameters for points in the neighborhood of the initial matching point. Neighboring points are selected by selecting neighboring points on a grid having a grid spacing of, for example, 5 or 10 pixels to avoid running Gruen's algorithm on all points.

Hu et al., “Matching Point Features with ordered Geometric, Rigidity an
d Disparity Constraints "IEEE Transactions on Pattern Analysis and Machi
ne Intelligence Vol 16 No 10, 1994 pp 1041-1049 (and references cited therein) disclose a further method for correlating the composition of overlapping images.

The above algorithm has been developed to generate correspondence between images (eg, aerial photographs) that have few defined configurations, while projecting a given light onto an object surface In the present invention, the problem of correlation is less important because it generates a separate local radiation distribution. Therefore, the exact correlation algorithm is not important. However, a number of improvements have been found over the Gruen algorithm, including: i) the additional radiometric shift used in the algorithm
shift) is not required. ii) During successive iterations, if the candidate matched point moves more than a certain number (eg, 3 pixels or more) per iteration, this point is not a valid point, and
Should be rejected. iii) While growing the matched region, is there sufficient contrast on at least three of the four sides of the region to ensure that there is enough data for stable convergence? It is useful to match
To facilitate this, it is desirable to make the algorithm configurable to allow parameters (eg, required contrast) to be optimized for different environments. iv) Applying an algorithm to the original grid points in the starting image to the points matched to the other image in order to determine the amount of validity of the correspondence between the images (ie the stereo matching process (stereo matching process) matching process
)) And that it is useful to obtain again by measuring the distance between the original and new grid points found in the image at the start from the inverse stereo matching. It is known. The shorter the distance, the better the response.

It is not necessary to correlate all possible configurations before determining the camera's viewpoint on the object (in fact, in many cases this is computationally inefficient)
. Typically, once a viewpoint is determined from a small number of sets (eg, eight) of correlated configurations, by searching for additional correlated configurations along the epipolar line determined from the viewpoint determination step, Obtaining the remaining correlation between the composition of each image is easier.

Once the correlation and the position and orientation of the camera are known, 3
The dimensional configuration can be obtained by projecting each image from a projector having the same focal length and viewpoint (position and orientation) as the camera that obtained the image. The chief rays from the corresponding configuration of each image intersect in (virtual) three-dimensional space at the location of the configuration of the object.

Thus, in another aspect, the invention provides an object comprising projecting an object image obtained by a mutually aligned camera into a simulated three-dimensional space from an aligned virtual projector. For generating a three-dimensional display of the virtual projector, wherein the distance between the virtual projectors can be changed by a user.

In this aspect, the invention comprises two aligned projector means arranged to project object images obtained by mutually aligned cameras into a simulated three-dimensional space. Provided is an apparatus for generating a three-dimensional display of a moving object, wherein a distance between virtual projectors is changeable by a user.

The difference in the alignment of the virtual projectors (ie the angle between these projectors) is preferably less than 45 °, more preferably less than 30 °, and less than 20 ° (eg 10 °). Preferably, this angle is less than 5 ° (eg, less than 3 °). This configuration allows the overhang configuration of the object to be captured from both camera perspectives, and furthermore, besides the configuration correlation between the images, the line connecting the optical centers of the cameras at various perspectives. Make decisions easier.

The origin of each projection is preferably located on a line in three-dimensional space that connects the corresponding optical centers of the camera at the two viewpoints. As will be described below, referring to FIG. 3, this results in a scaled partial three-dimensional reconstruction of the object.

In one embodiment, the distortion parameters are entered by a user and
Applied to three-dimensional reconstruction. For example, a two-dimensional overlap image was projected from a nominal viewpoint of the two-dimensional overlap image such that the projections intersect to form an initial three-dimensional reconstruction in a simulated three-dimensional space. later,
An initial three-dimensional reconstruction configuration (the initial three-dimensional reconstruction configuration is a correlated configuration of the projected image such that it is on a projection of the configuration from one of the two-dimensional images) (Generated from the cross-projection of) can be rotated while the initial three-dimensional reconstruction is rotated, thereby creating a further three-dimensional reconstruction. FIG. 9 (
As shown in below), this can be used to generate a reconstruction that is parallel to the actual object surface (and thus a scaled copy of the object surface). The rotations and constraints described above correct for the lateral distortion caused by the lack of parallelism in the initial three-dimensional reconstruction and the three-dimensional reconstruction generated from a correctly aligned virtual projector.

However, as noted above, in many applications, it is desirable to distort the three-dimensional reconstruction with respect to the original object in order to obtain the desired technical effect.

A feature of the present invention with respect to obtaining three-dimensional reconstructions is that a pre-acquired image is associated with a set of correlated points associated with these images, no matter how these images are obtained. Note that this is applicable.

The viewpoint is calculated from at least two (preferably at least three) sets of correlated configurations, and a three-dimensional reconstruction of an object is generated in response to the calculation of the viewpoint, and Preferably, the viewpoint calculation is performed on a smaller number of sets of correlated configurations than all available sets.

This preferred configuration for the present invention is shown in FIG. 3 (described below), which
Figure 4 shows obtaining lines connecting the viewpoints from the luminous flux of the projections from the three sets of correlated configurations. The third projection from the third set of correlated P3, P3 'intersects the point VP already defined by the intersection of the other two projections (and thus the unaltered orientation between the viewpoints) Strictly speaking, it is not necessary to find a straight line connecting the viewpoints. Thus, this line can only be found from the two sets of correlated configurations, unless the camera orientation is changed. However, more accuracy is obtained if more than two sets of correlated configurations are processed.

Preferably, the calculation is performed on less than 1000 sets of correlated configurations, and on less than 50 sets (eg, 8 or less) of correlated configurations. More preferably, it is performed. For example, the calculations can be performed on four, three, or two sets of correlated configurations.

If the viewpoints are parallel or almost parallel (eg, within 10 ° of each other), the above arrangement preferably results in some savings in processing.

[0071]

BEST MODE FOR CARRYING OUT THE INVENTION

Further features according to the invention are set out in the dependent claims. Hereinafter, non-limiting preferred embodiments of the present invention will be described with reference to the accompanying drawings.

First, referring to FIG. 1, the present apparatus includes a personal computer 4 (for example, Pent
ium (registered trademark) PC). The personal computer 4 has a conventional CPU,
ROM, RAM, and hard drives, connection of input ports connected to the digital camera 1, video output ports connected to the screen 5, and conventional inputs connected to a keyboard and mouse 6 or other pointing device. And a port. A conventional OS such as Windows (registered trademark) 95 is stored in the hard drive.
And software are loaded. The software includes: a) for displaying an image captured by the camera 1; b) for correlating points in an overlap region of the image input from the camera 1; c) from the obtained image For detecting a camera movement line, d) transferring an image obtained by the camera from a projector arranged at an interval selected by a user (using a keyboard or a pointing device) on the movement line into a pseudo three-dimensional space. E) projecting (obtaining a partial three-dimensional reconstructed image), e) dimensioning the partial three-dimensional reconstructed image in one or more axial directions, and reconstructing such a partial three-dimensional reconstructed image in FIG. 13) for providing the lateral distortion and field curvature shown in FIGS. 9 and 10A-10C,
Or to compensate.

The software for executing the function a) may be any suitable graphic program, and the software for executing the function b) may be IEEE Tra
nsactions on Pattern Analysis and Machine Intelligence Vol. 16 No.10, 19
Hu et al., “Matching Point Features with,” pp. 1041-1049, 1994.
An algorithm based on the algorithm disclosed in “Ordered Geometric, Rigidity and Disparity Constraints” can be used. One suitable algorithm is Grün
Algorithm.

Referring to FIG. 1, the camera CM is a Pulnix M-9701 progressive scan digital camera, which may be hand-held or fixed to a tripod T or other support device (as shown at 1 ′). May be. The camera CM includes a three-axis vibratory gyroscope G having a filter circuit (currently a Kalman filter is preferable) and an integration circuit for generating signals arranged in three axes and displaying the signals on a screen. The first set of x-, y-, and z-axes is defined, and is used to acquire an image of the object 3 and display the image on the screen 5.
The origin of the xyz coordinate system is located at the optical center of the camera lens. Then, the camera is moved by an arbitrary distance to the new position CM ′, and a second image of the object 3 is acquired without changing the orientation of the camera (with respect to the xyz coordinate system). The operation of not changing the azimuth is realized by using an appropriate support device as needed. The check on the change in the azimuth is performed based on the convergence of the projection images of the corresponding points of the two images on the image plane I of the camera. The method will be described later with reference to FIG. For this purpose, all that is required is a corresponding number of pairs (eg 3
, 4, 5, and at most 10) points, which can be visually determined by the user from the image displayed on the screen, or can be determined by software of the computer 4. . As is evident from FIG. 4, the camera movement (ie, the xy crossing at the optical center of the camera lens located at the two positions)
A line segment in the z coordinate system) can also be identified at this stage. However, when a high-precision reconstructed image is not required, the user may estimate the moving direction of the camera.

The remaining corresponding points in the two images are calculated by the computer 4 (preferably using the information about the movement of the camera obtained in the previous step, for example, along the epipolar line of one image, Correlation is performed (in a manner of searching up to corresponding points), and a partial three-dimensional reconstructed image of the object 3 in a pseudo three-dimensional space is formed from the corresponding points, as described below with reference to FIG. Since the distance between position 1 and position 1 'is not known, the parameter corresponding to this distance is entered by the user using, for example, a keyboard or a pointing device 6. The computer 4 is programmed to display the resulting three-dimensional reconstructed image on the screen, and the user may interact with the three-dimensional reconstructed image in the xyz coordinate system until the partial three-dimensional reconstructed image matches the real object 3 The parameter can be changed with. Usually, the three-dimensional reconstructed image is expanded or compressed in the moving direction from the position CM to the CM 'as compared with the actual object.

Depending on the accuracy of maintaining the orientation of the camera between the two positions (for example, also using the orientation signal of the gyroscope G) and the accuracy with respect to the estimated or optically detected movement of the camera, Distortion occurs in the three-dimensional reconstructed image. This distortion can be corrected at this stage. The correction of the distortion (the distortion may be intentionally added) will be described later with reference to FIGS. 9 and 10A to 10C.

By moving the camera CM to a new viewpoint CMA, a further partial three-dimensional reconstructed image is generated. In this movement, the object is placed in the overlap area. That is, at least one point P in the field of view of the camera at the viewpoints CM and CM 'is held in the field of view of the camera at the viewpoint CMA. This movement involves a rotation θ about the y-axis (to obtain new axes x _i , y _i , z _i ) followed by a rotation φ about the x _i axis.
(Obtain new axes x ', y', z '), followed by a rotation κ about the z' axis, followed by new axial translations ΔX, ΔY, ΔZ. z '
The rotation κ about the axis is often zero, as shown in FIG.

After acquiring an image at the viewpoint CMA, the camera is moved to a new position 1A ′ to acquire and display a further image (adjustment is performed so that the direction of the camera does not change as in the case of the viewpoint CMA). ). Then, in the same manner as in the case of the viewpoints CM and CM 'described above, a further partial three-dimensional reconstructed image of the object 3 is generated by the computer 4.

Assuming that the origin of the x ′, y ′, z ′ coordinate system moves to the optical center of the camera at the viewpoint CMA to obtain new coordinate systems X, Y, Z, the new coordinate system XYZ The relationship between an arbitrary point XYZ and the same point xyz in the xyz coordinate system is given by the following equation.

(Equation 1)

Each term in the 3 × 3 matrix can be expressed by the cosine of the angle formed by the XYZ coordinate system and the xyz coordinate system (for example, Boas Mathematical Methods in
The Physical Sciences Pub John Wiley & Sons, 2nd ed., pages 437-438).

Therefore, the partial three-dimensional reconstructed image can be converted into a general coordinate system, and in order to minimize the difference between the partial three-dimensional reconstructed images in the overlapping area of the partial three-dimensional reconstructed image, The reconstructed image can be decompressed and compressed along all three axes, and a more complete reconstructed image of the object 3 can be generated.

As will be described in more detail later with reference to FIG. 19, in the present embodiment,
This operation is performed by a user displaying a three-dimensional reconstructed image on a screen and changing the relative expansion / compression of the reconstructed image along three axes.

In another embodiment, the partially reconstructed 3D images are combined using an iterative nearest neighbor algorithm without user intervention. Since this algorithm is publicly available, a detailed description of the algorithm will be omitted. However, in a nutshell, it can be done by finding one group of the 10 nearest neighbors located on each edge.
One surface edge is recorded. The surfaces are then rearranged to minimize the close distance between the sets of points, and a new set of closest points is then found. Therefore, the rearrangement step is repeatedly performed. Another method of correlating three-dimensional surface areas is disclosed in GB 2,292,605. According to one feature of the invention, the scaling factor or other variable is used to ensure that the three-dimensional reconstructed images can match each other to a consistent global surface plot of the object. It is created either by the user to adjust the relative size of the dimensional reconstructed image, or iteratively created under software control.

Returning to the description of the embodiment of FIGS. 1 and 2, further partial three-dimensional reconstructed images can be obtained from the other side of the object 3 as well, and a complete three-dimensional representation of the object is achieved. Can be combined with each other and / or with existing combinations of partial 3D reconstructed images. The method is summarized in FIG.

An overlapping image has been captured (step S 10). Multiple points (or larger features, such as multiple lines or multiple regions) are correlated (
At the same time as the step S20), the camera movement at least close to the observation position is determined (step S30). In the preferred embodiment, this has been determined with high precision by processing the two images. A partial 3D reconstructed image of the object surface is then created in an apparent 3D space using the computer 4 to process the correlated sets of points and camera movements (step S).
Together with 40), they are preferably interlocked on the screen by the user to give a coherent but possibly distorted (eg compressed or decompressed) three-dimensional representation (step S50). Is desirable. Optionally, it is distorted or undistorted by providing the appropriate compression or decompression (step S60).

In another embodiment, steps S 30 and S 40 can be combined using a matrix processing method, for example, a commercially available INTERSECT program created by 3D Construction Inc.

Hereinafter, step S40 will be described in more detail with reference to FIG. FIG. 3 shows the object 3 in the field of view of the camera CM at the position CM and the position CM ′.
The chief rays from points Qa, Qb, Qc located on the surface of the object pass through the optical center OC, OC 'of the camera at each position CM, CM', and each chief ray is projected on the image plane. . Therefore, a plurality of sets of points formed on the image plane from each of the points Qa, Qb, Qc (and many other points not shown) are corresponding parts and are correlated with each other by a known algorithm. be able to.

Further, if the pseudo projectors pr1 and pr2 having the same pointing direction, focal length, and other optical characteristics as a camera at each observation point are placed at the points CM and CM ′, then, their correlations will be changed. Light rays are projected into a pseudo three-dimensional space from a plurality of sets of points taken. A plurality of sets of these correlated points correspond to corresponding portions Qa, Qb, Qc. On the object surface corresponding to a set of associated image points. . . And intersect at the true position of all other points. The pseudo-projector functions as an optical projector of the image by being created on the image by the operator. Appropriate software routines for processing images in the required manner could be readily written by those skilled in the art.

A projector having the same pointing direction in the reference frame of the object (corresponding to the common camera pointing direction) is shown in FIG. 3, but this is simply because the vector V is obtained by the image processing method disclosed in FIG. This is a preferred feature that can be determined more easily. In principle, the above analysis is also appropriate for pseudo-projectors with different pointing directions, each corresponding to a different camera pointing direction. As mentioned above,
The pointing direction of the camera can be determined by an inertial sensor and an integrating circuit similar to that disclosed in the above-mentioned British Patent 2,292,605.

If the vector V connecting the optical centers of the camera / projector lenses located at the positions CM and CM ′ extends, and the projector pr2 (for example, the illustrated position pr2
If moved along this vector with its unaltered pointing direction (for ') and the same rays are then projected from them,
The rays from pr1 will intersect at different points in the pseudo three-dimensional space.
For example, the intersections at Qa, Qb, and Qc are replaced with the intersections at Qa ', Qb', and Qc ', respectively. This is due to the ray from Op ′ parallel to the ray from Op, which lies in the same plane as the ray from OC intersecting with the ray from Op and therefore intersects with the ray from OC. Will be. For example, the lines Op ′, Qa ′ lie in the same plane as the triangle OCOC′Qa and therefore intersect the line OCQa at Qa ′ in this case.

Therefore, if the projected image is obtained and the vector V is parallel to the line connecting the optical centers of the camera lenses at two positions where the projector has the same directivity as the camera, the reduced display 3 / 3 'can be created by several sets of pseudo-projectors on vector V. This final state adjusts the direction about the axis that is perpendicular to the vector V until the corrected pointing direction is not initially known, ie, each ray from several correlated points intersects. Can be satisfied by This procedure can be performed manually by the user with the aid of the proper display of images and rays on the screen, or can be performed automatically in software.

Further, a three-dimensional display of an object capable of once creating the direction of the vector V is known. FIG. 3 is referenced again in connection with a very similar method suitable for a pseudo-projector in connection with the projector-camera configuration of the present invention.

FIG. 4 shows an induction of the vector V (step S40 in FIG. 2).
. For ease of understanding, the camera (with the lens having the optical center O) is considered stationary and the object 3 is considered to move along the line M toward the position 3 '. . Points P1, P2, and P3 are points p1 and p when the object is at position 3.
2, p3, and when the object is at position 3 ', points p1', p2 ', p
Projected as 3 '. Assuming that the pointing direction of the camera relative to the object frame is not changed and only between the two positions, then Pn and pn '(where n = 1 to 3 in this figure, but in practice are larger values) Connecting lines L1, L2, L3
Intersect at a common point VP. This common point is somewhat homogeneous in the perspective view as the vanishing point.

It can be seen that the line connecting VP with the optical center O is the vector V. This vector is the trajectory of the optical center of the camera with respect to the object (ie, the desired trajectory of the pseudo projector).

In general, if the camera is hand-held, lines L1, L2, L3,. . . , Ln do not intersect at a common point due to a change in the direction of the camera between the two positions. However, the pointing direction of the camera can be changed by the user in the second position about the X, Y and Z axes, while on the screen only when these lines converge on or near a common point. And lines L1, L2, L3,... Determined visually or by a software routine on the obtained second image. . . , Ln. Indeed, images can only be obtained under the control of software routines when the required convergence is achieved.

If a gyroscope G is used, its three-axis pointing signal can be used to maintain the pointing direction of the camera between two viewing points.

Considering that the camera is moved and the object is fixed, the movement of the camera from its initial position is based on the image plane and the image plane (and image) of the camera at its first position. Superimposing the correlated images of the camera at its new position, lines L1, L2, L in the image plane of the camera at its first position
3,. . . , Ln, and tying the computed point of intersection VP to the optical center of the camera at its first position. The operation vector V is the movement of the object in the coordinate frame of the camera at the first position.

In a variant implementation of the above method, the moving image or rapid continuation of the further image is such that the (simulated) linear movement of the camera between each image or between successive frames of the moving image is as the camera moves. Lines L1, L2, L3,... In the camera image plane to obtain a point VP for each incremental movement. . . , Ln are obtained by the camera when obtained by projecting. As the direction of movement of the camera moves, the operation vector V changes direction for each incremental movement of the camera, and its segments integrate to determine all movements (including changes in pointing direction) of the camera. Can be.

If the object moves little or no in the depth (Z) direction, the lines L1, L2, L3,. . . , Ln are parallel or very close, and the VP is infinite or too far to be accurately determined.

FIG. 5 and FIG. 6 show this special case. The points A, B, C located at the corners of the object 3 in the initial position are projected by the lens L as triangles abc on the image plane of the camera. When the object moves to a new position 3 ', these angles A', B ',
C ′ is a triangle a ′ similar to triangle abc (in a narrow sense of geometry)
Projected as b'c ', but not for example enabled for rotation. For reasons explained below, it is initially assumed that points A, B and C are planes substantially parallel to image plane I.

As shown by surface A1B1C1, image abc and image a′b′c ′
Has also been obtained from a smaller object of the same shape located relatively closer to the camera.

Referring to FIG. 6, which shows the object 3/3 'as viewed by the camera (not its image), various possible surfaces ABC, A1B1C1, A2B2C2 are shown. For the sake of clarity only shown above, there is a continuous range of possible sizes for the surface ABC. However, all possible surfaces have a common centroid P.

If the object moves to a new position 3 ′ illustrated by the surface A′B′C ′, the centroid moves to the new position Q and a line PQ parallel to the image plane I (FIG. 5)
Represents the direction of movement regardless of the true size and distance of the object. In fact, this holds true even with some rotation about the Z axis (such that the lines AA ', BB', CC 'are not parallel).

However, if the camera is rotated in a second position about the Z axis ensuring that the lines AA ′, BB ′, CC ′ are in fact parallel, the analysis will be performed at points A, B
, C are not parallel to image plane I, for example, image abc is at point A shown in FIG.
Retains truth even if obtained from 1BC. After position 3 'is similarly shifted from centroid Q, the corresponding centroid (not shown) of A1BC in FIG.
Moved from the corresponding centroid of 'B'C'. However, the line connecting these centroids is parallel to the line PQ, and thus accurately indicates the direction of movement of the object to the camera.

Even if the camera moves in the Z direction so that the lines AA ′, BB ′, and CC ′ are not parallel and concentrate on a distant point, the line PQ accurately indicates the moving direction of the object 3. . Therefore, the method described above somewhat overlaps the method of FIG.

FIG. 7 shows an overall method for determining the moving direction of the camera. The first image is obtained (
With step S11), the camera is moved to a new position with the objects still in the field of view, and the first image is displayed on the screen 5 and superimposed with the instantaneous image seen by the camera at the new position. The corresponding part is obtained by the eye (ie, the user) or by an appropriate software routine (step S13). Depending on the processing speed and the required accuracy, a small number of points is required at this stage, ie 100 or even less, for example 10 points.

If it is clear to the user that there has been a movement in the Z direction, the method proceeds to step S
Branch to 14. In this step S14, the directional directions of the cameras are adjusted with respect to the X-axis, Y-axis and Z-axis until the points correlated with each other converge on a common "vanishing point" VP (FIG. 4). At this stage, a second image is obtained and stored in memory.

In step S15, a line is projected from the point VP through the optical center of the camera lens to find the trajectory V (FIG. 4), further correlation of the image is performed with the help of this information, and A three-dimensional reconstructed image has been created by the method shown in FIG. 3 (FIG. 17).

Step S15 also includes or overrides the selection of the camera model by the user (eg, from a list displayed on the screen). The computer 4 associates the camera models M1, M2,... With respect to the parameters of each model required for the required projection calculations. . . , Mn are programmed. The following parameters can be stored:

Camera model Mn: Pixel size in x direction Pixel size in y direction Focal length x dimension of film plane (in pixels) y dimension of film plane (in pixels)

If it is apparent to the user that there has been no significant movement in the Z direction (eg, as a result of a failure to find a reasonably close “vanishing point” VP), the movement of the object will be in the image plane. Before proceeding to step S17, a line PQ is assumed to be parallel and is found, for example, from an image of a group of three or more points located or close to a plane parallel to the image plane (step S16). Have been.

Because most digital cameras on the market generally have a limited resolution (eg, 1024 × 768 pixels, 640 × 480 pixels, or less), camera movement Determination, and hence the relative position,
An error almost inevitably occurs in the directional directions of the pseudo projectors PR1 and PR2 shown in FIG. The effect of this error is shown in FIG.

FIG. 8 is a ray diagram perpendicular to the image plane, showing the imaging of the lines of points P1, P2, P3 on the object surface by the camera at positions CM and CM ′.
For clarity, lens L is shown only at first point CM. Note that the camera orientation is the same at the two points.

If the virtual projectors pr1 and pr are arranged at the above positions in the above directions,
2 will correctly reconstruct the object as a reconstructed image 30. Projector pr2
Moves to pr2 'along the direction of the camera movement V (FIG. 4), the reconstructed image 30' formed thereby has no curvature of the image plane, also having the correct magnification in the plane direction (i.e., , Length ratio P1P2 / P2P3 = P1′P2 ′ / P2′P3
') Would indicate points (P1', P2 ', P3') on a straight line.

If the projector is continuously rotated to the new positions pr2 ″ and pr2 ′ ″, a continuously larger curvature is introduced into the reconstructed image 30 ″ or 30 ′ ″ as shown. Is done. The relationship between the curvature of the image plane of the reconstructed image 30 and the angular error of the orientation of the second virtual projector with respect to the first virtual projector will be discussed below with reference to FIGS. 10A to 10C.

However, referring to FIG. 9, where each ray line from the correlation point of the images projected by projectors pr1 and pr2 is substantially orthogonal,
An error in the relative orientation of the projector leading to the projection of the second image from r2 'leads to negligible curvature of the image plane and is defined by the intersection of the ray lines from pr2' with the corresponding ray lines from projector pr1. Line 30 '
It is substantially straight, such as line 30 defined by the corresponding intersection of the ray lines from projectors pr1 and pr2, which are precisely oriented together. Therefore,
The camera position is substantially 90 ° relative to the ray line from a pair of correlation points at the intersection at the corresponding point on the object surface, for example 90 ° ± 30 °, preferably 90 °.
° ± 20 °, more preferably 90 ° ± 10 °. This minimizes the correction required by the curvature of the image plane.

The distortion in the plane direction of the reconstructed image, that is, the ratio P1P2 / P2P3 = P1′P2
There remains a mismatch between '/ P2'P3' (FIG. 8). However, as shown in FIG. 9, the partial three-dimensional reconstructed image 30 ′ is rotated to the position 30 C so as to be located parallel to the image plane of the projector pr 1, and all the points are formed by the ray lines from the projector pr 1. Shift so as to be blocked, and then these shifted points and pr1
The triangles defined by the ray lines from are geometrically similar to the triangles defined by the corresponding points in line 30 and these ray lines,
As a result, the point distribution along the line 30C becomes a replica in which the magnification of the corresponding distribution along the line 30 is changed. It should be understood that this has nothing to do with the angle to the ray line at the intersection at the corresponding point on the object surface.

Thus, to generate a partially reconstructed image and to use an axis perpendicular to the image plane of the projector used to constrain points in the partially three-dimensional reconstructed image to place on the ray line from that projector By rotating the partially reconstructed 3D image, distortion parallel to the image plane can be corrected (or can be intentionally caused).

FIGS. 10A to 10C show the curvature of the image plane due to the angular displacement of one projector pr2 ′ from the correct orientation pr2. In each case, the correct reconstructed image 30 defined by pr1 and pr2 is flat, indicating the center of curvature CN of the actual reconstructed image 30 '(derived geometrically).
In FIGS. 10A to 10C, the misalignment is caused by rotating the perspective center by 5 °, 10 °, and 15 °, respectively.

The curvature (curvature) radius R of the reconstructed image 30 ′ is inversely proportional to the displacement as shown in FIG. In FIGS. 10A to 10C, the angle to the ray line at the intersection at the corresponding point on (the reconstructed image 30 of) the object surface is much less than the optimal angle of 90 °, so that the degree of curvature is usually usually obtained. Much larger than the ones.

Thus, the method of the invention described above allows for a considerable latitude in the orientation of the projector, which means that a considerable uncertainty in the relative orientation of the camera from two viewpoints is allowed. .

The above corrections can be made at any stage during the generation of the partially reconstructed 3D image or during the fitting of each other.

However, it should be noted that only a limited displacement of the virtual projectors pr1 and pr2 (FIG. 3) is allowed. Referring to FIG. 3, all chief ray lines projected from the correlation point intersect only if the projector has the correct orientation. Latitude in azimuth is that the ray of light from the projector has a finite thickness,
Or it comes only from the finite resolution of the camera, which means it doesn't intersect exactly. Therefore, if the length of the shortest line does not exceed the predetermined limit corresponding to the resolution of the camera, the ray line from each projector is added to determine the center point of the shortest line on the reconstructed image 30. Can be determined.

A projector-camera embodiment very similar to the camera-camera embodiment of FIG. 1 will be described with reference to FIG.

Referring to FIG. 12, the device comprises a conventional CPU, ROM, RAM, a hard drive, an input port to the digital camera CM, a video output port connected to the screen 5 and a conventional keyboard connected to a keyboard. A personal computer 4 having a frame capture device and a mouse 6 or other pointing device connected to an input port of the mold. The hard drive loads a conventional operating system and software such as Windows 95: a) displaying images taken by the camera CM; b) mapping between overlapping areas of image input from the camera CM. C) deriving a baseline from the captured image and adding the perspective center of the camera and projector; d) a base selected by the user (using a keyboard or pointing device) or at a predetermined interval. Projecting an image taken by a camera into a simulated three-dimensional space from a virtual projector located on the line; e) changing the magnification of the partially three-dimensional reconstructed image along one or more axes,
Combining the partially reconstructed images as shown in FIG. 18, FIG. 18 and FIG. 19; f) determining the distance between the perspective center of the camera and the projector along the baseline from further correlation of the object image and the calibration image. Determining and thereby deriving an accurate partial 3D reconstructed image of the object surface.

Also, prior to processing the image as described above as part of the light flux adjustment step during the initial calibration step or during processing of the object image and the calibration image, for example, due to curvature of the image plane of the camera and projector optics It is preferable to provide software for correcting the image with respect to the generated distortion. Appropriate correction and calibration procedures are described by Tsai in “An Eff
icient and Accurate Camera Calibration Technique for 3D Machine Vision ”
Proc. It is described in IEEE CVPR, pages 364-374, and would not be otherwise.

A camera, the Parnix M-9701 progressive scan digital camera, is mounted on one end of the support frame F, for example on a ball and socket table, and the slide projector PR is securely mounted on the other end of the frame. As shown. The slide projector PR includes a speckle pattern slide S,
It is arranged on the surface of the region R of the object 3 in the field of view of the camera CM for projecting on the resulting speckle pattern.

The intrinsic camera parameters are determined by first obtaining an image of a reference plate (not shown) at a known location. The reference plate is flat and has a constant and well-spaced array of printed blots. It is assumed that the following parameters have been determined and are thereby known in the following description. i) camera focus ii) distortion parameters of camera and projector lenses iii) magnification iv) image coordinates of principal points

It is also assumed that the pixel size (determined by the camera operator) is known.

Optionally, determine the following external camera parameters: a) Camera position b) Camera orientation Alternatively, the camera position and orientation are used to determine the coordinate system involved in determining object plane coordinates. You can also use it.

Referring to FIG. 2, the camera CM is shown with the perspective center OC located on the baseline vector V, showing the (first) target surface T and the (subsequent) object 3. The (virtual) origin or perspective center OP of the projector PR is also on the baseline vector V, and is defined by the optical system of the projector including the field lens OL and the condenser lens CL. A point light source LS, such as a filament bulb, illuminates the slide S, causing a speckled pattern to be (first) on the target surface T and (after)
A speckle pattern is oriented on the surface of the object 3.

The baseline vector V is found by the following procedure: First, the image 11 (FIG. 14) of the surface area of the object 3 illuminated by the projected speckle pattern is taken into the memory of the computer 4 Any group of at least two spatially separated points Q1 and Q2 in this region is selected as points q1 and q2 in the image formed on the photodetector plane PD of the camera. . The group of points q1 and q2 is stored.

Second, the object 3 is replaced by the target plane T, and an image 12 (FIG. 3) of the illuminated area of the target plane is taken by the camera CM. The position and orientation of the target T with respect to the camera can be found by utilizing a known pattern of spots BL formed on the periphery of the target and obtaining an image of the target without any illumination from the parameters. The image 12 is stored and also compares the local radiometric intensity distribution by selecting the surrounding region R of the first 3 × 3 pixels and by the modified Gruens algorithm described above,
By searching for the corresponding region R ′ in the second image 12 that has been distorted by affine geometric distortion (for example, in the simple case shown in FIG. 14, when extending horizontally), the first image 11 The patch defined by the center point (eg, Qn in FIG. 3) correlates to the corresponding point Pn of the second image 12. The correlation patch is expanded (up to 19 × 19 pixels) and the process is repeated. In this way,
The corresponding point Pn is found.

This step is to find the majority of the corresponding PQ pairs (FIG. 14), and in particular P1
, P2 (FIG. 13) are repeated to point the patches in groups Q1, Q2 (FIG. 13) to points. The latter does not need to be centered on a special pixel since the algorithm has sub-pixel resolution.

In the geometric discussion below, correspondence is treated as a correlated pair of points for simplicity, but does not imply anything about its topography-especially that they lie at corners or edges of an object, for example. Note that this does not mean

Referring to FIG. 13, the origin (OP) of projector PR is P1Q1 and P2Q2.
At the intersection with However, the three-dimensional positions of these four points are not known, and they are the ray lines from a certain camera, ie, p1P1, q1Q1, p
Only 2Q2, q2Q2 are known. However, line P1Q1 is plane OCP1Q
1, the surface OCP1 available from the calibration step and the two images 11 and 12
Q1 and the line P2Q2 is at plane OCP1Q1, the plane OCP1Q1 also available from the calibration process and the two images 11 and 12. These planes, by their intersection, define a baseline vector V past the OC and the perspective center Op of the projector.

A particularly simple way of finding the baseline vector V is to project p1q1 and p2q2 that meet at point X on the plane of the photodetector PD. The baseline vector V is shown as the projection of point X through the perspective center OC is shown.

With this method, the position of the projector origin Op along this baseline cannot be determined, but the baseline vector V can be determined. In fact, the groups of points P and Q each comprise more than one pair, thereby overdetermining the baseline vector V. Therefore, such a vector bundle is determined as determined by the set of points PQ, and "outliers", i.e., vectors that deviate more than a predetermined threshold from the mean, are removed, and the remaining vectors, a well-known statistical method, And the vector V
It is preferable to include the computer 4 in order to perform the evaluation by the least square method.

The derivation of the three-dimensional representation of the object 3 is shown in the ray diagram of FIG. The camera CM and the projector PR located on the baseline vector V are shown. The first virtual projector pr1 is driven by the image processing software of the computer 4 and has the same optical properties as the camera (as determined in the initial calibration procedure). Image 11 (FIG. 14) is projected from this virtual projector in a three-dimensional space simulated by image processing software.

The second virtual projector pr1 is likewise driven by the image processing software of the computer 4, and preferably has the same optical properties as the projector PR (as shown in FIG. 3). This virtual projector has each physical projector ray PQ
Project a set of ray lines in a simulated three-dimensional space corresponding to each of the ray lines, as can be found in the image correlation process described with reference to FIG. Calling. Image 1
It will be appreciated that 2 and the target T define a set of rays starting at the perspective center OP of the projector and are equivalent thereto. Projector P
It is known that which ray line from R / pr1 intersects each ray line from the corresponding pixel in image 11, so that a point in three-dimensional space corresponding to each intersection can be found, whereby point Qa , Qb, Qc... Define the surface.

In fact, many ray lines do not intersect, and the best estimate of the corresponding three-dimensional surface is the center point of the vertical line that aligns them when they come closest. Algorithms for this purpose are known per se.

In the above discussion of FIG. 3, it has been assumed that the relative positions of the camera CM and the projector PR on the baseline vector V (and thereby the positions of the virtual projectors pr1 and pr2) are known. In fact, these have been assumed or entered by the user of the computer 4 as magnification variables.

Assuming that the ray line from each virtual projector does not change its orientation, regardless of the spacing between the virtual projectors, the intersection would result in an alternative virtual projector position corresponding to the assumed actual projector position PR ′ This is indicated in FIG. 3 by pr2 ′. The three-dimensional reconstruction of the object 3 'thereby differs in size but of the same shape: a single magnification is required to exchange the objects 3 and 3'. However, it is actually convenient to have different horizontal and vertical magnifications (eg, due to different horizontal and vertical magnifications of the camera), and generally only one set of magnifications is taken from different orientations of the same object. And consistently match each other with a given set of partial three-dimensional surfaces.

Thus, the software of the computer 4 changes the magnification (changes the scale) such that the obtained three-dimensional tables can be matched to one another so as to form a self-consistent three-dimensional representation of the indicating object. It is equipped with. This is shown in FIG.
This will be described below with reference to FIG.

However, before doing so, an alternative calibration procedure is described with reference to FIG. This figure shows that the orientation (and preferably the position) with respect to the camera axis system is to obtain those images without illumination from the projector and to process the images in a procedure similar to that described above in connection with FIG. FIG. 14 shows two flat calibration targets T1 and T2 (with peripheral spots or disks similar to the target T of FIG. 13), known as a result of photogrammetry, which independently include: Also shown are the perspective centers OC and OP of the camera and projector.

In the first stage of the calibration procedure, the target T1 is illuminated by predetermined light from the projector, and an image is taken by the camera CM. FIG. 15 shows three points p1, p2, and p3 where predetermined light is incident on the target T1. These (and many other points, not shown) are imaged by the camera CM.

In the second stage of the calibration procedure, the target T1 is removed and T2 is illuminated by a predetermined light from the projector. An image is taken by a camera CM. Three point points P1, P2 and P3 corresponding to points p1, p2 and p3 were previously obtained on T1 by the above procedure with reference to FIG. 13 and a newly obtained image of the predetermined radiation on target T2. It is found by correlating with the corresponding projection image.

FIG. 15 shows the positions of the two calibration targets T1 and T2 and the projector P
R shows the relationship of R (camera CM is assumed to be fixed on baseline vector V) to perspective center OP. Each pair of points P1 and P2 on the target T1 forms image points p1 and p2 on the photodetector array PD of the camera CM,
Also, the pairs of points P3 and P4 on target T2 (in the next stage following removal of target T1) form image points p3 and p4 on photodetector array PD, respectively.

Therefore, in the third stage, corresponding points on the targets T1 and T2 (for example, P1
And P3; P2 and P4) are constructed to find the position of the perspective center OP of the projector. In fact, the rays do not intersect at one point, but the best estimated point can be found with a least squares algorithm.

It should be understood that other calibration procedures are possible. For example, the camera uses the Tsai method (
IEEE Journal of Robotics an by Roger Tsai
d Automation RA-3, no. 32 of 4 (issued in August 1987)
See page 3 and the references cited therein).

【The invention's effect】 [Brief description of the drawings]

FIG. 1 is a schematic view showing an embodiment of a two-camera system according to the present invention.

FIG. 2 is a flowchart of one method according to the two-camera system of the present invention.

FIG. 3 is an optical path diagram showing a relationship among an object, a viewpoint of a camera and a projector, a viewpoint of a virtual projector, and a partially three-dimensional reconstructed image in one embodiment of the present invention.

FIG. 4 is a three-dimensional optical path diagram illustrating a method of detecting a moving direction of a camera from an acquired image in the method of FIG. 2 and the apparatus of FIG. 1 according to a feature of the present invention.

FIG. 5 is a three-dimensional optical path diagram showing a method of detecting a moving direction of a camera from an acquired image in a special case where the camera does not move in a Z direction with respect to an object in the method of FIG. 2 and the apparatus of FIG. It is.

FIG. 6 is a diagram illustrating movement of an image of an object on an image plane I of FIG. 5;

FIG. 7 is a flowchart summarizing the image processing steps used in the method of FIG. 2 and the apparatus of FIG. 1;

FIG. 8 is a two-dimensional optical path diagram showing the curvature of an image plane when one of the virtual projectors is misaligned with respect to another projector in the apparatus of FIG. 1 and the method of FIG.

FIG. 9 is a two-dimensional optical path diagram illustrating distortion correction in a partially three-dimensional reconstructed image according to an embodiment of the present invention.

FIG. 10A is a two-dimensional optical path diagram showing the curvature of the image plane when the virtual projector is displaced by 5 °, and FIG. 10B is a view showing the curvature of the image plane when the virtual projector is displaced by 10 °. FIG. 10C is a two-dimensional optical path diagram showing the curvature of the image plane when the virtual projector is displaced by 15 °.

FIG. 11 is a graph in which the curvature of an image is plotted with respect to the displacement states shown in FIGS. 10A to 10C.

FIG. 12 is a schematic view showing one embodiment of a projector-camera type apparatus of the present invention.

FIG. 13 is a perspective optical path diagram showing an example of an optical arrangement in the apparatus of FIG.

FIG. 14 shows an object image and a calibration image obtained by the apparatus of FIGS. 12 and 13;
FIG. 3 is a diagram illustrating a correlation between the two.

FIG. 15 is a perspective light path showing a modification of FIG. 13, that is, a state in which two reference planes arranged on a base line connecting the perspective centers of the camera and the projector are used for positioning the camera and the projector in the apparatus of FIG. FIG.

FIG. 16 is a flowchart illustrating a method of operating the apparatus of FIGS. 12 and 13 using the projector-camera method of the present invention.

FIG. 17 is a diagram illustrating a screen for explaining a method of synthesizing two three-dimensional surface portions of an object using the apparatus of FIGS. 1 and 2 or the apparatus of FIG.

FIG. 18 is a view of a screen showing a state where dimensions of the synthesized three-dimensional surface portion are adjusted in the vertical axis direction and the horizontal axis direction.

FIG. 19 is a diagram of a further screen showing a state in which intersecting three-dimensional surface portions have been dimensioned so as to fit each other.

FIG. 20 is an illustration of a screen provided by the apparatus of FIGS. 1 and 2 or 12 showing a user interface for manipulating images and three-dimensional surfaces.

[Explanation of symbols]

 1 ... Digital camera 3 ... Object 4 ... Computer 6 ... Pointing device

──────────────────────────────────────────────────続 き Continuation of front page (81) Designated country EP (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, NL, PT, SE ), OA (BF, BJ, CF, CG, CI, CM, GA, GN, GW, ML, MR, NE, SN, TD, TG), AP (GH, GM, KE, LS, MW, SD, SL, SZ, UG, ZW), EA (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), AE, AL, AM, AT, AU, AZ, BA, BB, BG, BR , BY, CA, CH, CN, CU, CZ, DE, DK, EE, ES, FI, GB, GD, GE, GH, GM, HR, HU, ID, IL, IN, IS , JP, KE, KG, KP, KR, KZ, LC, LK, LR, LS, LT, LU, LV, MD, MG, MK, MN, MW, MX, NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, SL, TJ, TM, TR, TT, UA, UG, US, UZ, VN, YU, ZA, ZW (72) Inventor Ivan Daniel Meier UK Buckinghamshire SL 9.8 JW Gellers Cross North Park 18 (72) Inventor Jonathan Anthony Holdback UK Ealing W5.2QN Montpellier Road Montpellier Court 2F Term (Reference) 2F065 AA04 AA53 BB05 FF04 FF07 HH05 JJ03 JJ26 QQ17 QQ24 QQ38 QQ41 SS02 SS13 5B050 BA09 DA07 EA05 EA07 EA12 EA13 EA18 EA27 FA02 5B057 BA02 CA12 CA16 CB13 CB17 DC05 CD11 CD05 CD11 080 BA05

Claims (52)

[Claims] 1. Different viewpoints (CM, CM ') spatially separated with respect to an object
A method for extracting a three-dimensional representation (30) of at least a part of an object (3) from a correlated overlapped two-dimensional image of the object obtained from, wherein the distance between the viewpoints is not exactly known The method comprises the steps of: forming a two-dimensional image in a simulated three-dimensional space to form a three-dimensional representation that depends on both the mutual offset and the scaling variable between the correspondences of the individual two-dimensional images; Digitally processing, wherein the magnification variable is indicative of an interval between viewpoints from which the two-dimensional image was acquired. 2. A method for deriving a three-dimensional representation (30) of at least a part of an object from a two-dimensional image of the object (3), said method comprising: illuminating the object with a predetermined projected optical radiation. Acquiring a two-dimensional image (11) of the illuminated object; correlating the two-dimensional image with a predetermined ray of optical radiation; and in a simulated three-dimensional space, a correlation and a magnification variable. Digitally processing the two-dimensional image to form a three-dimensional display that expands depending on both the magnification variable and the distance between the predetermined position and the predetermined viewpoint. And wherein a predetermined optical radiation is projected from the predetermined location, and a two-dimensional image is acquired at the predetermined viewpoint. 3. The simulated three-dimensional view of the display (30) is displayed, wherein the magnification variable is entered by a user. Method. 4. The method according to claim 1, further comprising the step of obtaining an overlapping two-dimensional image from a camera (CM) moved with respect to the object, wherein the net movement of the camera is not completely constrained. The method according to any one of claims 1 to 3. 5. The method according to claim 4, wherein the individual orientations of the cameras at different viewpoints with respect to a reference frame fixed with respect to the object differ by less than 45 degrees. 6. The difference between said individual orientations of the camera (CM) is 30
6. The method of claim 5, wherein the temperature is less than one degree. 7. The difference between said individual orientations of the camera (CM) is 10
7. The method of claim 6, wherein the temperature is less than one degree. 8. The method according to claim 4, wherein the two-dimensional image is acquired by a hand-held camera (CM). 9. The method according to claim 1, wherein at least one two-dimensional image is acquired by a camera (CM), and an orientation of the camera is determined based on an output signal of the inertial sensor (G). Item 9. The method according to any one of Items 8 to 9. 10. A three-dimensional representation (30) is generated by projection from a position on a straight line (V) in a simulated three-dimensional space, wherein the simulated three-dimensional space is an individual one of the two-dimensional images. Or a straight line connecting the individual transmission centers of said predetermined optical radiation with a two-dimensional image of the illuminated object (3). 10. The method according to any one of 9. 11. The simulated three-dimensional space (V) in the three-dimensional space is determined based on the luminous flux of the projection from the correspondence of the aligned two-dimensional images (I1, I2). The method of claim 10. 12. The method of claim 11, wherein the simulated three-dimensional space are linearly (V) includes a panel _{(O C P1O P, O C} P2O P) is determined based on the intersection of said panel, at least one transmission center ( O _C, and O _P), the method of claim 10, wherein a is defined by at least two pairs correspondence (PQ) between the acquired two-dimensional images (I1, I2). 13. The magnification variable is modified by a user to enable the three-dimensional display (R1) to be adapted to another similarly derived three-dimensional display (R2). The method according to claim 1, wherein: 14. A three-dimensional display (30) comprising a spatially separated transmission center (O _C).
, O based on the intersection of the rear projection from _P) is generated, the transmission center, the first pair between the respective two-dimensional images (I1, I2) corresponding (P
Q) and from a further mutual offset between the second pair of correspondences between the individual two-dimensional images, wherein the further pair correspondences are constrained by the transmission center determination. 14. The method according to any of the preceding claims, wherein the three-dimensional representation of the object is derived from a further search and a further pair correspondence and projection. 15. The method of claim 14, wherein the calculation is performed based on less than 1000 pairs of image correspondences (PQ). 16. The method of claim 15, wherein the calculation is performed based on less than 100 pairs of image correspondences (PQ). 17. The method of claim 16, wherein the calculation is performed based on less than 50 pairs of image correspondences (PQ). 18. The method of claim 17, wherein the calculation is performed based on eight or fewer pairs of image correspondences (PQ). 19. The method according to claim 1, wherein the calculation is performed on two pairs, three pairs or four pairs of image correspondences (PQ
19. The method according to claim 18, wherein the method is performed on the basis of: 20. The method as claimed in claim 19, further comprising the step of creating different three-dimensional representations with different further viewpoints (CMA, CMA ′).
Repeating the method of claim 1 or claim 2 by digitally processing a further two-dimensional image of the object obtained from the first three-dimensional representation and the further three-dimensional representation. , Simulated 3
Coupled by an operation in a three-dimensional space, the operation including one or more of rotation and translation, scaling one three-dimensional display with respect to another three-dimensional display along at least one axis. 20. A method according to any of the preceding claims, wherein any inconsistencies remaining during the three-dimensional representation are optionally reduced or eliminated. 21. At least two three-dimensional displays (R1, R2) are simultaneously displayed on a screen (5), wherein the operation of the two three-dimensional displays is performed in response to a command input by a user. 21. The method of claim 20, wherein: 22. The method according to claim 21, wherein the operation of the three-dimensional display (R1, R2) is performed under the control of a computer pointing device (6) operated by a user. 23. A method according to claim 20, wherein a distortion parameter is input by a user and is applied to the first mentioned three-dimensional display and / or the further three-dimensional display. The method according to any of the above. 24. Initial three-dimensional display in a simulated three-dimensional space (
30 ', FIG. 9) are generated by intersecting the projections from the spatially separated transmission centers and are on the projection of these features from the individual transmission centers of the initial three-dimensional representation. In turn, the initial three-dimensional display is rotated while constraining the points of the initial three-dimensional display generated based on the intersecting projections, thereby forming a further three-dimensional display (30C, FIG. 9). 24. The method of claim 23, wherein: 25. A further parameter indicating the curvature of the field is entered by the user and used to adjust the curvature of the field of the initially mentioned three-dimensional display and / or of the further three-dimensional display. The method according to any of claims 20 to 24, characterized in that: 26. An image processing apparatus for extracting a three-dimensional display of at least a part of an object from a correlated overlap two-dimensional image of the object obtained from different viewpoints spatially separated with respect to the object, The apparatus includes a two-dimensional image for forming a three-dimensional representation (30) that expands in a simulated three-dimensional space depending on both a mutual offset and a magnification variable between corresponding two-dimensional images. Image processing means (4) adapted to process in a digital manner, wherein the magnification variable indicates an interval between viewpoints (CM, CM ′) from which the two-dimensional image is obtained. Image processing device. 27. An image processing apparatus for extracting a three-dimensional display of at least a part of an object from a two-dimensional image of an illuminated object, wherein the object has a viewpoint (O _C) from which the two-dimensional image is acquired. ) Is illuminated with a predetermined optical radiation projected from a position (O _P ) spatially separated from the two-dimensional image, the two-dimensional image is correlated with the predetermined radiation, and the device comprises a simulated three-dimensional space , Comprising digital processing means (4) adapted to form a three-dimensional display that expands depending on both the correlation and the magnification variable, wherein the magnification variable is an interval between a predetermined position and a predetermined viewpoint. And a predetermined optical radiation is projected from the predetermined position, and a two-dimensional image is acquired at the predetermined viewpoint. 28. Display means (5) adapted to display a scene of a display in a simulated three-dimensional space, wherein the size of the displayed display depends on the value of a magnification variable. 28. The apparatus according to claim 26 or claim 27. 29. The apparatus further comprises a camera (CM), wherein the position and / or orientation of the camera is not completely constrained with respect to the frame of the object, and the camera may acquire the two-dimensional image. The image processing apparatus according to claim 28, wherein: 30. Apparatus according to claim 29, characterized in that the apparatus comprises inertial sensor means (G) adapted to determine the orientation of the camera with respect to the object when acquiring the two-dimensional image. 31. Apparatus adapted to generate a three-dimensional representation based on the intersection of individual projections from spatially separated transmission centers (pr1, Pr2 / r2 ′), wherein the transmission center determination is performed. In order to derive further pairs of correspondences from the search constrained by, and to derive a three-dimensional representation of the object from the further pairs of correspondences and projections, the transmission center is used to determine the first 31. The method of claim 26, further comprising deriving a mutual offset between a pair of correspondences and a further mutual offset between a second pair of correspondences between the individual two-dimensional images. An apparatus according to any of the preceding claims. 32. A device adapted to derive a further three-dimensional representation based on a further intersection of a further projection from a further transmission center (CMA · m, CMA ′), said device comprising: The three-dimensional display described in Section 3 and the further three-dimensional display
Coupling means (4) adapted to be coupled by an operation in a dimensional space, said operation comprising one or more of rotation and translation, said device comprising: Magnification adjusting means (BN, W1, W2) adapted to reduce or eliminate any inconsistencies remaining during the three-dimensional display by scaling one three-dimensional display with respect to the other three-dimensional display The apparatus according to any one of claims 26 to 31, further comprising: 33. Simultaneous display of both three-dimensional displays (R1, R2), operation of both three-dimensional displays in a simulated three-dimensional space in response to commands entered by a user. 33. The device of claim 32, wherein the device is adapted to: 34. A method for correcting distortion of said first mentioned three-dimensional representation and / or said further three-dimensional representation due to incorrect or incomplete calculation of said transmission center. Apparatus according to any of claims 32 or 33, characterized in that: 35. The first mentioned three-dimensional representation and / or the further three-dimensional representation (30 ′ diagram) due to incorrect or incomplete calculation of the transmission center (pr2′—FIG. 9). 35. Apparatus according to any one of claims 32 to 34, wherein the curvature of the field of 9) is corrected. 36. Camera (CM) for object (3) in the field of view of the camera
The movement of the feature of the image of the object to a common vanishing point (VP) (L1, L2, L3-FIG. 4).
And determining a vector (V) between the transmission center of the camera (O) and the vanishing point. 37. An apparatus for determining a movement of a camera relative to an object in a field of view of the camera, said apparatus projecting a path (L1, L2, L3-FIG. 4) of an image feature of the object to a common vanishing point. Means for determining the vector (V) between the transmission center (O) of the camera and this vanishing point. 38. A method for generating a three-dimensional reconstructed image of an object (3), the method comprising: aligning images of an object acquired by mutually aligned cameras (CM, CM '). Projecting the simulated three-dimensional space from the virtual projectors (pr1, pr2) caused by the virtual projectors, wherein the distance between the virtual projectors can be changed by a user. 39. An apparatus for generating a three-dimensional reconstructed image of an object (3), said apparatus comprising: a simulated 3D image of the object acquired by mutually aligned cameras. An apparatus comprising two aligned virtual projector means (pr1, pr2) adapted to project into a three-dimensional space, wherein the distance between the virtual projectors is changeable by a user. 40. An object image (I1) of a projection of a predetermined optical radiation onto an object surface.
) And the projection of predetermined optical radiation onto the surface transferred from the object surface.
Device for generating a three-dimensional representation of at least a part of an object (3) from one calibration image (I2), said device comprising at least one calibration image, an object image and optionally further calibration images An image processing means (4) adapted to generate a correspondence (PQ), and a second linking the first projection of the object image and the individual correspondence of at least two correlation images.
Reconstructing means adapted to simulate a projection and to derive said three-dimensional representation from the intersection of the first projection and the second projection. 41. The first projection and the second projection are projections from a baseline connecting the origin (O _P ) and the transmission center (O _C ) of a predetermined optical radiation associated with the image (I1, I2). 41. Apparatus according to claim 40, wherein the reconstruction processing means (4) is adapted to derive the baseline from a correlation. 42. An image processing means (4) adapted to correlate two or three or more calibration images, and between an origin of the first projection and an origin of the second projection (O _C , O _P ). 42. The method according to claim 40, wherein the distance is determined depending on a correlation of two or more calibration images and input distance information or storage distance information related to calibration. An apparatus according to claim 1. 43. The apparatus further comprising: projector means (PR) adapted to project the predetermined optical radiation onto the object surface and onto at least one calibration surface (T1, T2). 43. Apparatus according to any of claims 40 to 42, wherein the apparatus comprises: 44. The apparatus according to claim 40, wherein the apparatus further comprises a camera (CM) adapted to acquire an object image (I1) and at least one calibration image (I2). An apparatus according to any one of the above. 45. The apparatus according to claim 45, wherein in use, the at least one calibration target is illuminated by a predetermined radiation.
43. The apparatus according to claim 40, further comprising: (T1, T2). 46. The apparatus according to claim 46, wherein the local radiation distribution associated with the pixel is compared with individual locations.
Image processing means (4) adapted to correlate pixels in one image (I1) of the images with corresponding positions in another image (I2) of the images. 43. Apparatus according to any of claims 40 to 42. 47. Apparatus according to claim 46, wherein the image processing means (4) is adapted to tolerate radiation distortion and / or geometric distortion during the correlation process. 48. An object image (I1) of a projection of a predetermined optical radiation onto an object surface.
) And at least one calibration image (I2) of the projection of the predetermined optical radiation onto the surface transferred from the object surface, the method for generating a three-dimensional representation of the object (3). Wherein said method comprises the steps of: i) correlating at least one calibration image with an object image, and optionally correlating with a further calibration image; ii) first projection of the object image and predetermined optical radiation. Simulating a second projection; and iii) deriving the three-dimensional representation from an intersection of the first projection and the second projection. 49. The first projection and the second projection are projections from a baseline connecting a point of origin (O _P ) and a center of transmission (O _C ) of a predetermined optical radiation individually associated with the image; 49. The method of claim 48, wherein the baseline is derived from two or more pairs of image correspondences (PQ). 50. Two or more calibration images are correlated, and the distance between the first projection origin and the second projection origin (O _C , O _P ) is two or more calibration images. 50. The method according to claim 48 or claim 49, wherein the method is determined dependent on input distance information or stored distance information related to correlation and calibration. 51. The region (R) of the image (I1, I2) is correlated by comparing a local radiation distribution and / or a color distribution associated with the region. 50. The method according to any of the items 50. 52. The method according to claim 51, wherein radiation distortion and / or color distortion are tolerated between potentially corresponding regions (R).