JPH1139506A - Optional view point image generator - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce coordinate errors and to generate stable images no matter which spot of a space a view point is present at by transforming the coordinate of an affine coordinate system to an affine coordinate at a virtual view point position by a pinhole projection method and generating the image at the virtual view point position. SOLUTION: An image set A is photographed (50) and the reference vector of the affine coordinate is prepared (58). The image set B is photographed and the weak calibration of a camera is performed by using it (60). Scene photographing for generating an optional view point image is performed, the image set C is attained, correspondence between the images is obtained (62) by stereo matching by using the result of a processing 60, transformation to the affine coordinate is performed (64) and the affine coordinate of a virtual camera position is calculated (66) by a virtual camera parameter 56 stored beforehand. It is transformed (68) to a camera view and the image for which the virtual camera position is the view point is formed. Then, the virtual camera parameter 56 is changed and the image of an optional position is generated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for generating an image of a real space viewed from an arbitrary viewpoint,
The present invention relates to an arbitrary viewpoint image generation device that can easily generate a real space image in real time based on a stereo image.

[0002]

2. Description of the Related Art In recent years, the importance of remote area video communication has been increasing. However, it is difficult for an existing video conference system to provide an image that does not make the distance between users conscious.

[0003] It is thought that this problem can be solved by creating an environment where users between remote locations feel as if they coexist in one real world. For this purpose, it is necessary to generate a visual field image of the real space viewed from an arbitrary viewpoint.
Moreover, such generation must be performed in real time.

One of such techniques is to generate an arbitrary visual field image from a stereo image. Such conventional methods reconstruct a three-dimensional image from a stereo image,
Generally, an image of an arbitrary viewpoint is generated from a dimensional image. Therefore, in one method, a three-dimensional reconstruction problem is solved using strong camera calibration. In another method, the three-dimensional construction problem is solved by assuming that the correspondence between images is three-dimensional affine transformation or perspective projection transformation. By projecting the object represented by the three-dimensional coordinate system onto a new image in this manner, an image viewed from an arbitrary viewpoint is generated.

A more direct approach to scene reprojection and new view image synthesis is epipolar crossing. Also, recently, when two images are given,
A method of synthesizing an image from an arbitrary position on a line connecting the optical centers of two cameras using so-called morphing has also been proposed.

[0006]

However, the method of projecting a new image after reconstructing the three-dimensional structure is unstable with respect to a measurement error of the image in the process of reconstructing the three-dimensional structure or any other factors. There is a problem that there are factors and the obtained result is also unstable. Further, in the conventional method using the epipolar intersection method, it is necessary to perform strong calibration of the camera, and it is difficult to accurately perform the calibration. In addition, eight corresponding points are required between at least three images, and the measurement of the coordinates of these points includes an error. Therefore, larger coordinates are required at the time of reprojection than when fewer corresponding points are used. There is a problem that an error is caused. In addition, the technique using morphing can avoid the troublesome problem of stereo matching, but cannot be extended to image synthesis from a virtual camera at an arbitrary position. That is, the method using morphing has a problem that it can be used only when the virtual camera is on a straight line connecting the camera centers of the two real cameras.

SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an arbitrary viewpoint image generating apparatus which has a small coordinate error and can stably generate an image even at any point in space.

[0008]

According to a first aspect of the present invention, there is provided an arbitrary viewpoint image generating apparatus, comprising: an image obtaining unit for obtaining a stereo image; and a stereoscopic image obtained by the image obtaining unit. Means for creating a reference vector of a predetermined affine coordinate system, and means for weakly calibrating the relative position of the viewpoint at the time of image acquisition based on the stereo image acquired by the image acquisition means, On the basis of the result of the weak calibration, a means for associating points between the stereo images acquired by the image acquiring means, and each of the stereo images based on the output of the means for associating with the reference vector. First conversion means for converting the coordinates of the point into the coordinates of the affine coordinate system, and the coordinates of the affine coordinate system based on the virtual viewpoint position data. And second conversion means for converting the affine coordinates of the virtual viewpoint position by using the projection method, the output of the second conversion means, and means for generating an image in a virtual viewpoint position.

After the affine transformation reference vector is created from the stereo image, the relative position of the viewpoint when the stereo image to be processed is obtained is weakly calibrated. Further, the correspondence between points in the stereo images is obtained, and the coordinates of each point are affine-transformed using the reference vector already obtained, and then converted into affine coordinates at the virtual viewpoint position using the pinhole projection method. From the affine coordinates obtained in this way, an image at the virtual viewpoint position is generated.

Since weak calibration is used,
As compared with the three-dimensional reconstruction method, the image can be reprojected more directly, the number of necessary reference points is reduced, and the measurement error is reduced, so that a more stable result can be obtained. Further, since the position of the viewpoint is not limited in this processing, an image can be generated regardless of where the viewpoint is in the space.

[0011]

BEST MODE FOR CARRYING OUT THE INVENTION

[Principle of the Present Invention] First, the principle of the arbitrary viewpoint image generating apparatus of the present invention will be described. The present invention is based on the literature "DW Jacobs:" Space Efficient 3D Mode
l Indexing "(Proceedings of CVPR, pp. 439-444, 1992)
Based on a very simple principle proposed in This document states that a pair of two-dimensional images obtained by imaging a three-dimensional rigid body model can be optimally represented as two straight lines on a higher-order α-β affine space.
In addition, if this method is adopted, an orthographic projection can be assumed as a projection model of the camera, but in this case, deterioration of an obtained image is observed as described later. Therefore, in the present invention, in order to maintain the quality of the recombined image, a pinhole camera is used for projection.

The inventor employed a pinhole camera for projection, and examined the characteristics of the affine space at points corresponding to two images taken from a stereo camera. The characteristics thus obtained are used as a scene reprojection method for applications such as synthesis of a new visual field image.

[Camera Calibration Model] Here, the strong camera calibration and the weak camera calibration used in the present invention will be briefly described.

(1) Strong Camera Calibration In FIG. 1, when projecting a point P in space to a point p on an image plane based on strong camera calibration, 3 ×
4 need to be calculated. The coordinates of the point P in the homogeneous coordinate system are [XYZ 1], and the coordinates of the point p are (u,
If v), the following equation holds. [tu tv 1] '= C [XYZ 1]' (1) where t is a parameter.

Since the overall scaling factor of the transformation matrix C is not important, the calibration involves estimating the eleven elements of the transformation matrix C. That is, the strong camera calibration parameters can be estimated using the mapping information of at least six points in the three-dimensional coordinates as knowledge.

Given the stereo pair shown in FIG. 2 and letting the strong camera calibration parameters for both cameras be known, a point (u) in the first image
_{If (1,} v ₁ ) corresponds to points (u _2, v ₂ ) in the second image, the three-dimensional coordinates of point P corresponding to those points are calibrated using standard triangulation. Can be inferred from the parameters. Thereby, the point P can be re-projected into an arbitrary image using the projection matrix of the virtual camera. However, if the camera is not properly calibrated, a measurement error occurs during the calibration process, and the optical centers C1 ′ and C2 ′ including the error with respect to the optical centers C1 and C2 are set.
At the time of three-dimensional reconstruction, the coordinates of the point P are erroneously estimated as the coordinates of the point P 'as shown by a dotted line in FIG. This also leads to errors in the reprojection process when generating an image from a new viewpoint.

(2) Weak Camera Calibration On the other hand, weak camera calibration used in the present invention is a method for finding an epipolar positional relationship between two stereo images. That is, weak calibration refers to calibration of the relative position of the viewpoint at the time of image acquisition. As shown in FIG. 3, (u ₁ , v
_{If 1} ) represents the coordinates of point p in the first image, 3 ×
Using the matrix F of 3, the epipolar line L in the second image can be calculated. The point in the second image corresponding to point p is located on this epipolar line. If the epipolar line is represented by l ₁ u ₂ + l ₂ v ₂ + l ₃ = 0, the relationship is represented by the following equation. [l ₁ l ₂ l ₃ ] '= F [u ₁ v ₁ 1]' ... (2) To solve F including the scaling factor, 2
Eight points of correspondence are required in one image. This matrix F
Can be used to solve the stereo matching problem for finding point correspondences between images. In this method, the correspondence of points between images given by correlation matching with a limited search range on an epipolar line is obtained. Also,
The result of this weak camera calibration is also used when generating an image from a new viewpoint.

As will be described later, the coordinate values after reprojection of a point calculated using this method are compared with the reprojection method based on strong camera calibration with respect to the error in measuring the coordinate value of the reference point. Has more stable performance.

[Characteristics of Affine Coordinates] In the present invention, each point is re-projected using an affine transformation. Here, the concept of reprojection based on affine coordinates will be considered separately for two cases, parallel projection and pinhole projection.
In the present invention, the concept of pinhole projection is used. In the case of parallel projection, the point in the scene may be anywhere, and the projection method of the camera is assumed to be orthographic. In the case of pinhole projection, the point in the scene may be anywhere, and the projection method of the camera is assumed to be pinhole projection.

(1) In the case of parallel projection A method of simplifying a camera projection model by parallel projection onto a plane and subsequent affine transformation has already been proposed.
Referring to FIG. 4, _assuming that a set of points (P ₁ , P ₂ ,... P _n ) exists, a virtual plane (reference plane) 20 passing through _three points P ₁ , P ₂ , and P ₃ is defined. Think. The perpendicular foot drawn from the point P ₄ to the reference plane 20 and the point p ₄ '. (P ₁ of the point p ₄ ',
The affine coordinates based on (P ₂ , P ₃ ) are (a ₄ ,
b ₄ ). Similarly, let the leg of the perpendicular line lowered from the i-th point to the reference plane 20 be p _i ′, and the affine coordinates are (a _i , b _i ). Further, each of the d ₄ and d _i, the distance from the point P _4, P _i to the reference plane 20.

The affine coordinates (α ₄ , β ₄ ) indicate the affine coordinates of a point projected on the reference plane 20 when the point P ₄ is viewed from the viewpoint. A point having affine coordinates (α ₄ , β ₄ ) with respect to (P ₁ , P ₂ , P ₃ ) is defined as p _b4 . The line 24 connecting the point p _b4 and the point P ₄ is the viewing direction. This line image plane 22 (the normal of this plane sight parallel) intersect at a point q _4. Therefore, the point q ₄ becomes an image of the point P ₄ . Similarly projected to _{(P 1, P 2, P} 3) of each image plane _{22 (q 1, q 2,} q 3). When a plane determined by these points (q ₁ , q ₂ , q ₃ ) is selected as a reference, the point q is also obtained when an affine transformation (such as translation, rotation, enlargement / reduction, etc.) is performed on the image plane 22. ₄ has affine coordinates (α ₄ , β ₄ ).

Next, the affine coordinates (α _i , β _i ) of the remaining projection points are also calculated from the given viewing direction 24 by (α
₄ , β ₄ ). p _b ⁱ the reference plane 20 and the P
If it is an intersection with a parallel ray from the line of sight passing through _i , q _i will be mapped onto the image plane 22. Similarly,
The points p _bi and q _i are (P ₁ , P ₂ , P ₃ ), (q ₁ ,
q ₂ , q ₃ ) have affine coordinates (α _i , β _i ), respectively.

The triangle P ₄ p _b4 p ₄ ′ and the triangle P _i p
If the similarity of the triangle of _bi p _i ′ is used, the relational expression is expressed as follows.

[0024]

(Equation 1)

When this is represented by affine coordinates, it is as follows.

[0026]

(Equation 2)

Considering only α, it can be further expressed as follows.

[0028]

(Equation 3)

In this equation, a _i , a ₄ , d _i , and d ₄ are constants which are the same at all points in the image.
It can be generated from points on a dimension. That is, (P ₁ ,.
P _n ), α ₄
And α _i are plotted in a straight line, and the slope is d _i / d _{4 as} shown in FIG. The slope indicates whether the point P _i is how far away from the reference plane. Similarly, β ₄ and β _i are also straight lines with the same inclination as α.

(2) In the case of a pinhole Here, a standard three-dimensional point is projected on a plane.
The coordinate system of a typical pinhole camera is assumed. Then this
An affine transformation of these projected points is performed. Referring to FIG.
And (P₁, P_Two, P_Three,. . . , P_n) On three dimensions
Point P₁, P_Two, P_ThreeVirtual plane (reference plane) passing through
20 is defined as shown in FIG. Point P_FourTo the reference plane
P_Four'And its (P₁, P_Two, P_ThreeBased on
The standard affine coordinates are (a_Four, B_Four). Likewise
Point p_i’. (Α_Four, Β_Four) Is based on the viewpoint position
Point p projected on the reference plane 20_b4For (P₁, P_Two,
P_Three) As affine coordinates. Point p_b4And point p
_FourThe optical center C of the camera is on an extension of a straight line passing through the camera.
This position is arbitrarily determined. About line of sight 24
Also arbitrarily chosen. Straight line P_Fourp_b4Is the point q_FourAt image plane 22
Intersect with Point q_FourIs the point P_FourIt is an image of. Similarly, the point
P₁, P_Two, P_ThreeTo the image plane 22 at points q₁, Q
_Two, Q_ThreeIs mapped as Point q_FourIs this (q₁,
q_Two, Q_Three) To the affine coordinates (α_Four,
β_Four)have.

Point P_iAre projected onto the image plane 22 by q_i
And As mentioned above, point P_biAnd point q _iAnd
(P₁, P_Two, P_Three), (Q₁, Q_Two, Q_Three) Based on
Affine coordinates (α_i, Β_i)have. Ma
C 'and d_cAnd from the camera center C to the reference plane
20 perpendicular to the camera and from there to the camera center
And the distance. Triangle Cp_bic 'and triangle P_ip
_bip_i′, Using the approximation of two triangles,
To establish.

[0032]

(Equation 4)

Similarly, the triangle Cp _b4 c 'and the triangle P ₄ p _b4 p
Using the approximation with ₄ ', the following equation holds.

[0034]

(Equation 5)

From the equations (6) and (7), the following equation (8) is established.

[0036]

(Equation 6)

If the above equation is rewritten in affine coordinates, the following is obtained.

[0038]

(Equation 7)

When only α is extracted, the following is obtained.

[0040]

(Equation 8)

Here, the following equation is established.

[0042]

(Equation 9)

In equation (11), a _i , a ₄ , d _i and d ₄ have the same value for all images that can be generated, and a _c depends on camera parameters. That is, if a _c and a ₄ are known, α ₄ ′ can be easily calculated for a given image. That is, for all images that can be generated, α ₄ ′ and α _i are plotted on a straight line having a slope d _i / d ₄ . This characteristic is shown in FIG. The straight line drawn in the β space has the same inclination as the straight line drawn in the α space.

The slope of the straight line corresponding to the i-th point is directly proportional to the distance from that point to the reference plane.

[Synthesis of New View Image and Other Applications] Generation of New View Image It is finally necessary to generate a new view image from the affine coordinates obtained in this manner. even here,
Although there are two types of concepts, parallel projection and pinhole projection, the present invention employs pinhole projection. Hereinafter, a method using parallel projection and a method using pinhole projection as in the present invention will be sequentially described.

(1) In the case of parallel projection Two images I₁, I_TwoIs given. New perspective
For image generation, point correspondence between two images
Need. Therefore, image I₁Upper point p_i ¹Compatible with
Image I_TwoP above_i ^TwoIs defined. In α and β space
To generate a straight line, four reference points were needed
Here, the point p₁', P_Two', P _Three', P_Four'
You. For simplicity, the line segment P₁P_TwoAnd line segment P₁P_ThreeYou
And line segment P₁P_FourIs a right angle and | P₁P_Two| = | P₁
P_Three| = | P₁P_FourThe point p such that |₁', P_Two',
p_Three', P_Four'Determined. Before this experiment,
A point p after shooting with two cameras and projecting₁', P_Two', P
_Three', P_FourRecord the coordinates of '. Image I₁Middle point p_Four'
And point p_iAffine coordinates with (α_Four',
β_Four'), (Α_i ^', Β_i ^'). The straight line in α space is
Points corresponding to two images (α _Four', Β_Four') And (α_Four ^Two, Β
_i ^Two) And pass.

Here, the point P_i, P_Two, P_Three, P_FourWhirl
Code coordinates (0,0,0), (0,0,0), (0,1,0), (0,0,1) respectively
Assume. These points are the points p in the virtual image₁ ^v,
p_Two ^v, P_Three ^v, P_Four ^v3x3 virtual camera perspective change
It is projected using a permutation matrix. (Α_Four ^v, Β_Four ^v) Is a point
p_Four ^v(P₁ ^v, P_Two ^v, P_Three ^vAf) based on
Coordinates. Next, image I₁Middle point p_i ¹The virtual picture
The affine coordinates in the virtual image to reproject onto the image
To the point p_i ¹Can be calculated from the straight line in α space corresponding to
No. The equation of the straight line is α_i= Κ₀α_Four+ Κ₁Then α_i
^V= Κ₀α_Four ^v+ Κ₁Can be obtained as As well
And β_i ^VIs also calculated. Point p in the image _i ^vSeat
The target can be obtained using equation (12).

[0048]

(Equation 10)

Here, it should be noted that a plurality of points in the image I ₁ are mapped to one point on the virtual screen. This problem can be solved by using the inclination of a straight line in the affine space as an index for selecting a point to be mapped. A point on a straight line having a large inclination is farther from the reference plane, that is, if the virtual camera is looking into the reference plane, the distance to the virtual screen is closer. .

Pixels may be missing in the image thus selected and reprojected. Because the reprojection process because does not guarantee that the point of the image I in _one to all of the pixels is mapped. Here, these problems were solved using the Z buffer and texture mapping, which are computer graphics techniques.

First, as shown in FIG. 8, the image I ₁ is divided into rectangles each having a length of one pixel. X of P _i ^{1 in} the figure,
y coordinates, x each P _i ^v, it is projected onto the y-coordinate.
The value of the z coordinate is the reciprocal of the slope of the straight line in the affine space,
Texture use the one of the image I _1. 3 points P _{i + 1} ¹ of the processing remaining rectangle is repeated ^{_{P i + 2 1, P i}} + 3 to all the points on the ^first and the grid. Next, a texture is mapped to this polygon. By this processing, a newly generated image can be smoothly interpolated.

(2) In the case of a pinhole Two images I₁, I_TwoIs given. New perspective
For image generation, point correspondence between two images
Need. Then, image I₁Upper point p_i ¹Corresponding to
Image I_TwoP above_i ^TwoIs defined. Direct in α and β space
To create a line, four reference points are required. This
Here, image I_jThe point p on (j = 1,2)₁ ^j,
p_Two ^j, P_Three ^j, P_Four ^jIs determined. For simplicity,
As shown in FIG.₁P_TwoAnd line segment P₁P_ThreeAnd
Line segment P₁P_FourIs a right angle and | P₁P _Two| = | P₁P
_Three| = | P₁P_Four| This structure
Before shooting, shoot with two cameras before the experiment and after projection
The point p₁ ^j, P_Two ^j, P_Three ^j, P _Four ^jRecord the coordinates of
Keep it. Image I_jMiddle point p_Four ^jAnd point p_i ^jAffine with
The coordinates are (α_Four ^j, β_Four ^j), (Α_i ^j,
β_i ^j). A straight line in α space corresponds to two images
Point (α_Four ¹', α_i ¹) And (α_Four ^Two', α_i ^Two) And pass.

In the j-th image, using equation (11), α
_In order to calculate ₄ ^j ′, it is necessary to know a _C ^j and a ₄ . The selected points P ₁ ,. . . , The value of the P ₄ a ₄ is 0. To calculate a _C ^j , a fifth control point P ₅ is required. For convenience select the point P ₅ on the extension of the line segment _{P 1 P 4, | P 5} P 1 | = k | P 4 P 1 | defined (FIG. 9). The mapping of point P _{5 to} the j-th camera is point p ₅
^j and its affine coordinates are (α ₅ ^j , β ₅ ^j ). Since a ₅ = 0, the following equation can be obtained using equations (10) and (11).

[0054]

[Equation 11]

To generate a virtual image, the points P ₁ ,
The world coordinates of P ₂ , P ₃ , P ₄ , and P ₅ are respectively (0,
Assume that (0,0), (1,0,0), (0,1,0), (0,0,1), (0,0, k).
These points are projected onto the virtual image using the perspective transformation matrix of a 3 × 4 virtual camera, respectively (p ₁ ^v ,..., P ₅ ^v ). The reprojection of the i-th point p _i ¹ is (p ₁ ^v , p ₂
^v , p ₃ ^v ) as the affine coordinates (α _i ^V , β _i
^V ). Assuming that the equation of the straight line in the α space is α _i = κ ₀ α ₄ ′ + κ ₁ , the following equations are obtained from the equations (10) and (11).

[0056]

(Equation 12)

Here, α ₄ ^v and a _C ^v are points p
₄ ^v affine coordinates, and a mapping of the center of the virtual camera. Similarly, β _i ^V is calculated. All points in the image can be projected on this virtual screen using this technique. The problem of pixel omission and Z value occurring in a new image is solved by the method described in the case of parallel projection.

[Structure of Apparatus] FIG. 10 shows the structure of an apparatus for carrying out the present invention. Referring to FIG. 10, the apparatus includes a computer 30 having RS232C terminals 40 and 42 and a video output terminal 44, and an RS232C terminal.
It includes digital cameras 32 and 34 connected to the computer 30 at 2C terminals 40 and 42 for photographing a scene or an object 38, and a monitor 36 connected to the computer 30 at a video output terminal 44.
The computer 30 has a well-known configuration, and functions as an arbitrary viewpoint image generation device by executing a program for generating an arbitrary viewpoint image inside. In this example, two digital cameras are used as digital cameras, but any camera can be used as long as it can capture a stereo image. An image may be taken. Although a digital camera is used in this example, a normal video camera may be used. In that case,
The computer 30 must have a video input terminal and have the ability to digitize video images.

FIG. 11 is a block diagram showing the flow of processing of an arbitrary viewpoint image generation program executed by the computer 30. Referring to FIG. 11, image set A is first photographed as control point photographing (50). From this image set A, a reference vector of affine coordinates is created (58). Next, image set B is photographed as a scene photograph. Using this image set B, weak calibration of the camera is performed (60).

Further, a scene for generating an arbitrary viewpoint image is photographed. The image set at this time is referred to as an image set C. The aforementioned image set B may be the same as the image set C. Processing 60 for image set C
The correspondence between images is obtained by stereo matching using the result of (62). Further, the image to which the correspondence has been obtained is converted into affine coordinates using the affine coordinate reference vector obtained in the processing 58 (64).

A virtual camera parameter 56 that defines the optical center position of the virtual camera is stored in a memory or the like in advance, and the affine coordinates calculated at the virtual camera position are calculated with respect to the affine coordinates obtained in the process 64 (FIG. 66). The affine coordinates at the virtual camera position thus obtained are converted into a camera view (68), and an image is generated with the virtual camera position as a viewpoint. By arbitrarily changing the virtual camera parameters 56, an image at an arbitrary position can be generated in real time.

[Experimental Comparison of Performance to Error] The effect of the arbitrary viewpoint image generating apparatus according to the present invention will be described below.
First, experimentally verify that the reprojection algorithm based on affine coordinates is more stable than the three-dimensional reconstruction method. Therefore, considering a scene (stereo image) including a box with a check pattern as shown in FIG. 12, six points were selected from the box for camera calibration. We manually matched the points, calculated the three-dimensional coordinates of the points, and projected them onto a new image. The camera matrix was arbitrarily determined. Assuming that the measurement errors of the x coordinate values of each of the six reference points are uniformly distributed between −1 and 1,
The distribution of the coordinate values of the points after reprojection was simulated.

Five reference points were selected for the reprojection algorithm based on affine coordinates, and the effect on the coordinate values of the reprojected points was examined, assuming the same properties for the measurement error. Three different points are selected from an arbitrary set of images, and the results of comparison between a three-dimensional reconstruction method and a method based on affine coordinates are considered. For these three points (i) (ii) (iii), x,
FIGS. 13 to 18 show histograms of the y-coordinate plotted for each of the three-dimensional reconstruction method (a) and the affine reprojection (b). 13, 15, and 17 show x
FIGS. 14, 16 and 18 are for the y coordinate. FIGS. 13 and 14 correspond to point (i), and FIGS. 15 and 16 correspond to point (ii).
And FIG. 18 is for point (iii). In the figure,
"A" is used for the three-dimensional reconstruction method, and "b" is used for the method of the present invention based on the affine coordinate transformation.
Is attached to each.

It can be seen that the histogram of the x coordinate of the reprojection method based on the affine coordinates has clearly sharp peaks at all three points, and that the variance is much smaller than that of the three-dimensional reconstruction method. On the other hand, for the y coordinate, the points (i) (i
In i), a sharper peak appears in the three-dimensional reconstruction method. However, examination of the histogram in this case also shows that the variance is larger in the three-dimensional reconstruction method than in the method based on affine coordinates. Exact x,
The lack of one basis, the y-coordinate, poses a question of how to evaluate the accuracy of the two approaches, but if the accuracy of the approaches is determined by the height of the peak and the corresponding histogram spread, For example, it can be said that the direct reprojection method of the present invention provides more stable results and is more accurate than the method based on three-dimensional reconstruction.

[Application] In order to verify the technique of scene reprojection using affine coordinates, three images of a certain scene were photographed by a digital camera connected to a computer having the configuration shown in FIG. The images are shown in FIGS.
As shown in FIG. (Images (c) and (d) are the same) In the description so far, the third image is always a virtual image, but the technique of the present invention is also used when the third image is a real image. it can.

Referring to FIG. 19, five corresponding points in three images are selected from the check-like blocks on the image.
Marked with "x". In addition, seven corresponding points were selected from the first two images, and are indicated by black dots in FIGS. 19 and 20. FIG. 21 and FIG. 22 are diagrams showing how these points are projected on the third image by the pinhole model and the parallel projection model, respectively. It can be seen that the pinhole model shown in FIG. 21 has less deviation of the “x” point than the parallel projection model shown in FIG. 22 and shows a good reprojection result.

Next, a new visual field image is synthesized based on the two stereo images as shown in FIG. FIGS. 24 and 25 show the results of generating a new visual field image. FIG.
Indicates an image generated from the virtual camera further left than the real camera on the left. FIG. 25 shows an image generated from the virtual camera at the extreme upper left. In particular, in such an example, the corresponding error of the point is clearly reflected in the reprojection result as compared with the example described above. This is mainly caused by an error in stereo matching due to poor quality of the input image, but can be improved by increasing the quality of the input image. The algorithm for generating the new visual field image can be executed in a computer in almost real time.

[Summary] As described above, according to the present invention, it is possible to realize a method using a unified concept using features of affine coordinates in a stereo image for an application of generating a new visual field image. The calculation for scene reprojection is characterized by being very simple, as shown in equation (14). For this reason, the synthesis of a new visual field image can be realized almost in real time by using a graphics computer. In addition, this method can be realized with a small number of corresponding points of five points as compared with the standard approach, and can minimize the influence of errors in stereo matching on the reprojected image.

It is considered that the present method can be applied to a dynamic scene. The biggest problem in that case is stereo matching at the frame rate. In order to apply the invention to dynamic scenes, it is necessary to combine it with a more efficient approach than standard stereo matching algorithms.

[Comparison between the present method and another direct method] As described above, the epipolar line intersection method, which is one of the reprojection methods, uses strong camera calibration or three images (two images are real images). (One image is a new image). It is difficult to accurately estimate the strong camera calibration parameters, which leads to errors in reprojection. Also, the measurement errors contained in the eight reference point coordinates in the three images lead to larger coordinate errors during reprojection as compared to methods based on affine coordinates that require five reference points. Also, the reprojection of the scene is "sparse"
It just generates an image. In the method based on affine coordinates as in the present invention, “filling in holes” can be performed using the inclination of a straight line in the affine space as an index of the distance from the corresponding plane. In the epipolar line intersection algorithm, the depth corresponding to the reprojected point (z
Value) is not calculated explicitly and is therefore difficult to extend to the generation of new fields of view.

As already mentioned, the morphing method is
While the troublesome stereo matching problem can be avoided, it can be used only when the virtual camera is on a straight line connecting the camera centers of the two real cameras. On the other hand, in the method of the present invention, it is necessary to first solve the point correspondence problem, but there is an advantage that the virtual camera may be located anywhere in the space.

[Brief description of the drawings]

FIG. 1 is a diagram for describing strong camera calibration.

FIG. 2 is a diagram for explaining the principle of a three-dimensional reconstruction method.

FIG. 3 is a diagram for explaining an epipolar line.

FIG. 4 is a diagram for explaining the principle of affine transformation in the case of parallel projection.

FIG. 5 is a diagram showing the relationship between α ₄ and α _{i in} the case of parallel projection.

FIG. 6 is a diagram for explaining affine coordinates of a pinhole projection system.

FIG. 7 is a diagram showing the relationship between α ₄ ^′ and α _{i in} the case of pinhole projection.

FIG. 8 is a diagram for explaining the principle of processing for preventing pixel omission;

FIG. 9 is a diagram showing an arrangement of five reference points in the present invention.

FIG. 10 is a diagram illustrating a configuration of an arbitrary viewpoint image generation device according to an embodiment of the present invention.

FIG. 11 is a diagram showing, in a block diagram form, a flow of processing performed in the arbitrary viewpoint image generating device according to the embodiment of the present invention;

FIG. 12 is a diagram showing a stereo image used in an experiment for verifying the effect of the device according to the embodiment of the present invention.

FIG. 13 is a diagram showing the difference between the effect of the three-dimensional reconstruction method and the method according to the present invention on the first point, in the form of a histogram of x coordinates.

FIG. 14 is a diagram illustrating a difference between the effect of the three-dimensional reconstruction method and the method according to the present invention on the first point in the form of a histogram of y-coordinates.

FIG. 15 is a diagram showing the difference between the effect of the three-dimensional reconstruction method and the method according to the present invention on the second point in the form of a histogram of x coordinates.

FIG. 16 is a diagram showing the difference between the effect of the three-dimensional reconstruction method and the method according to the present invention on the second point in the form of a histogram of y-coordinates.

FIG. 17 is a diagram showing the difference between the effect of the three-dimensional reconstruction method and the method according to the present invention on the third point, in the form of a histogram of x coordinates.

FIG. 18 is a diagram showing the difference between the effect of the three-dimensional reconstruction method and the method according to the present invention on the third point in the form of a histogram of y-coordinates.

FIG. 19 is a diagram showing an image used in an experiment for verifying the effect of scene reprojection according to the present invention.

FIG. 20 is a diagram showing another image used in an experiment for verifying the effect of scene reprojection according to the present invention.

FIG. 21 is a diagram showing still another image used in an experiment for verifying the effect of scene reprojection according to the present invention, and a projection result by a pinhole model.

FIG. 22 is a diagram showing still another image used in an experiment for verifying the effect of scene reprojection according to the present invention, and a projection result by a parallel projection model.

FIG. 23 is a diagram showing an image used in another experiment for verifying the effect of scene reprojection according to the present invention.

FIG. 24 is a diagram showing a projection result by a pinhole model in an experiment for verifying the effect of scene reprojection according to the present invention.

FIG. 25 is a diagram illustrating a projection result by a pinhole model from an extremely upper left camera position in an experiment for verifying the effect of scene reprojection according to the present invention.

[Explanation of symbols]

 Reference Signs List 20 Reference plane 22 Image plane 24 Viewing direction 30 Computer 32, 34 Digital camera 36 Monitor

 ──────────────────────────────────────────────────の Continuing on the front page (72) Inventor Tatsumi Sakaguchi 5th Sanraya, Inaya small character, Seika-cho, Soraku-gun, Kyoto Pref. 5 Shiraya, Seiya-cho, Seika-cho, Soraku-gun ATI Intelligent Motion Picture Communication Laboratory, Inc.

Claims (1)

[Claims] An image acquisition unit for acquiring a stereo image; a unit for creating a reference vector in a predetermined affine coordinate system based on the stereo image acquired by the image acquisition unit; A means for weakly calibrating the relative position of the viewpoint at the time of image acquisition based on the stereo image acquired by the means; and a stereo image acquired by the image acquisition means based on the result of the weak calibration. And converting the coordinates of each point of the stereo image into the coordinates of the predetermined affine coordinate system based on the means for establishing correspondence between the points and the output of the means for establishing the correspondence with the reference vector. The first affine coordinate system based on the virtual viewpoint position data and the virtual viewpoint position using a pinhole projection method. And second conversion means for converting the affine coordinates, from the output of said second converting means, and means for generating an image in the virtual viewpoint position, arbitrary viewpoint image generating device.