JP2008021210A - Method, device and program for generating free-viewpoint image using multi-viewpoint image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a free-viewpoint image generating method or the like capable of generating an image of an optional viewpoint by detecting a parallax vector by block matching and using the parallax vector even when the surface of an object is inclined with respect to optical axes of a plurality of camera. <P>SOLUTION: One and the same rotational transformation is applied so as to make the optical axes of the two cameras perpendicular to the same plane going through the cameras. Depth distance Z of an optional point and a unit normal vector at an optional point of the surface of an object are determined arbitrarily, moved by a parallax vector based on t and Z, transformed by a primary transformation matrix based on t, Z and n and matched to a block of a second image. While changing Z and n arbitrarily, a second block of the second image that resembles a first block of a first image most is retrieved. A block of an image from an optional viewpoint is generated in accordance with an internal division ratio of an optional viewpoint on a straight line connecting the two cameras. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a free viewpoint image generation method, apparatus, and program using multi-viewpoint images.

  Conventionally, there is a technique called “free viewpoint image” in which a viewer can freely select a viewpoint position or a viewpoint direction. The free viewpoint image is generated by a multi-viewpoint image obtained by photographing a subject with a plurality of cameras. An image from a viewpoint that is not photographed by the camera is generated by interpolation between images using a disparity vector. Therefore, it is necessary to solve the corresponding point search problem between multi-viewpoint images. Of course, the closer the camera interval, the higher the quality of the free viewpoint image.

  “Block matching” is a typical method for solving the corresponding point search problem between multi-viewpoint images. Detection of a disparity vector using “block matching” is performed as follows.

  The first image taken by the first camera (first viewpoint) is divided into small blocks. Each of the first blocks of the first image is translated in the second image of the second camera (other viewpoint) to search for a region having the highest similarity. Specifically, the second block of the second image having the smallest absolute value error or square error is searched for the first block of the first image. Then, the distance amount between the searched second block and the first block is calculated as a disparity vector. When an arbitrary viewpoint is selected on a line connecting the first camera and the second camera with a straight line, the parallax is determined according to the internal ratio of the distance between the arbitrary viewpoint and the first and second cameras. Adjust the vector. By interpolating the first image of the first camera in accordance with the adjusted disparity vector, an image of the arbitrary viewpoint can be generated.

JP 11-37721 A Japanese Patent Laid-Open No. 10-191393 Japanese Patent Laid-Open No. 10-191394

  Conventional block matching is to match blocks when a subject in a first image is translated in a second image. Only when the surface of the subject is perpendicular to the optical axis of the camera as viewed from a plurality of cameras and is equidistant from the camera, complete block matching can be performed.

  However, when the surface of the subject is inclined with respect to the optical axes of a plurality of cameras, the surface of the subject also looks different. That is, even if the first block of the first image is translated in the second image, it is not possible to search for the second block that perfectly matches the first block. This is because the shape of the subject in the first image is different from the shape of the subject in the second image.

  Therefore, the present invention detects a disparity vector by block matching even when the surface of an object is inclined with respect to the optical axes of a plurality of cameras, and uses the disparity vector to obtain an image of an arbitrary viewpoint. It is an object of the present invention to provide a free viewpoint image generation method, apparatus, and program that can be generated.

According to the present invention, the first camera and the second camera whose optical axes are parallel to the Z axis have the first relative position (t _x , t _y , t _z ) (where t _z ≠ 0). Is a free viewpoint image generation method for generating an image of an arbitrary viewpoint using a first image of a first camera and a second image of a second camera that have taken a subject. ,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. A first step of applying the same rotational transformation to the camera to transform the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera;
A second step of determining an arbitrary point on the subject;
The first image coordinates (x _R , y _R ) obtained by projecting the arbitrary point to the first image of the first camera to which the rotation transformation is applied, and the second image of the second camera to which the arbitrary point is subjected to the rotation transformation. A third step of deriving second image coordinates (x _R ′, y _R ′) projected onto the second image;
A fourth step of selecting a first block including an arbitrary point for the first image;
A depth distance Z of an arbitrary point and a unit normal vector ( _nx , _ny , _nz ) at an arbitrary point on the surface of the subject are arbitrarily determined, and the depth distance Z _{R with} respect to the same plane subjected to the rotation transformation And a fifth step of deriving a unit normal vector (n _Rx , n _Ry , n _Rz ),
The first block is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotation transformation is applied, and the second image is used with the first image coordinates as the origin. Based on the relative position (t _Rx , t _Ry , 0), the depth distance Z _R , the unit normal vector (n _Rx , n _Ry , n _Rz ) and the second image coordinates (x _R ′, y _R ′). A sixth step of transforming with a primary transformation matrix to match a block of the second image;
The first block of the first image in which the depth distance Z and the unit normal vector (n _x , n _y , n _z ) are arbitrarily changed, and the fifth and sixth steps are repeated, moved, and deformed. Searching for a second block of a second image most similar to, and deriving a depth distance Z and unit normal vectors ( _nx , _ny , _nz );
Relative position (t _x ″, t _y ″, t _z ″) of the arbitrary viewpoint and image coordinates (in accordance with the internal ratio of the arbitrary viewpoint on the straight line connecting the first camera and the second camera) an eighth step of determining x ″, y ″);
Depth distance Z and unit normal vector _{(n x, n y, n} z) and the relative position of the arbitrary viewpoints _{(t x ", t y"} , t z ") and image coordinates (x", y ") And a ninth step of generating a block of an image from an arbitrary viewpoint.

According to another embodiment of the free viewpoint image generation method of the present invention,
An arbitrary point on the subject is further determined for a portion other than the first block of the first image, and the second to seventh steps are repeated,
Finally, it is also preferable to generate all parts of an image from an arbitrary viewpoint on a straight line connecting the first camera and the second camera.

第２のステップについて、任意点の画像座標は、正規化画像座標であり、
回転変換の結果、新たな奥行き距離Ｚ_Ｒ及び新たな単位法線ベクトル（ｎ_Ｒｘ，ｎ_Ｒｙ，ｎ_Ｒｚ）と、任意の視点の新たな相対的位置（ｔ_Ｒｘ，ｔ_Ｒｙ，０）及び新たな画像座標（ｘ_Ｒ’，ｙ_Ｒ’）が得られたとし、
第６のステップについて、視差ベクトルは、以下のものであり、

一次変換行列は、以下のものである

ことも好ましい。 According to another embodiment of the free viewpoint image generation method of the present invention,
For the first step, the matrix representing the rotation transformation is:

For the second step, the image coordinates of the arbitrary point are normalized image coordinates,
As a result of the rotation transformation, a new depth distance Z _R and a new unit normal vector (n _Rx , n _Ry , n _Rz ), a new relative position (t _Rx , t _Ry , 0) of the arbitrary viewpoint, and a new Suppose that the correct image coordinates (x _R ', y _R ') are obtained,
For the sixth step, the disparity vector is:

The primary transformation matrix is

It is also preferable.

According to another embodiment of the free viewpoint image generation method of the present invention,
When the optical axis of the first camera and / or the second camera is not parallel to the Z axis,
The camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera are converted using camera internal parameters so that the optical axes of the first camera and the second camera are parallel to the Z axis. It is also preferable to have the step to do.

According to another embodiment of the free viewpoint image generation method of the present invention,
Even when three or more cameras are arranged, the depth distance Z and the unit normal vector ( _nx , _ny , _nz ) are constant,
In a seventh step, a prediction error between one base camera block and another camera block is calculated, and a depth distance Z and a unit normal vector (n _x , n _y , It is also preferable to derive _nz ).

According to the present invention, the first camera and the second camera whose optical axes are parallel to the Z axis have the first relative position (t _x , t _y , t _z ) (where t _z ≠ 0). Is a free viewpoint image generation device that generates an image of an arbitrary viewpoint using a first image of a first camera and a second image of a second camera that have photographed a subject. ,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. Camera coordinate system coordinate conversion means for converting the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera by applying the same rotational transformation to the camera;
Arbitrary point selection means for determining an arbitrary point on the subject;
The first image coordinates (x _R , y _R ) obtained by projecting the arbitrary point to the first image of the first camera to which the rotation transformation is applied, and the second image of the second camera to which the arbitrary point is subjected to the rotation transformation. Image coordinate calculating means for deriving second image coordinates (x _R ′, y _R ′) projected onto the image of 2;
Block selection means for selecting a first block including an arbitrary point for the first image;
A depth distance Z of an arbitrary point and a unit normal vector ( _nx , _ny , _nz ) at an arbitrary point on the surface of the subject are arbitrarily determined, and the depth distance Z _{R with} respect to the same plane subjected to the rotation transformation And parameter determining means for deriving unit normal vectors (n _Rx , n _Ry , n _Rz ),
The first block is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotation transformation is applied, and the second image is used with the first image coordinates as the origin. Based on the relative position (t _Rx , t _Ry , 0), the depth distance Z _R , the unit normal vector (n _Rx , n _Ry , n _Rz ) and the second image coordinates (x _R ′, y _R ′). Block matching means for transforming with a primary transformation matrix and matching with a block of the second image;
While arbitrarily changing the depth distance Z and the unit normal vector (n _x , n _y , n _z ), the parameter determination means and the block matching means are repeatedly controlled to move and deform the first of the first image Matching control means for searching for a second block of the second image most similar to the block and deriving a depth distance Z and a unit normal vector ( _nx , _ny , _nz );
Relative position (t _x ″, t _y ″, t _z ″) of the arbitrary viewpoint and image coordinates (in accordance with the internal ratio of the arbitrary viewpoint on the straight line connecting the first camera and the second camera) arbitrary viewpoint parameter determining means for determining x ″, y ″);
Depth distance Z and unit normal vector _{(n x, n y, n} z) and the relative position of the arbitrary viewpoints _{(t x ", t y"} , t z ") and image coordinates (x", y ") And free viewpoint image generation means for generating a block of an image from an arbitrary viewpoint.

According to another embodiment of the free viewpoint image generation device of the present invention,
The arbitrary point selection means further determines an arbitrary point on the subject for a portion other than the first block of the first image,
A block control unit that repeats an image coordinate calculation unit, a parameter determination unit, a block matching unit, a matching control unit, an arbitrary viewpoint parameter determination unit, and a free viewpoint image generation unit for an arbitrary point,
Finally, it is also preferable to generate all parts of an image from an arbitrary viewpoint on a straight line connecting the first camera and the second camera.

画像座標算出手段は、任意点の画像座標として正規化画像座標を導出し、
回転変換の結果、新たな奥行き距離Ｚ_Ｒ及び新たな単位法線ベクトル（ｎ_Ｒｘ，ｎ_Ｒｙ，ｎ_Ｒｚ）と、任意の視点の新たな相対的位置（ｔ_Ｒｘ，ｔ_Ｒｙ，０）及び新たな画像座標（ｘ_Ｒ’，ｙ_Ｒ’）が得られたとし、
ブロックマッチング手段は、視差ベクトルを以下のものとし、

一次変換行列を以下のものとする

ことも好ましい。 According to another embodiment of the free viewpoint image generation device of the present invention,
For the camera coordinate system coordinate transformation means, the matrix representing the rotation transformation is as follows:

The image coordinate calculation means derives normalized image coordinates as image coordinates of an arbitrary point,
As a result of the rotation transformation, a new depth distance Z _R and a new unit normal vector (n _Rx , n _Ry , n _Rz ), a new relative position (t _Rx , t _Ry , 0) of the arbitrary viewpoint, and a new Suppose that the correct image coordinates (x _R ', y _R ') are obtained,
The block matching means uses the following disparity vectors:

Let the primary transformation matrix be

It is also preferable.

According to another embodiment of the free viewpoint image generation device of the present invention,
When the optical axis of the first camera and / or the second camera is not parallel to the Z axis,
The camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera are converted using camera internal parameters so that the optical axes of the first camera and the second camera are parallel to the Z axis. It is also preferable to do.

According to another embodiment of the free viewpoint image generation device of the present invention,
Even when three or more cameras are arranged, the depth distance Z and the unit normal vector ( _nx , _ny , _nz ) are constant,
In a seventh step, a prediction error between one base camera block and another camera block is calculated, and a depth distance Z and a unit normal vector (n _x , n _y , It is also preferable to derive _nz ).

According to the present invention, the first camera and the second camera whose optical axes are parallel to the Z axis have the first relative position (t _x , t _y , t _z ) (where t _z ≠ 0). Free viewpoint image generation that causes a computer that generates an image of an arbitrary viewpoint to function using the first image of the first camera and the second image of the second camera that have been photographed for the subject. A program,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. Camera coordinate system coordinate conversion means for converting the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera by applying the same rotational transformation to the camera;
Arbitrary point selection means for determining an arbitrary point on the subject;
The first image coordinates (x _R , y _R ) obtained by projecting the arbitrary point to the first image of the first camera to which the rotation transformation is applied, and the second image of the second camera to which the arbitrary point is subjected to the rotation transformation. Image coordinate calculating means for deriving second image coordinates (x _R ′, y _R ′) projected onto the image of 2;
Block selection means for selecting a first block including an arbitrary point for the first image;
A depth distance Z of an arbitrary point and a unit normal vector ( _nx , _ny , _nz ) at an arbitrary point on the surface of the subject are arbitrarily determined, and the depth distance Z _{R with} respect to the same plane subjected to the rotation transformation And parameter determining means for deriving unit normal vectors (n _Rx , n _Ry , n _Rz ),
The first block is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotation transformation is applied, and the second image is used with the first image coordinates as the origin. Based on the relative position (t _Rx , t _Ry , 0), the depth distance Z _R , the unit normal vector (n _Rx , n _Ry , n _Rz ) and the second image coordinates (x _R ′, y _R ′). Block matching means for transforming with a primary transformation matrix and matching with a block of the second image;
While arbitrarily changing the depth distance Z and the unit normal vector (n _x , n _y , n _z ), the parameter determination means and the block matching means are repeatedly controlled to move and deform the first of the first image Matching control means for searching for a second block of the second image most similar to the block and deriving a depth distance Z and a unit normal vector ( _nx , _ny , _nz );
Relative position (t _x ″, t _y ″, t _z ″) of the arbitrary viewpoint and image coordinates (in accordance with the internal ratio of the arbitrary viewpoint on the straight line connecting the first camera and the second camera) arbitrary viewpoint parameter determining means for determining x ″, y ″);
Depth distance Z and unit normal vector _{(n x, n y, n} z) and the relative position of the arbitrary viewpoints _{(t x ", t y"} , t z ") and image coordinates (x", y ") And a computer that functions as free viewpoint image generation means for generating an image block from an arbitrary viewpoint.

  According to the free viewpoint image generation method, apparatus, and program of the present invention, even when the surface of a subject is tilted with respect to the optical axes of a plurality of cameras, a disparity vector is detected by block matching, and the disparity is detected. An image of an arbitrary viewpoint can be generated using a vector. By performing block matching while moving and deforming the first block of the first image based on an arbitrary depth distance Z and an arbitrary direction n of the normal of the subject surface, not only the disparity vector The primary transformation matrix can be detected. By adjusting the parallax vector and the primary transformation matrix according to the position of an arbitrary viewpoint, an image of an arbitrary viewpoint close to reality can be generated from the first block of the first image.

  Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a system configuration diagram according to the present invention.

  A plurality of cameras 1 arranged at different positions capture the subject 3. The plurality of cameras 1 are arranged with the optical axis parallel to the Z axis. However, these cameras are not arranged on the same plane perpendicular to the optical axis (a plane parallel to the XY plane). In other words, they are arranged scattered in the front, rear, left and right. According to FIG. 1, nine units are arranged in three vertical rows and three horizontal rows (viewpoints (0, 0) to (2, 2)). The plurality of cameras 1 are connected to a free viewpoint image generation device 2. According to the present invention, multi-viewpoint images taken by at least two cameras are required.

  The camera 1 transmits an image obtained by photographing the subject 3 to the free viewpoint image generation device 2. Here, when the camera 1 is movable, the camera position information is also transmitted to the free viewpoint image generation device 2. Of course, the free viewpoint image generation device 2 may store all camera position information in advance.

  FIG. 2 is a layout view of a plurality of cameras viewed from the front and side in the system configuration of FIG.

  According to FIG. 2, when viewed from the front, they are arranged in three rows and three rows in the same width. When viewed from the side, the cameras (0, 1) (1, 1) (2, 1) are arranged rearward with respect to the z-axis. That is, with respect to the subject 3, the plurality of cameras 1 are arranged on the cylindrical surface with respect to the y-axis.

  FIG. 3 is a screen diagram showing how the subject is seen from each camera in FIG.

  According to FIG. 3, the subject is a trapezoidal solid. The camera (1, 1) is located in front of the subject. Since the cameras (0, 1) (1, 1) (2, 1) are arranged rearward with respect to the z-axis, the subject looks a little small. When viewed from the left camera (1, 0) in front of the camera (1, 1), the subject appears to be large as a whole, the left side is widened, and the right side is reduced. Further, when viewed from the right camera (1, 2) in front of the camera (1, 1), the subject is large as a whole, the left side is reduced, and the right side is expanded.

  Similarly, when viewed from the camera (0, 1) above the camera (1, 1), the upper side of the subject appears to expand and the lower side appears to be reduced. When viewed from the lower camera (2, 1) of the camera (1, 1), the upper side of the subject is reduced and the lower side is expanded.

  As is clear from FIG. 3, even if the block is moved in consideration of parallax, there is no region that matches exactly. Therefore, correct matching cannot be obtained.

  4 to 6 show images taken by each camera according to the inclination of the surface of the subject.

  FIG. 4 is an image taken from each camera when the surface of the subject is inclined obliquely in the horizontal direction.

  The subject is on the back on the left and the front on the right. At this time, the image of the camera (1, 0) in front of the camera (1, 1) and the camera (1, 0) on the left side is larger than the image of the camera (1, 1). It seems to spread. Also, the image of the camera (1, 2) on the right side of the camera (1, 1) in front of the camera (1, 1) is larger as a whole and narrower in the lateral direction than the image of the camera (1, 1). appear.

  FIG. 5 is an image taken from each camera when the surface of the subject is tilted in the vertical direction.

  The subject has an upper side at the back and a lower side at the front. At this time, the plane of the image of the camera (0, 1) on the upper side of the camera (1, 1) appears to expand in the vertical direction than the image of the camera (1, 1). Further, the plane of the image of the camera (2, 1) below the camera (1, 1) appears narrower in the vertical direction than the image of the camera (1, 1).

  FIG. 6 is an image taken from each camera when the surface of the subject is tilted obliquely in the horizontal direction and tilted in the vertical direction. In other words, the slope of FIG. 4 and the slope of FIG. 5 are combined.

  As shown in FIGS. 3 and 4 to 6, such a difference in appearance occurs because the side surface of the trapezoidal solid of the subject is inclined with respect to the optical axes of a plurality of cameras. According to the present invention, even when the surface of the subject is inclined with respect to the optical axis of the camera, it is possible to search for a second block that perfectly matches the first block.

  FIG. 7 is an explanatory diagram of a first relative positional relationship between the two cameras.

In general, since the subject is not a plane, the deformation of the block is also nonlinear. However, if attention is paid to a sufficiently small block, the surface of the subject in the block can be approximated to a plane. The two cameras are arranged at the first relative position (t _x , t _y , t _z ) with the optical axes in parallel. According to FIG. 7, the xy planes of the camera C and the camera C ′ are at different positions with respect to the subject. Therefore, t _z ≠ 0.

  FIG. 8 is an explanatory diagram of a second relative positional relationship in which two cameras are rotated.

According to the present invention, the same rotational transformation is applied to camera C and camera C ′ using camera internal parameters. This rotation R causes the optical axes of camera C and camera C ′ to be perpendicular to the same plane passing through camera C and camera C ′. In this way, the camera coordinate system coordinates of the camera C and the camera coordinate system coordinates of the camera C ′ are converted. As a result, the second relative positions of the camera C and the camera C ′ are (t _Rx , t _Ry , 0). That is, t _Rz = 0.

And the rotation R in addition to the camera coordinate system coordinates, the depth distance Z _R from the xy plane to any point of the object, and orientation (the unit normal vector) n _R of the approximate plane, is used as a parameter for block matching.

  FIG. 9 is an explanatory diagram in which block matching is performed on the first block of the first image of the camera C and the second block of the second image of the camera C ′.

In general, since the subject is not a plane, the deformation of the block is also nonlinear. However, if attention is paid to a sufficiently small block, the surface of the subject in the block can be approximated to a plane. The two cameras are arranged at relative positions (t _Rx , t _Ry , 0) on the same plane that is parallel to the optical axis and perpendicular to the optical axis. The present invention includes a depth distance Z _R from the approximate plane to any point of the object, and its orientation (unit normal vector) of the approximate plane n _R, used as a parameter for block matching.

  Hereinafter, a process of deriving a primary transformation matrix that is a movement or deformation of a block or an image will be described in detail.

ｆk_u、ｆk_v：焦点距離（カメラの距離のピクセル数）
θ＝π／２：座標軸の角度
ｕ_０、ｖ_０：画像中心 First, an internal parameter (internal camera matrix) A of the camera is expressed by the following equation. This value is assumed to be known.

fk _u , fk _v : focal length (number of pixels of camera distance)
θ = π / 2: Coordinate axis angle u ₀ , v ₀ : Image center

The camera coordinate system of the camera C ′ is obtained by moving the camera coordinate system of the camera C in parallel (relative position t). t is one of the external parameters of the camera and represents the displacement vector of the camera C ′ with respect to the camera coordinate system of the camera C. Cameras C and C ′ are arranged with their optical axes in parallel. The relative position vector t of the camera C ′ with respect to the camera coordinate system of the camera C always has a non-zero z-axis component (t _Z ≠ 0).

The following relationship is established between the camera coordinate system coordinates of the camera C and the camera coordinate system coordinates of the camera C ′. That is, the camera coordinate system coordinates of the camera C are converted to the camera coordinate system coordinates of the camera C ′ through the external parameters of the camera.
M = M ′ + t Formula (2)
M: Camera coordinate system coordinates of the arbitrary point P of the subject in the camera C M ′: Camera coordinate system coordinates of the arbitrary point P of the subject in the camera C ′

ｄ：カメラＣ’のカメラ座標系の原点から平面までの距離 Here, with respect to the approximate plane π of the subject, the following expression is established in the camera coordinate system of the camera C ′.
n ^T M ′ = d Formula (5)
n (| n | = 1): normal vector of the plane in the camera coordinate system of the camera C ′

d: Distance from the origin of the camera coordinate system of camera C ′ to the plane

Next, the digital image coordinates of the arbitrary point P are as follows.
m: digital image coordinates of an arbitrary point P in the camera C m ′: digital image coordinates of the arbitrary point P in the camera C ′

Further, the normalized image coordinates of the arbitrary point P are as follows. A “normalized image” refers to an image when it is assumed that the image plane is taken by a camera (normalized camera) whose unit length is from the focal point. The coordinates of an arbitrary point in the normalized image are referred to as “normalized image coordinates”.
x: Normalized image coordinates of the arbitrary point P in the camera C x ′: Normalized image coordinates of the arbitrary point P in the camera C ′

As a result, the following central projective transformation is established.

Next, when Expression (7) and Expression (3) are substituted into Expression (9), the following expression is derived.

  Comparing the third row on both sides leads to s = Z.

Substituting this into equation (9) yields the following equation:

On the other hand, when the formula (2) is substituted into the formula (10), the following formula is established.

By substituting Equation (1), Equation (3), Equation (4), and Equation (8), the following equation is derived.

When comparing the third row on both sides, s ′ = Z ′ = Z− _tz is derived.

Substituting this into equation (10) yields the following equation:

Next, it is assumed that the camera coordinate system of the camera C is rotationally transformed and t is matched with the direction of a vector t _xy obtained by orthogonally projecting it onto the XY plane. However, t _xy is as follows.

In general, the matrix R representing the rotation of the angle θ with the unit vector r as the rotation axis is expressed as follows according to the Rodrigues formula.

Here, it is as follows.

Since r is orthogonal to t and t _xy , the following equation is obtained.

When the rotation angle is θ, the following formula is obtained.

As described above, when Expression (14) and Expression (15) are substituted into Expression (13), the rotation transformation matrix to be obtained is as follows.

Similarly, the inverse transformation matrix is as follows.

Using the rotation transformation matrix R, the translation vector t and the unit normal vector n of the approximate plane π are rotationally transformed. When the results of rotational transformation are respectively denoted as t _R and n _R , the results are as follows.

The following relationship is established between the camera coordinate system coordinates of the camera C after rotation conversion and the camera coordinate system coordinates of the camera C ′ after rotation conversion. That is, the camera coordinate system coordinates of the camera C are converted to the camera coordinate system coordinates of the camera C ′ through the external parameters of the camera.
M _R = M _R '+ t _R formula (20)
M _R : Camera coordinate system coordinates of the arbitrary point P of the subject in the camera C after rotation conversion M _R ′: Camera coordinate system coordinates of the arbitrary point P of the subject in the camera C ′ after rotation conversion

Here, in the camera coordinate system of the camera C ′ after rotation conversion with respect to the approximate plane π of the subject, the value of the distance d between the approximate plane π and the camera C ′ is not changed by rotation conversion. Holds.
n _R ^T M _R '= d Equation (23)
^{_{∴n R T M R '/ d}} = 1 Equation (24)

Substituting equation (24) into equation (20) yields the following planar projective transformation equation.

If H is a homography matrix, the result is as follows.

Next, the digital image coordinates of the arbitrary point P are as follows.
m _R : digital image coordinates of the arbitrary point P in the camera C after the rotation conversion m _R ': digital image coordinates of the arbitrary point P in the camera C' after the rotation conversion

Further, the normalized image coordinates of the arbitrary point P are as follows.
x _R : normalized image coordinates of the arbitrary point P after rotation conversion in the camera C x _R ′: normalized image coordinates of the arbitrary point P after rotation conversion in the camera C ′

Thus, the following expression is established as the central projective transformation.

By substituting Equation (26) and Equation (21) into Equation (28), the following equation is derived.

Comparing the third row on both sides leads to s _R = Z _R.

Substituting this into equation (28) yields the following equation:

On the other hand, when the formula (20) is substituted into the formula (29), the following formula is established.

Substituting Equation (21), Equation (22), Equation (1), Equation (18), and Equation (27), the following equation is derived.

Comparing the third row on both sides,
s _R '= Z _R ' = Z _R formula (41)
Is guided.

Substituting this into equation (29) gives the following equation:

Therefore, when substituting into the equation (23), the following equation is established.

Substituting Equation (19) and Equation (27), the following equation is established.
_{d = Z R (n Rx x} R '+ n Ry y R' + n Rz) formula (31)

Substituting equation (31) into equation (13) yields the following equation:

The conversion formula for x and _{x R,} and obtains a conversion equation x 'and _{x R'.} For simplicity, the definition is as follows.

Then, from the equation (11), the following is obtained.

Comparing the third row on both sides gives the following formula.

As a result, the following expression is established.

Further, by substituting equation (41) into equation (32) becomes a following formula, Z, t, x ', y' can _{Z R} is derived from.

Here, in the formula (40), _'in place of the normalized image coordinates x _R and x _R' camera coordinate system coordinates M _R and M _R With, the conversion equation is obtained.

Comparing the third row on both sides leads to s _R = 1.

Substituting this into equation (35) gives the following equation:

Further, when the digital image coordinates m _R and m _R ′ are used, the following conversion formula is obtained.

When the disparity vector in the normalized image coordinate system is D, the following equation is established.

When the difference between the right side and the left side is obtained, the following equation is established.

For the second block of the rotation-converted second image, the following D translation is performed in the rotation-converted first image, and the first image coordinates (x _R , y _R ) are used as the origin. A block of the first image that is most similar to the second block is searched by transforming with the following L linear transformation matrix. According to the primary transformation matrix, block matching can be performed by deforming a block based on the parallax vector and the inclination of the surface of the subject (approximate to a plane).

  Next, a method for synthesizing an image from the viewpoint C ″ that internally divides the line connecting the first camera and the second camera into p: q will be described.

The relative position vector t ″ of the viewpoint C ″ with respect to the camera coordinate system of the camera C has the following value.

Further, if the normalized image coordinate at the viewpoint C ″ of the arbitrary point P is x ″, x ″ is a point that is internally divided into x, x ′, and p: q.

The image of the viewpoint C ″ is obtained by transforming each block of the first image by the following D ″ parallel transformation and deforming with the following L ″ linear transformation matrix with the first image coordinates as the origin: Can be synthesized.

  Hereinafter, specific examples will be described.

  First, an example in which block matching is performed on images of two cameras that are not on the same plane will be described.

It is assumed that the camera internal parameter A and the relative position vector t are obtained in advance. For simplicity, the definition is as follows.

  This relative position vector t corresponds to a situation in which two cameras are installed on the circumference of radius R by an angle θ.

First, attention is focused on the point M ₀ in the image of the camera C. Digital image coordinate m ₀ of the point M ₀ is the following was value.

Therefore, it is converted into normalized image coordinates. The normalized image coordinate x ₀ of the point M ₀ has the following value.

  Similarly, the image of the camera C and the image of the camera C ′ are converted from the digital image coordinates to the normalized image coordinates.

Next, the image of the camera C and the image of the camera C ′ are rotationally converted. The rotation transformation matrix is as follows based on Equation (16).

The relative position vector is converted into the following vector based on Expression (18).

Based on the equation (33), rotation conversion is performed. For example, the point M ₀ is converted into the following point M _R0 .

It is assumed that the search is performed with a block of size W × H with the point M ₀ as the center point.

According to the present invention, the normal vector n _x, n _y, and n _z and depth distance Z, the block matching by changing arbitrarily performed. For example, when n _x = 0, n _y = 1, n _z = 0, and Z = R / 2, the depth distance and the normal vector in the rotation-transformed coordinate system are expressed by equations (32) and (19). Is used as follows.

Accordingly, the parallax vector D ₀ at this time is as follows from the equation (37).

Further, the primary transformation matrix L ₀ at this time is as follows from the equation (38).

At this time, if the error of the block matching is the minimum, D ₀ and L ₀ are the target disparity vector and the primary transformation matrix.

  Second, an example in which block matching is performed on images of K cameras on the same plane will be described.

The internal parameter A of the camera and the relative position vector t _k (representing the relative position vector of the kth camera with respect to the 0th camera, _where the 0th camera is the base camera, k = 1 to K−1) are obtained in advance. It is a premise. For simplicity, suppose that it is as follows.

This relative position vector t _k corresponds to a situation in which K cameras are are installed at Δ intervals on the X axis.

First, focus on the point M ₀ in the image of the 0th camera. Digital image coordinate m ₀ of the point M ₀ is the following was value.

Therefore, it is converted into normalized image coordinates. The normalized image coordinate x ₀ of the point M ₀ has the following value.

  Similarly, the images of the other K-1 cameras are converted from digital image coordinates to normalized image coordinates.

According to the present invention, block matching is performed by arbitrarily changing the normal vector (n _x , n _y , n _z ) and the depth distance Z. For _example, when _{n x = n x0, n y} = n y0, n z = n z0, Z = Z 0, disparity vectors of the corresponding point of the k camera images for a point _{M 0} in the zeroth camera images D _k is as follows.

Further, the primary transformation matrix L _k of the block in the image of the kth camera with respect to the block in the image of the 0th camera is as follows.

Here, in the normal vector _{(n x, n y, n} z) and depth distance Z remains fixed _{(n x = n x0, n} y = n y0, n z = n z0, Z = Z 0), In each camera, the blocks that are translated based on D _k and linearly converted based on L _k are compared, and the difference between the two (corresponding blocks in the 0th camera image and the kth camera image) is evaluated. To do. For example, the least square error is obtained.

The corresponding evaluation value of the difference between blocks in the zeroth camera image and the k camera image is denoted by d _k. At this time, the calculation is performed as follows for each combination of the normal vector and the depth distance.

  The combination of the normal vector and the depth distance that minimizes S is the target disparity vector and the primary transformation matrix.

  Third, an embodiment in which block matching is performed on images of K cameras not on the same plane will be described.

The internal parameter A of the camera and the relative position vector t _k (representing the relative position vector of the kth camera with respect to the 0th camera, _where the 0th camera is the base camera, k = 1 to K−1) are obtained in advance. It is a premise. For simplicity, suppose that it is as follows.

This relative position vector t _k corresponds to a situation in which K cameras are are installed at an angle Θ intervals on the circumference of radius R.

First, focus on the point M ₀ in the image of the 0th camera. Digital image coordinate m ₀ of the point M ₀ is the following was value.

Therefore, it is converted into normalized image coordinates. The normalized image coordinate x ₀ of the point M ₀ has the following value.

  Similarly, the images of the other K-1 cameras are converted from digital image coordinates to normalized image coordinates.

Next, the K camera images are rotationally converted. The rotation transformation matrix in the combination of the 0th camera and the kth camera is as follows based on Equation (16).

The relative position vector in the combination of the 0th camera and the kth camera is converted into the following vector based on Expression (18).

Based on the equation (33), rotation conversion is performed. For example, the point _{M 0} is converted to a point _{M k} below.

And performing a search block of a point M ₀ and the upper left corner.

According to the present invention, block matching is performed by arbitrarily changing the normal vector (n _x , n _y , n _z ) and the depth distance Z. For example, when n _x = 0, n _y = 1, n _z = 0, and Z = R / 2, the depth distance and normal vector in the coordinate system rotationally transformed by the combination of the 0th camera and the kth camera are Using Equation (32) and Equation (19), the following is obtained.

Accordingly, the disparity vector D _k in the combination of the 0th camera and the kth camera at this time is as follows from the equation (37).

Further, the primary transformation matrix L _k in the combination of the 0th camera and the kth camera at this time is as follows from the equation (38).

Here, the normal vector _{(n x, n y, n} z) in and remain depth distance Z was fixed _{(n x = 0, n y} = 1, n z = 0, Z = R / 2), the cameras , The blocks that have been translated based on D _k and linearly transformed based on L _k are compared, and the difference between the two (corresponding blocks in the image of the 0th camera and the image of the kth camera) is evaluated. For example, the least square error is obtained.

The corresponding evaluation value of the difference between blocks in the zeroth camera image and the k camera image is denoted by d _k. At this time, the calculation is performed as follows for each combination of the normal vector and the depth distance.

  The combination of the normal vector and the depth distance that minimizes S is the target disparity vector and the primary transformation matrix.

At this time, if the error of the block matching is the minimum, D ₀ and L ₀ are the target disparity vector and the primary transformation matrix.

  FIG. 10 is an explanatory diagram of modified block matching by two cameras.

  According to FIG. 10, the deformed block matching is independently performed for each image by a one-to-one combination. In this case, a different depth distance and normal vector are required for each combination, and a large amount of calculation is required.

  FIG. 11 is an explanatory diagram of modified block matching using three or more cameras.

According to FIG. 11, a prediction error between one base camera block and another camera block is calculated, and a depth distance Z and a unit normal vector (n _x , n _y , _nz ) is derived. With a set of depth distances and normal vectors, matching of the corresponding blocks of all images can be achieved.

  FIG. 12 is a flowchart in the present invention.

(S901) The position information of all cameras is acquired. All the cameras are arranged on the same plane parallel to the optical axis and perpendicular to the optical axis. Accordingly, the first relative position (t _x , t _y , t _z ) of the world coordinate system of the second camera with respect to the first camera is acquired. The camera may be movable. When the camera itself has a positioning function such as GPS, the position information is received. If the camera is fixed, the position information may be registered in advance.

(S902) All the camera images are acquired. Here, when the optical axes of the first camera and / or the second camera are not parallel to the Z axis, the optical axes of the first camera and the second camera are parallel to the Z axis. Using the parameters, the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera are transformed.

(S903) Using the camera internal parameters, the first camera and the second camera so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. The same rotational transformation is applied to the second camera to transform the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera. The matrix representing the rotation transformation is as follows.

(S904) An arbitrary point on the subject is determined.
(S905) First image coordinates (x, y) obtained by projecting an arbitrary point to the first image subjected to rotation transformation, and a second image obtained by projecting the arbitrary point to the second image subjected to rotation transformation. Image coordinates (x ′, y ′) are derived. Here, the image coordinates of an arbitrary point are normalized image coordinates.
(S906) A first block including an arbitrary point is selected for the first image.

(S907) Hereinafter, S908 to S911 are repeated.
(S908) The depth distance Z of an arbitrary point is arbitrarily determined.
(S909) unit normal vector at any point of the surface of the object _{(n x, n y, n} z) arbitrarily determined.

一次変換行列は、以下のものである。

(S910) Here, as a result of the rotation transformation, a new depth distance Z _R and a new unit normal vector (n _Rx , n _Ry , n _Rz ) and a new relative position (t _Rx , t of an arbitrary viewpoint) _Ry, 0) and the new image coordinates _{(x R} ', _{y R'} and) was obtained. Then, the disparity vector is as follows.

The primary transformation matrix is:

The first block is moved with a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R , and the first image coordinates (x _R , y _R ) are used as the origin, Second relative position (t _Rx , t _Ry , 0), depth distance Z _R , unit normal vector (n _Rx , n _Ry , n _Rz ) and second image coordinates (x _R ′, y _R ′) And is matched with the second image block.

(S911) depth distance Z and unit normal vector _{(n x, n y, n} z) while arbitrarily changed, repeated S907～S910. Then, the second block of the second image that is most similar to the first block of the first image that has been moved and deformed is searched for. As a result, the depth distance Z and the unit normal vector ( _nx , _ny , _nz ) are derived.

(S912) An arbitrary point on the subject is further determined for a portion other than the first block of the first image, and S904 to S911 are repeated. Finally, all parts of an image from an arbitrary viewpoint on a straight line connecting the first camera and the second camera are generated.

(S913) The relative position (t _x ″, t _y ″, 0) of the arbitrary viewpoint according to the internal ratio of the arbitrary viewpoint on the straight line connecting the first camera and the second camera, and Image coordinates (x ″, y ″) are determined.
(S914) depth distance Z and unit normal vector _{(n x, n y, n} z) and the relative position of the arbitrary viewpoints _{(t x '', t y} '', 0) and image coordinates (x '' , Y ″) to generate a block of an image from an arbitrary viewpoint.

  FIG. 13 is a functional configuration diagram of the free viewpoint image generation device according to the present invention.

  According to FIG. 13, the free viewpoint image generation device 2 includes a position information acquisition unit 21, an image acquisition unit 22, a camera coordinate system coordinate conversion unit 203, an arbitrary point selection unit 204, and a normalized image coordinate derivation unit 205. A block selection unit 206, a block matching unit 207, a parameter determination unit 208, a matching control unit 209, a block control unit 210, an arbitrary viewpoint parameter determination unit 211, and a free viewpoint image generation unit 212. These functional units can also be realized by a program executed by a computer.

  The position information acquisition unit 21 acquires position information of all cameras. It has the same function as S901 in FIG.

  The image acquisition unit 22 acquires synchronized images from all cameras. It has the same function as S902 in FIG.

  The camera coordinate system coordinate conversion unit 203 uses the camera internal parameters so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. The same rotation transformation is applied to the first camera and the second camera to transform the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera. It has the same function as S903 in FIG.

  The arbitrary point selection unit 204 determines an arbitrary point on the subject. The arbitrary point selection unit 204 further determines an arbitrary point on the subject for a portion other than the first block of the first image in response to an instruction from the block control unit 210. It has the same function as S904 in FIG.

  The normalized image coordinate deriving unit 205 projects a first normalized image coordinate (x, y) obtained by projecting an arbitrary point onto the first image subjected to rotation transformation, and a second obtained by performing rotation transformation on the arbitrary point. And the second normalized image coordinates (x ′, y ′) projected onto the image. It has the same function as S905 of FIG.

  The block selection unit 206 selects a first block including an arbitrary point for the first image. It has the same function as S906 in FIG.

The block matching unit 207 moves the first block by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z, and the first image coordinates (x _R , y _R ) As the origin, second relative position (t _Rx , t _Ry , 0), depth distance Z, unit normal vector (n _Rx , n _Ry , n _Rz ) and second image coordinates (x _R ′, The first transformation matrix based on _yR ′) is transformed to match the second image block. This is the same as S910 in FIG.

When the block selection unit 206 selects a second block including an arbitrary point with respect to the second image, the block matching unit 207 moves the first image by a parallax vector, and the first image coordinates ( x _{R 1} , y _R ) is used as an origin, and is transformed with a primary transformation matrix to be matched with the second block of the second image.

The parameter determining unit 208 arbitrarily determines the depth distance Z of an arbitrary point and the unit normal vector ( _nx , _ny , _nz ) at the arbitrary point on the surface of the subject. This is the same as S908 and S909 of FIG.

The matching control unit 209 repeatedly controls the parameter determination unit and the block matching unit while arbitrarily changing the depth distance Z and the unit normal vector (n _x , n _y , n _z ), and moves and deforms the first unit. The second block of the second image that is most similar to the first block of one image is searched to derive the depth distance Z and the unit normal vector ( _nx , _ny , _nz ). This is the same as S907 and S911 of FIG.

  The block control unit 210 repeats the normalized image coordinate calculation unit 205, the parameter determination unit 208, the block matching unit 207, and the matching control unit 209 for an arbitrary point. Finally, all parts of an image from an arbitrary viewpoint on a straight line connecting the first camera and the second camera are generated. This is the same as S904 and S912 in FIG.

The arbitrary viewpoint parameter determination unit 211 determines the relative position (t _x ″, t _y ′) of the arbitrary viewpoint according to the internal division ratio of the arbitrary viewpoint on the straight line connecting the first camera and the second camera. ', 0) and image coordinates (x ″, y ″) are determined. This is the same as S913 in FIG.

The free viewpoint image generation unit 212 includes a depth distance Z and a unit normal vector (n _x , n _y , n _z ), a relative position (t _x ″, t _y ″, 0) of an arbitrary viewpoint, and an image. A block of an image from an arbitrary viewpoint is generated using the coordinates (x ″, y ″). This is the same as S914 in FIG.

  As described above in detail, according to the free viewpoint image generation method, apparatus, and program of the present invention, even if the surface of the subject is inclined with respect to the optical axes of a plurality of cameras, block matching is performed. Can detect a disparity vector and generate an image of an arbitrary viewpoint using the disparity vector. By performing block matching while moving and deforming the first block of the first image based on an arbitrary depth distance Z and an arbitrary direction n of the normal of the subject surface, not only the disparity vector The primary transformation matrix can be detected. By adjusting the parallax vector and the primary transformation matrix according to the position of an arbitrary viewpoint, an image of an arbitrary viewpoint close to reality can be generated from the first block of the first image.

  According to the various embodiments of the present invention described above, those skilled in the art can easily make various changes, modifications and omissions within the scope of the technical idea and the viewpoint of the present invention. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

It is a system configuration diagram in the present invention. FIG. 2 is a layout view of a plurality of cameras viewed from the front and side in the system configuration of FIG. 1. FIG. 3 is a screen diagram showing how a subject is viewed from each camera in FIG. 2. This is an image taken from each camera when the surface of the subject is tilted in the horizontal direction. This is an image taken from each camera when the surface of the subject is tilted in the vertical direction. It is an image taken from each camera when the surface of the subject is tilted obliquely in the horizontal direction and tilted in the vertical direction. It is explanatory drawing of the 1st relative positional relationship in two cameras. It is explanatory drawing of the 2nd relative positional relationship which added rotation to two cameras. It is explanatory drawing which made the block matching of the 1st block of the 1st image of the camera C and the 2nd block of the 2nd image of the camera C '. It is explanatory drawing of the deformation | transformation block matching by two cameras. It is explanatory drawing of the deformation | transformation block matching by three or more cameras. It is a flowchart in this invention. It is a functional block diagram of the free viewpoint image generation apparatus in this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera 2 Free viewpoint image generation apparatus 201 Position information acquisition part 202 Image acquisition part 203 Camera coordinate system coordinate transformation part 204 Arbitrary point selection part 205 Normalized image coordinate derivation part 206 Block selection part 207 Block matching part 208 Parameter determination part 209 Matching Control unit 210 Block control unit 211 Arbitrary viewpoint parameter determination unit 212 Free viewpoint image generation unit 3 Subject

Claims (11)

A first camera and a second camera having an optical axis parallel to the Z axis are arranged at a first relative position (t _x , t _y , t _z ) (where t _z ≠ 0); A free viewpoint image generation method for generating an image of an arbitrary viewpoint using a first image of a first camera and a second image of a second camera that photograph a subject,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. A first step of applying the same rotational transformation to the camera to transform the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera;
A second step of determining an arbitrary point on the subject;
First image coordinates (x _R , y _R ) obtained by projecting the arbitrary point onto the first image of the first camera subjected to the rotation transformation, and a second image obtained by applying the rotation transformation to the arbitrary point. A third step of deriving second image coordinates (x _R ′, y _R ′) projected onto the second image of the camera of
A fourth step of selecting a first block including an arbitrary point for the first image;
A depth distance Z of an arbitrary point and a unit normal vector ( _nx , _ny , _nz ) at an arbitrary point on the surface of the subject are arbitrarily determined, and the depth distance Z _{R with} respect to the same plane subjected to the rotation transformation And a fifth step of deriving a unit normal vector (n _Rx , n _Ry , n _Rz ),
The first block is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotational transformation is applied, and the first image coordinates are used as the origin and 2 relative positions (t _Rx , t _Ry , 0), depth distance Z _R , unit normal vector (n _Rx , n _Ry , n _Rz ) and second image coordinates (x _R ′, y _R ′). A sixth step of transforming with a primary transformation matrix based to match a block of the second image;
The first block of the first image in which the depth distance Z and the unit normal vector (n _x , n _y , n _z ) are arbitrarily changed, and the fifth and sixth steps are repeated, moved, and deformed. Searching for a second block of a second image most similar to, and deriving a depth distance Z and unit normal vectors ( _nx , _ny , _nz );
Relative position (t _x ″, t _y ″, t _z ″) of the arbitrary viewpoint and image coordinates (in accordance with the internal ratio of the arbitrary viewpoint on the straight line connecting the first camera and the second camera) an eighth step of determining x ″, y ″);
Depth distance Z and unit normal vector _{(n x, n y, n} z) and the relative position of the arbitrary viewpoints _{(t x ", t y"} , t z ") and image coordinates (x", y ") And a ninth step of generating a block of an image from an arbitrary viewpoint. An arbitrary point on the subject is further determined for a portion other than the first block of the first image, and the second to seventh steps are repeated,
The free viewpoint image generation method according to claim 1, wherein all parts of an image from an arbitrary viewpoint on a straight line connecting the first camera and the second camera are finally generated. For the first step, the matrix representing the rotational transformation is:

For the second step, the image coordinates of the arbitrary point are normalized image coordinates,
As a result of the rotation transformation, a new depth distance Z _R and a new unit normal vector (n _Rx , n _Ry , n _Rz ), a new relative position (t _Rx , t _Ry , 0) of an arbitrary viewpoint, and Suppose that new image coordinates (x _R ', y _R ') are obtained,
For the sixth step, the disparity vector is:

The primary transformation matrix is

The method of generating a free viewpoint image according to claim 1 or 2. When the optical axis of the first camera and / or the second camera is not parallel to the Z axis,
The camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera are converted using camera internal parameters so that the optical axes of the first camera and the second camera are parallel to the Z axis. The free viewpoint image generation method according to claim 1, further comprising a step of: Even when three or more cameras are arranged, the depth distance Z and the unit normal vector ( _nx , _ny , _nz ) are constant,
In a seventh step, a prediction error between one base camera block and another camera block is calculated, and a depth distance Z and a unit normal vector (n _x , n _y , The method of generating a free viewpoint image according to any one of claims 1 to 4, wherein _nz ) is derived. A first camera and a second camera having an optical axis parallel to the Z axis are arranged at a first relative position (t _x , t _y , t _z ) (where t _z ≠ 0); A free viewpoint image generation device that generates an image of an arbitrary viewpoint using a first image of a first camera and a second image of a second camera that photograph a subject,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. Camera coordinate system coordinate conversion means for converting the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera by applying the same rotational transformation to the camera;
Arbitrary point selection means for determining an arbitrary point on the subject;
First image coordinates (x _R , y _R ) obtained by projecting the arbitrary point onto the first image of the first camera subjected to the rotation transformation, and a second image obtained by applying the rotation transformation to the arbitrary point. Image coordinate calculation means for deriving second image coordinates (x _R ′, y _R ′) projected onto the second image of the camera of
Block selection means for selecting a first block including an arbitrary point for the first image;
A depth distance Z of an arbitrary point and a unit normal vector ( _nx , _ny , _nz ) at an arbitrary point on the surface of the subject are arbitrarily determined, and the depth distance Z _{R with} respect to the same plane subjected to the rotation transformation And parameter determining means for deriving unit normal vectors (n _Rx , n _Ry , n _Rz ),
The first block is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotational transformation is applied, and the first image coordinates are used as the origin and 2 relative positions (t _Rx , t _Ry , 0), depth distance Z _R , unit normal vector (n _Rx , n _Ry , n _Rz ) and second image coordinates (x _R ′, y _R ′). A block matching means for deforming with a primary transformation matrix based on and matching with a block of the second image;
While arbitrarily changing the depth distance Z and the unit normal vector (n _x , n _y , n _z ), the parameter determination means and the block matching means are repeatedly controlled to move and deform the first of the first image Matching control means for searching for a second block of the second image most similar to the block and deriving a depth distance Z and a unit normal vector ( _nx , _ny , _nz );
Relative position (t _x ″, t _y ″, t _z ″) of the arbitrary viewpoint and image coordinates (in accordance with the internal ratio of the arbitrary viewpoint on the straight line connecting the first camera and the second camera) arbitrary viewpoint parameter determining means for determining x ″, y ″);
Depth distance Z and unit normal vector _{(n x, n y, n} z) and the relative position of the arbitrary viewpoints _{(t x ", t y"} , t z ") and image coordinates (x", y ") And a free viewpoint image generation means for generating a block of an image from an arbitrary viewpoint. The arbitrary point selecting means further determines an arbitrary point on the subject for a portion other than the first block of the first image;
A block control unit that repeats an image coordinate calculation unit, a parameter determination unit, a block matching unit, a matching control unit, an arbitrary viewpoint parameter determination unit, and a free viewpoint image generation unit for the arbitrary point;
7. The free viewpoint image generation apparatus according to claim 6, wherein all parts of an image from an arbitrary viewpoint on a straight line connecting the first camera and the second camera are finally generated. For the camera coordinate system coordinate transformation means, the matrix representing the rotation transformation is as follows:

The image coordinate calculation means derives normalized image coordinates as the image coordinates of the arbitrary point,
As a result of the rotation transformation, a new depth distance Z _R and a new unit normal vector (n _Rx , n _Ry , n _Rz ), a new relative position (t _Rx , t _Ry , 0) of an arbitrary viewpoint, and Suppose that new image coordinates (x _R ', y _R ') are obtained,
The block matching means sets the disparity vector as follows:

The primary transformation matrix is

The free viewpoint image generation device according to claim 6 or 7, characterized in that. When the optical axis of the first camera and / or the second camera is not parallel to the Z axis,
The camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera are converted using camera internal parameters so that the optical axes of the first camera and the second camera are parallel to the Z axis. The free viewpoint image generation device according to any one of claims 6 to 8, wherein: Even when three or more cameras are arranged, the depth distance Z and the unit normal vector ( _nx , _ny , _nz ) are constant,
In a seventh step, a prediction error between one base camera block and another camera block is calculated, and a depth distance Z and a unit normal vector (n _x , n _y , _nz ) is derived, The free viewpoint image generation device according to any one of claims 6 to 9. A first camera and a second camera having an optical axis parallel to the Z axis are arranged at a first relative position (t _x , t _y , t _z ) (where t _z ≠ 0); A free viewpoint image generation program for causing a computer that generates an image of an arbitrary viewpoint to function using a first image of a first camera and a second image of a second camera that photograph a subject,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. Camera coordinate system coordinate conversion means for converting the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera by applying the same rotational transformation to the camera;
Arbitrary point selection means for determining an arbitrary point on the subject;
First image coordinates (x _R , y _R ) obtained by projecting the arbitrary point onto the first image of the first camera subjected to the rotation transformation, and a second image obtained by applying the rotation transformation to the arbitrary point. Image coordinate calculation means for deriving second image coordinates (x _R ′, y _R ′) projected onto the second image of the camera of
Block selection means for selecting a first block including an arbitrary point for the first image;
A depth distance Z of an arbitrary point and a unit normal vector ( _nx , _ny , _nz ) at an arbitrary point on the surface of the subject are arbitrarily determined, and the depth distance Z _{R with} respect to the same plane subjected to the rotation transformation And parameter determining means for deriving unit normal vectors (n _Rx , n _Ry , n _Rz ),
The first block is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotational transformation is applied, and the first image coordinates are used as the origin and 2 relative positions (t _Rx , t _Ry , 0), depth distance Z _R , unit normal vector (n _Rx , n _Ry , n _Rz ) and second image coordinates (x _R ′, y _R ′). A block matching means for deforming with a primary transformation matrix based on and matching with a block of the second image;
While arbitrarily changing the depth distance Z and the unit normal vector (n _x , n _y , n _z ), the parameter determination means and the block matching means are repeatedly controlled to move and deform the first of the first image Matching control means for searching for a second block of the second image most similar to the block and deriving a depth distance Z and a unit normal vector ( _nx , _ny , _nz );
Relative position (t _x ″, t _y ″, t _z ″) of the arbitrary viewpoint and image coordinates (in accordance with the internal ratio of the arbitrary viewpoint on the straight line connecting the first camera and the second camera) arbitrary viewpoint parameter determining means for determining x ″, y ″);
Depth distance Z and unit normal vector _{(n x, n y, n} z) and the relative position of the arbitrary viewpoints _{(t x ", t y"} , t z ") and image coordinates (x", y ") And a free viewpoint image generation program that causes a computer to function as free viewpoint image generation means for generating a block of an image from an arbitrary viewpoint.

Images