JP4605716B2 - Multi-view image compression encoding method, apparatus, and program - Google Patents

Description

  The present invention relates to a multi-view image compression encoding method, apparatus, and program.

  There is a strong correlation between images simultaneously captured by cameras at different positions (hereinafter referred to as “multi-viewpoint images”), except for differences due to parallax. Accordingly, information can be compressed by regarding these images as a series of video sequences and encoding them using motion compensation (parallax compensation) (see, for example, Patent Document 1).

  “Block matching” is a typical method for performing parallax compensation between multi-viewpoint images. Detection of a disparity vector using “block matching” is performed as follows.

  The first image taken by the first camera (first viewpoint) is divided into small blocks. Each of the first blocks of the first image is translated in the second image of the second camera (other viewpoint) to search for a region having the highest similarity. Specifically, the second block of the second image having the smallest absolute value error or square error is searched for the first block of the first image. Then, the distance amount between the searched second block and the first block is calculated as a disparity vector. By encoding the prediction error between the first block and the second block and adding the disparity vector to the encoded data, the multi-viewpoint image can be compression encoded.

JP 2005-260464 A

  Conventional block matching is to match blocks when a subject in a first image is translated in a second image. Only when the surface of the subject is perpendicular to the optical axis of the camera as viewed from a plurality of cameras and is equidistant from the camera, complete block matching can be performed.

  However, when the surface of the subject is inclined with respect to the optical axes of a plurality of cameras, the surface of the subject also looks different. That is, even if the first block of the first image is translated in the second image, it is not possible to search for the second block that perfectly matches the first block. This is because the shape of the subject in the first image is different from the shape of the subject in the second image.

  Therefore, the present invention detects a disparity vector by block matching even when the surface of an object is tilted with respect to the optical axes of a plurality of cameras, and performs multi-viewpoint image compression encoding using the disparity vector. It is an object to provide a method, an apparatus, and a program.

According to the present invention, the first camera and the second camera whose optical axes are parallel to the Z axis have the first relative position (t _x , t _y , t _z ) (where t _z ≠ 0). Is a multi-viewpoint image compression encoding method using a first image of a first camera and a second image of a second camera that photograph a subject,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. A first step of applying the same rotational transformation to the camera to transform the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera;
A second step of determining an arbitrary point on the subject;
The first image coordinates (x _R , y _R ) obtained by projecting the arbitrary point to the first image of the first camera to which the rotation transformation is applied, and the second image of the second camera to which the arbitrary point is subjected to the rotation transformation. A third step of deriving second image coordinates (x _R ′, y _R ′) projected onto the second image;
A fourth step of selecting a first block including an arbitrary point for the first image;
A depth distance Z of any point, unit normal vector at any point of the surface of the object _{(n x, n y, n} z) and arbitrarily determine, depth distance Z _R and the unit for the same plane plus rotational transformation A fifth step of deriving normal vectors (n _Rx , n _Ry , n _Rz );
The first block is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotation transformation is applied, and the second image is used with the first image coordinates as the origin. Based on the relative position (t _Rx , t _Ry , 0), the depth distance Z _R , the unit normal vector (n _Rx , n _Ry , n _Rz ) and the second image coordinates (x _R ′, y _R ′). A sixth step of transforming with a primary transformation matrix to match a block of the second image;
The first block of the first image in which the depth distance Z and the unit normal vector (n _x , n _y , n _z ) are arbitrarily changed, and the fifth and sixth steps are repeated, moved, and deformed. Searching for a second block of a second image most similar to, and deriving a depth distance Z and unit normal vectors ( _nx , _ny , _nz );
An eighth step of deriving and encoding a prediction error between the moved and deformed first block of the first image and the second block of the second image that is most similar;
And a ninth step of adding the derived depth distance Z and the unit normal vector (n _x , n _y , n _z ) to the encoded data.

According to another embodiment of the multi-view image compression encoding method of the present invention,
The eighth step uses the MPEG (Moving Picture Experts Group) prediction error encoding method,
It is also preferable that the first image is a picture to be encoded and the second image is a reference picture.

According to the present invention, the first camera and the second camera whose optical axes are parallel to the Z axis have the first relative position (t _x , t _y , t _z ) (where t _z ≠ 0). Is a multi-viewpoint image compression encoding method using a first image of a first camera and a second image of a second camera that photograph a subject,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. A first step of applying the same rotational transformation to the camera to transform the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera;
A second step of determining an arbitrary point on the subject;
The first image coordinates (x _R , y _R ) obtained by projecting the arbitrary point to the first image of the first camera to which the rotation transformation is applied, and the second image of the second camera to which the arbitrary point is subjected to the rotation transformation. A third step of deriving second image coordinates (x _R ′, y _R ′) projected onto the second image;
A fourth step of selecting a second block containing an arbitrary point for the second image;
A depth distance Z of any point, unit normal vector at any point of the surface of the object _{(n x, n y, n} z) and arbitrarily determine, depth distance Z _R and the unit for the same plane plus rotational transformation A fifth step of deriving normal vectors (n _Rx , n _Ry , n _Rz );
The first image is moved by the disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotation transformation is applied, and the first image coordinates (x _R , y _R ) are moved. ) As the origin, the second relative position (t _Rx , t _Ry , 0), the depth distance Z _R , the unit normal vector (n _Rx , n _Ry , n _Rz ), and the second image coordinates (x _R ′) , Y _R ′) with a first transformation matrix to match with the second block of the second image,
Depth distance Z and unit normal vector _{(n x, n y, n} z) while arbitrarily changed, repeating the fifth and sixth step of, first it is most similar to the second block of the second image 1 Searching for the first block of the image and deriving a depth distance Z and a unit normal vector ( _nx , _ny , _nz );
An eighth step of derived and coding prediction error between the first block of the first image that is most similar to the second block of the second image,
And a ninth step of adding the derived depth distance Z and the unit normal vector (n _x , n _y , n _z ) to the encoded data.

According to another embodiment of the multi-view image compression encoding method of the present invention,
The eighth step uses the MPEG prediction error encoding method,
It is also preferable that the first image is a reference picture and the second image is an encoding target picture.

According to another embodiment of the multi-view image compression encoding method of the present invention,
While repeating the second to ninth steps ,
In the iteration, the second step further determines arbitrary points on the subject for parts other than the block previously selected by the fourth step ,
Finally, it preferred to as benzalkonium compress encoding all parts of the first and second images.

第２のステップについて、任意点の画像座標は、正規化画像座標であり、
回転変換の結果、新たな奥行き距離Ｚ_Ｒ及び新たな単位法線ベクトル（ｎ_Ｒｘ，ｎ_Ｒｙ，ｎ_Ｒｚ）と、任意の視点の新たな相対的位置（ｔ_Ｒｘ，ｔ_Ｒｙ，０）及び新たな画像座標（ｘ_Ｒ’，ｙ_Ｒ’）が得られたとし、
第６のステップについて、視差ベクトルは、以下のものであり、

一次変換行列は、以下のものである

ことも好ましい。 According to another embodiment of the multi-view image compression encoding method of the present invention,
For the first step, the matrix representing the rotation transformation is:

For the second step, the image coordinates of the arbitrary point are normalized image coordinates,
As a result of the rotation transformation, a new depth distance Z _R and a new unit normal vector (n _Rx , n _Ry , n _Rz ), a new relative position (t _Rx , t _Ry , 0) of the arbitrary viewpoint, and a new Suppose that the correct image coordinates (x _R ', y _R ') are obtained,
For the sixth step, the disparity vector is:

The primary transformation matrix is

It is also preferable.

According to another embodiment of the multi-view image compression encoding method of the present invention,
When the optical axis of the first camera and / or the second camera is not parallel to the Z axis,
The camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera are converted using camera internal parameters so that the optical axes of the first camera and the second camera are parallel to the Z axis. It is also preferable to have the step to do.

According to another embodiment of the multi-view image compression encoding method of the present invention,
Even when three or more cameras are arranged, the depth distance Z and the unit normal vector ( _nx , _ny , _nz ) are constant,
In a seventh step, a prediction error between one base camera block and another camera block is calculated, and a depth distance Z and a unit normal vector (n _x , n _y , It is also preferable to derive _nz ).

According to the present invention, the first camera and the second camera whose optical axes are parallel to the Z axis have the first relative position (t _x , t _y , t _z ) (where t _z ≠ 0). Is a multi-viewpoint image compression encoding apparatus using a first image of a first camera and a second image of a second camera that photograph a subject.
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. Camera coordinate system coordinate conversion means for converting the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera by applying the same rotational transformation to the camera;
Arbitrary point selection means for determining an arbitrary point on the subject;
The first image coordinates (x _R , y _R ) obtained by projecting an arbitrary point to the first image of the first camera to which the rotational transformation is applied, and the arbitrary point of the second camera to which the rotational transformation is applied. Image coordinate calculation means for deriving second image coordinates (x _R ′, y _R ′) projected on the second image;
Block selection means for selecting a first block including an arbitrary point for the first image;
A depth distance Z of any point, unit normal vector at any point of the surface of the object _{(n x, n y, n} z) and arbitrarily determine, depth distance Z _R and the unit for the same plane plus rotational transformation Parameter determining means for deriving normal vectors (n _Rx , n _Ry , n _Rz );
The first block is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotation transformation is applied, and the second image is used with the first image coordinates as the origin. Based on the relative position (t _Rx , t _Ry , 0), the depth distance Z _R , the unit normal vector (n _Rx , n _Ry , n _Rz ) and the second image coordinates (x _R ′, y _R ′). Block matching means for transforming with a primary transformation matrix and matching with a block of the second image;
While arbitrarily changing the depth distance Z and the unit normal vector (n _x , n _y , n _z ), the parameter determination means and the block matching means are repeatedly controlled to move and deform the first of the first image Matching control means for searching for a second block of the second image most similar to the block and deriving a depth distance Z and a unit normal vector ( _nx , _ny , _nz );
Prediction error encoding means for deriving and encoding a prediction error between the moved and deformed first block of the first image and the second block of the second image most similar to the first block;
Parameter addition means for adding the derived depth distance Z and the unit normal vector (n _x , n _y , n _z ) to the encoded data.

According to another embodiment of the multi-view image compression encoding apparatus of the present invention,
The prediction error encoding means uses the MPEG prediction error encoding method,
It is also preferable that the first image is a picture to be encoded and the second image is a reference picture.

According to the present invention, the first camera and the second camera whose optical axes are parallel to the Z axis have the first relative position (t _x , t _y , t _z ) (where t _z ≠ 0). Is a multi-viewpoint image compression encoding apparatus using a first image of a first camera and a second image of a second camera that photograph a subject.
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. Camera coordinate system coordinate conversion means for converting the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera by applying the same rotational transformation to the camera;
Arbitrary point selection means for determining an arbitrary point on the subject;
The first image coordinates (x _R , y _R ) obtained by projecting the arbitrary point to the first image of the first camera to which the rotation transformation is applied, and the second image of the second camera to which the arbitrary point is the rotation transformation. Image coordinate calculating means for deriving second image coordinates (x _R ′, y _R ′) projected onto the image of 2;
Block selecting means for selecting a second block including an arbitrary point for the second image;
A depth distance Z of any point, unit normal vector at any point of the surface of the object _{(n x, n y, n} z) and arbitrarily determine, depth distance Z _R and the unit for the same plane plus rotational transformation Parameter determining means for deriving normal vectors (n _Rx , n _Ry , n _Rz );
The first image is moved by the disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotation transformation is applied, and the first image coordinates (x _R , y _R ) are moved. ) As the origin, the second relative position (t _Rx , t _Ry , 0), the depth distance Z _R , the unit normal vector (n _Rx , n _Ry , n _Rz ), and the second image coordinates (x _R ′) , Y _R ′), a block matching means for transforming with a primary transformation matrix to match the second block of the second image;
Depth distance Z and unit normal vector _{(n x, n y, n} z) while changing the optionally repeatedly control parameter determining means and the block matching means, most similar to the second block of the second image and matching control means for deriving a first block of the first image and search, depth distance Z and unit normal vector _{(n x, n y, n} z) and,
And prediction error encoding means for deriving and and encodes the prediction error between the first block of the first image that is most similar to the second block of the second image,
Parameter addition means for adding the derived depth distance Z and the unit normal vector (n _x , n _y , n _z ) to the encoded data.

According to another embodiment of the multi-view image compression encoding apparatus of the present invention,
The prediction error encoding means uses the MPEG prediction error encoding method,
It is also preferable that the first image is a reference picture and the second image is an encoding target picture.

According to another embodiment of the multi-view image compression encoding apparatus of the present invention,
And the arbitrary point selecting means, and the previous SL image coordinate calculation unit, and said block selecting means, and parameter determining means, a block matching unit, and a matching control unit, and the prediction error encoding unit, the processing of a parameter adding means further it has a block control means for repeating,
In the repetition, the arbitrary point selecting means further determines an arbitrary point on the subject for a part other than the block previously selected by the block selecting means ,
Finally, it preferred to as benzalkonium compress encoding all parts of the first and second images.

画像座標算出手段は、任意点の画像座標として正規化画像座標を導出し、
回転変換の結果、新たな奥行き距離Ｚ_Ｒ及び新たな単位法線ベクトル（ｎ_Ｒｘ，ｎ_Ｒｙ，ｎ_Ｒｚ）と、任意の視点の新たな相対的位置（ｔ_Ｒｘ，ｔ_Ｒｙ，０）及び新たな画像座標（ｘ_Ｒ’，ｙ_Ｒ’）が得られたとし、
ブロックマッチング手段は、視差ベクトルを以下のものとし、

一次変換行列を以下のものとする

ことも好ましい。 According to another embodiment of the multi-view image compression encoding apparatus of the present invention,
For the camera coordinate system coordinate transformation means, the matrix representing the rotation transformation is as follows:

The image coordinate calculation means derives normalized image coordinates as image coordinates of an arbitrary point,
As a result of the rotation transformation, a new depth distance Z _R and a new unit normal vector (n _Rx , n _Ry , n _Rz ), a new relative position (t _Rx , t _Ry , 0) of the arbitrary viewpoint, and a new Suppose that the correct image coordinates (x _R ', y _R ') are obtained,
The block matching means uses the following disparity vectors:

Let the primary transformation matrix be

It is also preferable.

According to another embodiment of the multi-view image compression encoding apparatus of the present invention,
When the optical axis of the first camera and / or the second camera is not parallel to the Z axis,
The camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera are converted using camera internal parameters so that the optical axes of the first camera and the second camera are parallel to the Z axis. It is also preferable to do.

According to another embodiment of the multi-view image compression encoding apparatus of the present invention,
Even when three or more cameras are arranged, the depth distance Z and the unit normal vector ( _nx , _ny , _nz ) are constant,
In a seventh step, a prediction error between one base camera block and another camera block is calculated, and a depth distance Z and a unit normal vector (n _x , n _y , It is also preferable to derive _nz ).

According to the present invention, the first camera and the second camera whose optical axes are parallel to the Z axis have the first relative position (t _x , t _y , t _z ) (where t _z ≠ 0). And a multi-viewpoint that causes the computer to function to compress and encode a multi-viewpoint image using the first image of the first camera and the second image of the second camera that photograph the subject. An image compression encoding program,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. Camera coordinate system coordinate conversion means for converting the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera by applying the same rotational transformation to the camera;
Arbitrary point selection means for determining an arbitrary point on the subject;
The first image coordinates (x _R , y _R ) obtained by projecting the arbitrary point to the first image of the first camera to which the rotation transformation is applied, and the second image of the second camera to which the arbitrary point is the rotation transformation. Image coordinate calculating means for deriving second image coordinates (x _R ′, y _R ′) projected onto the image of 2;
Block selection means for selecting a first block including an arbitrary point for the first image;
A depth distance Z of any point, unit normal vector at any point of the surface of the object _{(n x, n y, n} z) and arbitrarily determine, depth distance Z _R and the unit for the same plane plus rotational transformation Parameter determining means for deriving normal vectors (n _Rx , n _Ry , n _Rz );
The first block is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotation transformation is applied, and the second image is used with the first image coordinates as the origin. Based on the relative position (t _Rx , t _Ry , 0), the depth distance Z _R , the unit normal vector (n _Rx , n _Ry , n _Rz ) and the second image coordinates (x _R ′, y _R ′). Block matching means for transforming with a primary transformation matrix and matching with a block of the second image;
While arbitrarily changing the depth distance Z and the unit normal vector (n _x , n _y , n _z ), the parameter determination means and the block matching means are repeatedly controlled to move and deform the first of the first image Matching control means for searching for a second block of the second image most similar to the block and deriving a depth distance Z and a unit normal vector ( _nx , _ny , _nz );
Prediction error encoding means for deriving and encoding a prediction error between the moved and deformed first block of the first image and the second block of the second image most similar to the first block;
The computer is caused to function as parameter addition means for adding the derived depth distance Z and the unit normal vector ( _nx , _ny , _nz ) to the encoded data.

According to the present invention, the first camera and the second camera whose optical axes are parallel to the Z axis have the first relative position (t _x , t _y , t _z ) (where t _z ≠ 0). And a multi-viewpoint that causes the computer to function to compress and encode a multi-viewpoint image using the first image of the first camera and the second image of the second camera that photograph the subject. An image compression encoding program,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. Camera coordinate system coordinate conversion means for converting the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera by applying the same rotational transformation to the camera;
Arbitrary point selection means for determining an arbitrary point on the subject;
The first image coordinates (x _R , y _R ) obtained by projecting the arbitrary point to the first image of the first camera to which the rotation transformation is applied, and the second image of the second camera to which the arbitrary point is subjected to the rotation transformation. Image coordinate calculating means for deriving second image coordinates (x _R ′, y _R ′) projected onto the image of 2;
Block selecting means for selecting a second block including an arbitrary point for the second image;
A depth distance Z of any point, unit normal vector at any point of the surface of the object _{(n x, n y, n} z) and arbitrarily determine, depth distance Z _R and the unit for the same plane plus rotational transformation Parameter determining means for deriving normal vectors (n _Rx , n _Ry , n _Rz );
The first image is moved by the disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotation transformation is applied, and the first image coordinates (x _R , y _R ) are moved. ) As the origin, the second relative position (t _Rx , t _Ry , 0), the depth distance Z _R , the unit normal vector (n _Rx , n _Ry , n _Rz ), and the second image coordinates (x _R ′) , Y _R ′), a block matching means for transforming with a primary transformation matrix to match the second block of the second image;
Depth distance Z and unit normal vector _{(n x, n y, n} z) while changing the optionally repeatedly control parameter determining means and the block matching means, most similar to the second block of the second image and matching control means for deriving a first block of the first image and search, depth distance Z and unit normal vector _{(n x, n y, n} z) and,
And prediction error encoding means for deriving and and encodes the prediction error between the first block of the first image that is most similar to the second block of the second image,
The computer is caused to function as parameter addition means for adding the derived depth distance Z and the unit normal vector ( _nx , _ny , _nz ) to the encoded data.

  According to the multi-viewpoint image compression encoding method, apparatus, and program of the present invention, even when the surface of the subject is inclined with respect to the optical axes of a plurality of cameras, a disparity vector is detected by block matching, A multi-viewpoint image can be compression-encoded using the disparity vector. According to MPEG, block matching is performed while moving and deforming the first block of the picture to be encoded or the reference picture itself based on an arbitrary depth distance Z and an arbitrary direction n of the normal of the subject surface. By doing so, a primary transformation matrix is also detected with a parallax vector. By deriving the prediction error between the block or image deformed based on the primary transformation matrix with the first image coordinates as the origin and the other block, the data amount of the prediction error is reduced, and the multi-viewpoint image is highly efficient. Can be compression-encoded.

  Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a system configuration diagram according to the present invention.

  A plurality of cameras 1 arranged at different positions capture the subject 3. The plurality of cameras 1 are arranged with the optical axis parallel to the Z axis. However, these cameras are not arranged on the same plane perpendicular to the optical axis (a plane parallel to the XY plane). In other words, they are arranged scattered in the front, rear, left and right. According to FIG. 1, nine units are arranged in three vertical rows and three horizontal rows (viewpoints (0, 0) to (2, 2)). The plurality of cameras 1 are connected to a multi-view image compression encoding device 2. According to the present invention, multi-viewpoint images taken by at least two cameras are required.

  The camera 1 transmits an image obtained by photographing the subject 3 to the multi-viewpoint image compression encoding device 2. Here, when the camera 1 is movable, the camera position information is also transmitted to the multi-viewpoint image compression coding apparatus 2. Of course, the multi-view image compression encoding apparatus 2 may store all camera position information in advance.

  FIG. 2 is a layout view of a plurality of cameras viewed from the front and side in the system configuration of FIG.

  According to FIG. 2, when viewed from the front, they are arranged in three rows and three rows in the same width. When viewed from the side, the cameras (0, 1) (1, 1) (2, 1) are arranged rearward with respect to the z-axis. That is, with respect to the subject 3, the plurality of cameras 1 are arranged on the cylindrical surface with respect to the y-axis.

  FIG. 3 is a screen diagram showing how the subject is seen from each camera in FIG.

  According to FIG. 3, the subject is a trapezoidal solid. The camera (1, 1) is located in front of the subject. Since the cameras (0, 1) (1, 1) (2, 1) are arranged rearward with respect to the z-axis, the subject looks a little small. When viewed from the left camera (1, 0) in front of the camera (1, 1), the subject appears to be large as a whole, the left side is widened, and the right side is reduced. Further, when viewed from the right camera (1, 2) in front of the camera (1, 1), the subject is large as a whole, the left side is reduced, and the right side is expanded.

  Similarly, when viewed from the camera (0, 1) above the camera (1, 1), the upper side of the subject appears to expand and the lower side appears to be reduced. When viewed from the lower camera (2, 1) of the camera (1, 1), the upper side of the subject is reduced and the lower side is expanded.

  As is clear from FIG. 3, even if the block is moved in consideration of parallax, there is no region that matches exactly. Therefore, correct matching cannot be obtained.

  4 to 6 show images taken by each camera according to the inclination of the surface of the subject.

  FIG. 4 is an image taken from each camera when the surface of the subject is inclined obliquely in the horizontal direction.

  The subject is on the back on the left and the front on the right. At this time, the image of the camera (1, 0) in front of the camera (1, 1) and the camera (1, 0) on the left side is larger than the image of the camera (1, 1). It seems to spread. Also, the image of the camera (1, 2) on the right side of the camera (1, 1) in front of the camera (1, 1) is larger as a whole and narrower in the lateral direction than the image of the camera (1, 1). appear.

  FIG. 5 is an image taken from each camera when the surface of the subject is tilted in the vertical direction.

  The subject has an upper side at the back and a lower side at the front. At this time, the plane of the image of the camera (0, 1) on the upper side of the camera (1, 1) appears to expand in the vertical direction than the image of the camera (1, 1). Further, the plane of the image of the camera (2, 1) below the camera (1, 1) appears narrower in the vertical direction than the image of the camera (1, 1).

  FIG. 6 is an image taken from each camera when the surface of the subject is tilted obliquely in the horizontal direction and tilted in the vertical direction. In other words, the slope of FIG. 4 and the slope of FIG. 5 are combined.

  As shown in FIGS. 3 and 4 to 6, such a difference in appearance occurs because the side surface of the trapezoidal solid of the subject is inclined with respect to the optical axes of a plurality of cameras. According to the present invention, even when the surface of the subject is inclined with respect to the optical axis of the camera, it is possible to search for a second block that perfectly matches the first block.

  FIG. 7 is an explanatory diagram of a first relative positional relationship between the two cameras.

In general, since the subject is not a plane, the deformation of the block is also nonlinear. However, if attention is paid to a sufficiently small block, the surface of the subject in the block can be approximated to a plane. The two cameras are arranged at the first relative position (t _x , t _y , t _z ) with the optical axes in parallel. According to FIG. 7, the xy planes of the camera C and the camera C ′ are at different positions with respect to the subject. Therefore, t _z ≠ 0.

  FIG. 8 is an explanatory diagram of a second relative positional relationship in which two cameras are rotated.

According to the present invention, the same rotational transformation is applied to camera C and camera C ′ using camera internal parameters. This rotation R causes the optical axes of camera C and camera C ′ to be perpendicular to the same plane passing through camera C and camera C ′. In this way, the camera coordinate system coordinates of the camera C and the camera coordinate system coordinates of the camera C ′ are converted. As a result, the second relative positions of the camera C and the camera C ′ are (t _Rx , t _Ry , 0). That is, t _Rz = 0.

And the rotation R in addition to the camera coordinate system coordinates, the depth distance Z _R from the xy plane to any point of the object, and orientation (the unit normal vector) n _R of the approximate plane, is used as a parameter for block matching.

  FIG. 9 is an explanatory diagram of the first block matching.

MPEG is applied to the predictive coding method. According to FIG. 9, the first block including an arbitrary point is selected for the picture to be encoded (first image). Next, the depth distance Z of the arbitrary point and the unit normal vector ( _nx , _ny , _nz ) at the arbitrary point on the surface of the subject are arbitrarily determined. Then, the first block is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R , and the first image coordinates (x _R , y _R ) are set to the origin. Second relative position (t _Rx , t _Ry , 0), depth distance Z _R , unit normal vector (n _Rx , n _Ry , n _Rz ) and second image coordinates (x _R ′, y _R) It is transformed with the primary transformation matrix based on ') and matched with the block of the reference picture.

FIG. 10 is an explanatory diagram of the second block matching.

According to FIG. 10 , the second block including an arbitrary point is selected for the picture to be encoded (second image). Next, the depth distance Z of the arbitrary point and the unit normal vector ( _nx , _ny , _nz ) at the arbitrary point on the surface of the subject are arbitrarily determined. Then, the reference picture (first image) is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R , and the first image coordinates (x _R , y _R ) as the origin, second relative position (t _Rx , t _Ry , 0), depth distance Z _R , unit normal vector (n _Rx , n _Ry , n _Rz ) and second image coordinates (x _R The first transformation matrix based on ', y _R ') is transformed to match with the second block of the picture to be encoded.

  Hereinafter, a process of deriving a primary transformation matrix that is a movement or deformation of a block or image will be described in detail.

ｆk_u、ｆk_v：焦点距離（カメラの距離のピクセル数）
θ＝π／２：座標軸の角度
ｕ_０、ｖ_０：画像中心 First, an internal parameter (internal camera matrix) A of the camera is expressed by the following equation. This value is assumed to be known.

fk _u , fk _v : focal length (number of pixels of camera distance)
θ = π / 2: Coordinate axis angle u ₀ , v ₀ : Image center

The camera coordinate system of the camera C ′ is obtained by moving the camera coordinate system of the camera C in parallel (relative position t). t is one of the external parameters of the camera and represents the displacement vector of the camera C ′ with respect to the camera coordinate system of the camera C. Cameras C and C ′ are arranged with their optical axes in parallel. The relative position vector t of the camera C ′ with respect to the camera coordinate system of the camera C always has a non-zero z-axis component (t _Z ≠ 0).

The following relationship is established between the camera coordinate system coordinates of the camera C and the camera coordinate system coordinates of the camera C ′. That is, the camera coordinate system coordinates of the camera C are converted to the camera coordinate system coordinates of the camera C ′ through the external parameters of the camera.
M = M ′ + t Formula (2)
M: Camera coordinate system coordinates of the arbitrary point P of the subject in the camera C M ′: Camera coordinate system coordinates of the arbitrary point P of the subject in the camera C ′

ｄ：カメラＣ’のカメラ座標系の原点から平面までの距離 Here, with respect to the approximate plane π of the subject, the following expression is established in the camera coordinate system of the camera C ′.
n ^T M ′ = d Formula (5)
n (| n | = 1): normal vector of the plane in the camera coordinate system of the camera C ′

d: Distance from the origin of the camera coordinate system of camera C ′ to the plane

Next, the digital image coordinates of the arbitrary point P are as follows.
m: digital image coordinates of an arbitrary point P in the camera C m ′: digital image coordinates of the arbitrary point P in the camera C ′

Further, the normalized image coordinates of the arbitrary point P are as follows. A “normalized image” refers to an image when it is assumed that the image plane is taken by a camera (normalized camera) whose unit length is from the focal point. The coordinates of an arbitrary point in the normalized image are referred to as “normalized image coordinates”.
x: Normalized image coordinates of the arbitrary point P in the camera C x ′: Normalized image coordinates of the arbitrary point P in the camera C ′

As a result, the following central projective transformation is established.

Next, when Expression (7) and Expression (3) are substituted into Expression (9), the following expression is derived.

  Comparing the third row on both sides leads to s = Z.

Substituting this into equation (9) yields the following equation:

On the other hand, when the formula (2) is substituted into the formula (10), the following formula is established.

By substituting Equation (1), Equation (3), Equation (4), and Equation (8), the following equation is derived.

When comparing the third row on both sides, s ′ = Z ′ = Z− _tz is derived.

Substituting this into equation (10) yields the following equation:

Next, it is assumed that the camera coordinate system of the camera C is rotationally transformed and t is matched with the direction of a vector t _xy obtained by orthogonally projecting it onto the XY plane. However, t _xy is as follows.

In general, the matrix R representing the rotation of the angle θ with the unit vector r as the rotation axis is expressed as follows according to the Rodrigues formula.

Here, it is as follows.

Since r is orthogonal to t and t _xy , the following equation is obtained.

When the rotation angle is θ, the following formula is obtained.

As described above, when Expression (14) and Expression (15) are substituted into Expression (13), the rotation transformation matrix to be obtained is as follows.

Similarly, the inverse transformation matrix is as follows.

Using the rotation transformation matrix R, the translation vector t and the unit normal vector n of the approximate plane π are rotationally transformed. When the results of rotational transformation are respectively denoted as t _R and n _R , the results are as follows.

The following relationship is established between the camera coordinate system coordinates of the camera C after rotation conversion and the camera coordinate system coordinates of the camera C ′ after rotation conversion. That is, the camera coordinate system coordinates of the camera C are converted to the camera coordinate system coordinates of the camera C ′ through the external parameters of the camera.
M _R = M _R '+ t _R formula (20)
M _R : Camera coordinate system coordinates of the arbitrary point P of the subject in the camera C after rotation conversion M _R ′: Camera coordinate system coordinates of the arbitrary point P of the subject in the camera C ′ after rotation conversion

Here, in the camera coordinate system of the camera C ′ after rotation conversion with respect to the approximate plane π of the subject, the value of the distance d between the approximate plane π and the camera C ′ is not changed by rotation conversion. Holds.
n _R ^T M _R '= d Equation (23)
^{_{∴n R T M R '/ d}} = 1 Equation (24)

Substituting equation (24) into equation (20) yields the following planar projective transformation equation.

If H is a homography matrix, the result is as follows.

Next, the digital image coordinates of the arbitrary point P are as follows.
m _R : digital image coordinates of the arbitrary point P in the camera C after the rotation conversion m _R ': digital image coordinates of the arbitrary point P in the camera C' after the rotation conversion

Further, the normalized image coordinates of the arbitrary point P are as follows.
x _R : normalized image coordinates of the arbitrary point P after rotation conversion in the camera C x _R ′: normalized image coordinates of the arbitrary point P after rotation conversion in the camera C ′

Thus, the following expression is established as the central projective transformation.

By substituting Equation (26) and Equation (21) into Equation (28), the following equation is derived.

Comparing the third row on both sides leads to s _R = Z _R.

Substituting this into equation (28) yields the following equation:

On the other hand, when the formula (20) is substituted into the formula (29), the following formula is established.

Substituting Equation (21), Equation (22), Equation (1), Equation (18), and Equation (27), the following equation is derived.

Comparing the third row on both sides,
s _R '= Z _R ' = Z _R formula (41)
Is guided.

Substituting this into equation (29) gives the following equation:

Therefore, when substituting into the equation (23), the following equation is established.

Substituting Equation (19) and Equation (27), the following equation is established.
_{d = Z R (n Rx x} R '+ n Ry y R' + n Rz) formula (31)

Substituting equation (31) into equation (13) yields the following equation:

The conversion formula for x and _{x R,} and obtains a conversion equation x 'and _{x R'.} For simplicity, the definition is as follows.

Then, from the equation (11), the following is obtained.

Comparing the third row on both sides gives the following formula.

As a result, the following expression is established.

Further, by substituting equation (41) into equation (32) becomes a following formula, Z, t, x ', y' can _{Z R} is derived from.

Here, in the formula (40), _'in place of the normalized image coordinates x _R and x _R' camera coordinate system coordinates M _R and M _R With, the conversion equation is obtained.

Comparing the third row on both sides leads to s _R = 1.

Substituting this into equation (35) gives the following equation:

Further, when the digital image coordinates m _R and m _R ′ are used, the following conversion formula is obtained.

When the disparity vector in the normalized image coordinate system is D, the following equation is established.

When the difference between the right side and the left side is obtained, the following equation is established.

For the second block of the rotation-converted second image, the following D translation is performed in the rotation-converted first image, and the first image coordinates (x _R , y _R ) are used as the origin. A block of the first image that is most similar to the second block is searched by transforming with the following L linear transformation matrix. According to the primary transformation matrix, block matching can be performed by deforming a block based on the parallax vector and the inclination of the surface of the subject (approximate to a plane).

  Here, a method of compressing and encoding the reference picture and the encoding target picture will be described.

First, the picture to be encoded is subjected to intra prediction encoding. Next, the reference picture is divided into a plurality of blocks, and each block is deformed with the first image coordinates (x, y) as the origin, based on the translation D and the primary transformation matrix L described above. Then, a block of the reference picture (reference block) that minimizes the prediction error is obtained, and the translation vector D and the primary transformation matrix L (that is, the depth distance Z and the unit normal vector ( _nx , _ny , n) at that time are obtained. _z )) is derived. Here, in order to compare the block with the modified reference block and calculate a prediction error, the modified reference block is interpolated by linear interpolation or the like, and the corresponding pixel value for comparison with each pixel of the block Is calculated. For example, when linear interpolation is used, the pixel value at the point X in FIG. X is obtained from the following equation using the pixel values at the four surrounding points (A, B, C, D).
X = (1-dx) (1-dy) A + dx (1-dy) B + (1-dy) dyC + dxdyD

Finally, the prediction error is compression-encoded, and the depth distance Z (or translation vector D) and the unit normal vector (n _x , n _y , n _z ) are encoded.

  However, if the compression efficiency is higher in the second image than in the case of encoding the prediction error with the reference block of the first image in the second image, In-screen predictive coding.

  Hereinafter, specific examples will be described.

  First, an example in which block matching is performed on images of two cameras that are not on the same plane will be described.

It is assumed that the camera internal parameter A and the relative position vector t are obtained in advance. For simplicity, the definition is as follows.

  This relative position vector t corresponds to a situation in which two cameras are installed on the circumference of radius R by an angle θ.

First, attention is focused on the point M ₀ in the image of the camera C. Digital image coordinate m ₀ of the point M ₀ is the following was value.

Therefore, it is converted into normalized image coordinates. The normalized image coordinate x ₀ of the point M ₀ has the following value.

  Similarly, the image of the camera C and the image of the camera C ′ are converted from the digital image coordinates to the normalized image coordinates.

Next, the image of the camera C and the image of the camera C ′ are rotationally converted. The rotation transformation matrix is as follows based on Equation (16).

The relative position vector is converted into the following vector based on Expression (18).

Based on the equation (33), rotation conversion is performed. For example, the point M ₀ is converted into the following point M _R0 .

It is assumed that the search is performed with a block of size W × H having the point M ₀ as the upper left corner.

According to the present invention, block matching is performed by arbitrarily changing the normal vector (n _x , n _y , n _z ) and the depth distance Z. For example, when n _x = 0, n _y = 1, n _z = 0, and Z = R / 2, the depth distance and the normal vector in the rotation-transformed coordinate system are expressed by equations (32) and (19). Is used as follows.

Accordingly, the parallax vector D ₀ at this time is as follows from the equation (37).

Further, the primary transformation matrix L ₀ at this time is as follows from the equation (38).

At this time, if the error of the block matching is the minimum, D ₀ and L ₀ are the target disparity vector and the primary transformation matrix.

  Second, an example in which block matching is performed on images of K cameras on the same plane will be described.

The internal parameter A of the camera and the relative position vector t _k (representing the relative position vector of the kth camera with respect to the 0th camera, _where the 0th camera is the base camera, k = 1 to K−1) are obtained in advance. It is a premise. For simplicity, suppose that it is as follows.

This relative position vector t _k corresponds to a situation in which K cameras are are installed at Δ intervals on the X axis.

First, focus on the point M ₀ in the image of the 0th camera. Digital image coordinate m ₀ of the point M ₀ is the following was value.

Therefore, it is converted into normalized image coordinates. The normalized image coordinate x ₀ of the point M ₀ has the following value.

  Similarly, the images of the other K-1 cameras are converted from digital image coordinates to normalized image coordinates.

According to the present invention, block matching is performed by arbitrarily changing the normal vector (n _x , n _y , n _z ) and the depth distance Z. For _example, when _{n x = n x0, n y} = n y0, n z = n z0, Z = Z 0, disparity vectors of the corresponding point of the k camera images for a point _{M 0} in the zeroth camera images D _k is as follows.

Further, the primary transformation matrix L _k of the block in the image of the kth camera with respect to the block in the image of the 0th camera is as follows.

Here, in the normal vector _{(n x, n y, n} z) and depth distance Z remains fixed _{(n x = n x0, n} y = n y0, n z = n z0, Z = Z 0), In each camera, the blocks that are translated based on D _k and linearly converted based on L _k are compared, and the difference between the two (corresponding blocks in the 0th camera image and the kth camera image) is evaluated. To do. For example, the least square error is obtained.

The corresponding evaluation value of the difference between blocks in the zeroth camera image and the k camera image is denoted by d _k. At this time, the calculation is performed as follows for each combination of the normal vector and the depth distance.

  The combination of the normal vector and the depth distance that minimizes S is the target disparity vector and the primary transformation matrix.

  Third, an embodiment in which block matching is performed on images of K cameras not on the same plane will be described.

The internal parameter A of the camera and the relative position vector t _k (representing the relative position vector of the kth camera with respect to the 0th camera, _where the 0th camera is the base camera, k = 1 to K−1) are obtained in advance. It is a premise. For simplicity, suppose that it is as follows.

This relative position vector t _k corresponds to a situation in which K cameras are are installed at an angle Θ intervals on the circumference of radius R.

First, focus on the point M ₀ in the image of the 0th camera. Digital image coordinate m ₀ of the point M ₀ is the following was value.

Therefore, it is converted into normalized image coordinates. The normalized image coordinate x ₀ of the point M ₀ has the following value.

  Similarly, the images of the other K-1 cameras are converted from digital image coordinates to normalized image coordinates.

Next, the K camera images are rotationally converted. The rotation transformation matrix in the combination of the 0th camera and the kth camera is as follows based on Equation (16).

The relative position vector in the combination of the 0th camera and the kth camera is converted into the following vector based on Expression (18).

Based on the equation (33), rotation conversion is performed. For example, the point _{M 0} is converted to a point _{M k} below.

And performing a search block of a point M ₀ and the upper left corner.

According to the present invention, block matching is performed by arbitrarily changing the normal vector (n _x , n _y , n _z ) and the depth distance Z. For example, when n _x = 0, n _y = 1, n _z = 0, and Z = R / 2, the depth distance and normal vector in the coordinate system rotationally transformed by the combination of the 0th camera and the kth camera are Using Equation (32) and Equation (19), the following is obtained.

Accordingly, the disparity vector D _k in the combination of the 0th camera and the kth camera at this time is as follows from the equation (37).

Further, the primary transformation matrix L _k in the combination of the 0th camera and the kth camera at this time is as follows from the equation (38).

Here, the normal vector _{(n x, n y, n} z) in and remain depth distance Z was fixed _{(n x = 0, n y} = 1, n z = 0, Z = R / 2), the cameras , The blocks that have been translated based on D _k and linearly transformed based on L _k are compared, and the difference between the two (corresponding blocks in the image of the 0th camera and the image of the kth camera) is evaluated. For example, the least square error is obtained.

The corresponding evaluation value of the difference between blocks in the zeroth camera image and the k camera image is denoted by d _k. At this time, the calculation is performed as follows for each combination of the normal vector and the depth distance.

  The combination of the normal vector and the depth distance that minimizes S is the target disparity vector and the primary transformation matrix.

At this time, if the error of the block matching is the minimum, D ₀ and L ₀ are the target disparity vector and the primary transformation matrix.

  FIG. 11 is an explanatory diagram of modified block matching by two cameras.

  According to FIG. 11, the deformed block matching is independently performed for each image by a one-to-one combination. In this case, a different depth distance and normal vector are required for each combination, and a large amount of calculation is required.

  FIG. 12 is an explanatory diagram of modified block matching using three or more cameras.

According to FIG. 12, the prediction error between one base camera block and another camera block is calculated, and the depth distance Z and the unit normal vector ( _nx , _ny , _nz ) is derived. With a set of depth distances and normal vectors, matching of the corresponding blocks of all images can be achieved.

  FIG. 13 is a flowchart in the present invention. FIG. 13 will be described based on the matching of FIG.

(S901) The position information of all cameras is acquired. All the cameras are arranged on the same plane parallel to the optical axis and perpendicular to the optical axis. Accordingly, the first relative position (t _x , t _y , t _z ) of the world coordinate system of the second camera with respect to the first camera is acquired. The camera may be movable. When the camera itself has a positioning function such as GPS, the position information is received. If the camera is fixed, the position information may be registered in advance.

(S902) All the camera images are acquired. Here, when the optical axes of the first camera and / or the second camera are not parallel to the Z axis, the optical axes of the first camera and the second camera are parallel to the Z axis. Using the parameters, the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera are transformed.

(S903) Using the camera internal parameters, the first camera and the second camera so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. The same rotational transformation is applied to the second camera to transform the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera. The matrix representing the rotation transformation is as follows.

(S904) An arbitrary point on the subject is determined.
(S905) First image coordinates (x, y) obtained by projecting an arbitrary point to the encoding target picture to which rotation transformation has been applied, and a second image obtained by projecting the arbitrary point to a reference picture to which rotation transformation has been applied. The coordinates (x ′, y ′) are derived. Here, the image coordinates of an arbitrary point are normalized image coordinates.
(S906) A first block including an arbitrary point is selected for the current picture.

(S907) Hereinafter, S907 to S910 are repeated.
(S908) The depth distance Z of an arbitrary point is arbitrarily determined.
(S909) unit normal vector at any point of the surface of the object _{(n x, n y, n} z) arbitrarily determined.

一次変換行列は、以下のものである。

(S910) Here, as a result of the rotation transformation, a new depth distance Z _R and a new unit normal vector (n _Rx , n _Ry , n _Rz ) and a new relative position (t _Rx , t of an arbitrary viewpoint) _Ry, 0) and the new image coordinates _{(x R} ', _{y R'} and) was obtained. Then, the disparity vector is as follows.

The primary transformation matrix is:

The first block is moved with a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R , and the first image coordinates (x _R , y _R ) are used as the origin, Second relative position (t _Rx , t _Ry , 0), depth distance Z _R , unit normal vector (n _Rx , n _Ry , n _Rz ) and second image coordinates (x _R ′, y _R ′) Is transformed with a primary transformation matrix based on the above and matched with a block of a reference picture.

(S911) depth distance Z and unit normal vector _{(n x, n y, n} z) while arbitrarily changed, repeated S906～S909. Then, the second block of the second image that is most similar to the first block of the first image that has been moved and deformed is searched for. As a result, the depth distance Z and the unit normal vector ( _nx , _ny , _nz ) are derived.

(S912) Deriving and encoding a prediction error between the moved and transformed first block of the current picture to be encoded and the second block of the reference picture that is most similar. The encoding method of the prediction error is a general one defined in MPEG.
(S913) depth distance Z and unit normal vector _{(n x, n y, n} z) and adding it to the encoded data.

(S914) An arbitrary point on the subject is further determined for a portion other than the first block of the picture to be encoded, and S904 to S913 are repeated.

  FIG. 14 is a functional configuration diagram of the multi-viewpoint image compression coding apparatus according to the present invention.

  According to FIG. 14, the multi-viewpoint image compression coding apparatus 2 includes a position information acquisition unit 21, an image acquisition unit 22, a camera coordinate system coordinate conversion unit 203, an arbitrary point selection unit 204, and a normalized image coordinate derivation. A unit 205, a block selection unit 206, a block matching unit 207, a parameter determination unit 208, a matching control unit 209, a block control unit 210, a prediction error encoding unit 211, and a parameter addition unit 212. These functional units can also be realized by a program executed by a computer.

  The position information acquisition unit 21 acquires position information of all cameras. It has the same function as S901 in FIG.

  The image acquisition unit 22 acquires synchronized images from all cameras. It has the same function as S902 in FIG. 13 described above.

  The camera coordinate system coordinate conversion unit 203 uses the camera internal parameters so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. The same rotation transformation is applied to the first camera and the second camera to transform the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera. It has the same function as S903 in FIG.

  The arbitrary point selection unit 204 determines an arbitrary point on the subject. The arbitrary point selection unit 204 further determines an arbitrary point on the subject for a portion other than the first block of the first image in response to an instruction from the block control unit 210. It has the same function as S904 in FIG. 13 described above.

  The normalized image coordinate deriving unit 205 projects a first normalized image coordinate (x, y) obtained by projecting an arbitrary point onto the first image subjected to rotation transformation, and a second obtained by performing rotation transformation on the arbitrary point. And the second normalized image coordinates (x ′, y ′) projected onto the image. It has the same function as S905 in FIG.

  The block selection unit 206 selects a first block including an arbitrary point for the first image. It has the same function as S906 in FIG. 13 described above.

The block matching unit 207 moves the first block by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R , and the first image coordinates (x _R , y _R ) as the origin, the second relative position (t _Rx , t _Ry , 0), the depth distance Z _R , the unit normal vector (n _Rx , n _Ry , n _Rz ) and the second image coordinates (x _R It is transformed with a primary transformation matrix based on ', y _R ') and matched with the block of the second image. This is the same as S910 in FIG.

When the block selection unit 206 selects a second block including an arbitrary point with respect to the second image, the block matching unit 207 moves the first image by a parallax vector, and the first image coordinates ( x _{R 1} , y _R ) is used as an origin, and is transformed with a primary transformation matrix to be matched with the second block of the second image.

The parameter determining unit 208 arbitrarily determines the depth distance Z of an arbitrary point and the unit normal vector ( _nx , _ny , _nz ) at the arbitrary point on the surface of the subject. This is the same as S908 and S909 of FIG.

The matching control unit 209 repeatedly controls the parameter determination unit and the block matching unit while arbitrarily changing the depth distance Z and the unit normal vector (n _x , n _y , n _z ), and moves and deforms the first unit. The second block of the second image that is most similar to the first block of the one image is searched, and the depth distance Z and the unit normal vector ( _nx , _ny , _nz ) are derived. This is the same as S907 and S911 of FIG.

  The block control unit 210 includes a normalized image coordinate calculation unit 205, a parameter determination unit 208, a block matching unit 207, a matching control unit 209, a prediction error encoding unit 211, and a parameter addition unit 212 for arbitrary points. repeat. Finally, all parts of the first image and the second image are compression encoded. This is the same as S904 and S914 in FIG.

  The prediction error encoding unit 211 derives and encodes a prediction error between the moved and deformed first block of the first image and the second block of the second image that is most similar. This is the same as S912 in FIG.

The parameter adding unit 212 adds the derived depth distance Z and the unit normal vector ( _nx , _ny , _nz ) to the encoded data. This is the same as S913 in FIG.

  As described above in detail, according to the multi-view image compression encoding method, apparatus, and program of the present invention, even when the surface of the subject is inclined with respect to the optical axes of a plurality of cameras, A parallax vector can be detected by block matching, and a multi-viewpoint image can be compression-encoded using the parallax vector. According to MPEG, block matching is performed while moving and deforming the first block of the picture to be encoded or the reference picture itself based on an arbitrary depth distance Z and an arbitrary direction n of the normal of the subject surface. By doing so, a primary transformation matrix is also detected with a parallax vector. By deriving the prediction error between the block or image deformed based on the primary transformation matrix with the first image coordinates as the origin and the other block, the data amount of the prediction error is reduced, and the multi-viewpoint image is highly efficient. Can be compression-encoded.

  According to the various embodiments of the present invention described above, those skilled in the art can easily make various changes, modifications and omissions within the scope of the technical idea and the viewpoint of the present invention. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

It is a system configuration diagram in the present invention. FIG. 2 is a layout view of a plurality of cameras viewed from the front and side in the system configuration of FIG. 1. FIG. 3 is a screen diagram showing how a subject is viewed from each camera in FIG. 2. This is an image taken from each camera when the surface of the subject is tilted in the horizontal direction. This is an image taken from each camera when the surface of the subject is tilted in the vertical direction. It is an image taken from each camera when the surface of the subject is tilted obliquely in the horizontal direction and tilted in the vertical direction. It is explanatory drawing of the 1st relative positional relationship in two cameras. It is explanatory drawing of the 2nd relative positional relationship which added rotation to two cameras. It is explanatory drawing of the 1st block matching. It is explanatory drawing of the 2nd block matching. It is explanatory drawing of the deformation | transformation block matching by two cameras. It is explanatory drawing of the deformation | transformation block matching by three or more cameras. It is a flowchart in this invention. It is a functional block diagram of the multiview image compression coding apparatus in this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera 2 Multiview image compression coding apparatus 201 Position information acquisition part 202 Image acquisition part 203 Camera coordinate system coordinate transformation part 204 Arbitrary point selection part 205 Normalized image coordinate derivation part 206 Block selection part 207 Block matching part 208 Parameter determination part 209 Matching control unit 210 Block control unit 211 Prediction error encoding unit 212 Parameter addition unit 3 Subject

Claims (18)

A first camera and a second camera having an optical axis parallel to the Z axis are arranged at a first relative position (t _x , t _y , t _z ) (where t _z ≠ 0); A multi-viewpoint image compression encoding method using a first image of a first camera and a second image of a second camera that photograph a subject,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. A first step of applying the same rotational transformation to the camera to transform the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera;
A second step of determining an arbitrary point on the subject;
First image coordinates (x _R , y _R ) obtained by projecting the arbitrary point onto the first image of the first camera subjected to the rotation transformation, and a second image obtained by applying the rotation transformation to the arbitrary point. A third step of deriving second image coordinates (x _R ′, y _R ′) projected onto the second image of the camera of
A fourth step of selecting a first block including an arbitrary point for the first image;
A depth distance Z of an arbitrary point and a unit normal vector ( _nx , _ny , _nz ) at an arbitrary point on the surface of the subject are arbitrarily determined, and the depth distance Z _{R with} respect to the same plane subjected to the rotation transformation And a fifth step of deriving a unit normal vector (n _Rx , n _Ry , n _Rz ),
The first block is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotational transformation has been applied, and the first image coordinates are used as the origin and 2 relative positions (t _Rx , t _Ry , 0), depth distance Z _R , unit normal vector (n _Rx , n _Ry , n _Rz ) and second image coordinates (x _R ′, y _R ′). A sixth step of transforming with a primary transformation matrix based to match a block of the second image;
The first block of the first image in which the depth distance Z and the unit normal vector ( _nx , _ny , _nz ) are arbitrarily changed, and the fifth and sixth steps are repeated, moved, and deformed. Searching for a second block of a second image most similar to, and deriving a depth distance Z and unit normal vectors ( _nx , _ny , _nz );
An eighth step of deriving and encoding a prediction error between the moved and deformed first block of the first image and the second block of the second image that is most similar;
And a ninth step of adding the derived depth distance Z and the unit normal vector (n _x , n _y , n _z ) to the encoded data. The eighth step uses the MPEG (Moving Picture Experts Group) prediction error encoding method,
The multi-view image compression encoding method according to claim 1, wherein the first image is a picture to be encoded, and the second image is a reference picture. A first camera and a second camera having an optical axis parallel to the Z axis are arranged at a first relative position (t _x , t _y , t _z ) (where t _z ≠ 0); A multi-viewpoint image compression encoding method using a first image of a first camera and a second image of a second camera that photograph a subject,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. A first step of applying the same rotational transformation to the camera to transform the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera;
A second step of determining an arbitrary point on the subject;
First image coordinates (x _R , y _R ) obtained by projecting the arbitrary point onto the first image of the first camera subjected to the rotation transformation, and a second image obtained by applying the rotation transformation to the arbitrary point. A third step of deriving second image coordinates (x _R ′, y _R ′) projected onto the second image of the camera of
A fourth step of selecting a second block containing an arbitrary point for the second image;
A depth distance Z of an arbitrary point and a unit normal vector ( _nx , _ny , _nz ) at an arbitrary point on the surface of the subject are arbitrarily determined, and the depth distance Z _{R with} respect to the same plane subjected to the rotation transformation And a fifth step of deriving a unit normal vector (n _Rx , n _Ry , n _Rz ),
The first image is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotational transformation has been applied, and the first image coordinates (x _R , y _R ) as the origin, second relative position (t _Rx , t _Ry , 0), depth distance Z _R , unit normal vector (n _Rx , n _Ry , n _Rz ) and second image coordinates (x _R A sixth step of transforming with a linear transformation matrix based on ', y _R ') to match the second block of the second image;
Depth distance Z and unit normal vector _{(n x, n y, n} z) while arbitrarily changed, repeating the fifth and sixth step of, first it is most similar to the second block of the second image 1 Searching for the first block of the image and deriving a depth distance Z and a unit normal vector ( _nx , _ny , _nz );
An eighth step of derived and coding prediction error between the first block of the first image that is most similar to the second block of the second image,
And a ninth step of adding the derived depth distance Z and the unit normal vector (n _x , n _y , n _z ) to the encoded data. The eighth step uses the MPEG (Moving Picture Experts Group) prediction error encoding method,
The multi-view image compression encoding method according to claim 3, wherein the first image is a reference picture, and the second image is an encoding target picture. While repeating the second to ninth steps ,
In the iteration, the second step further determines arbitrary points on the subject for parts other than the block previously selected by the fourth step ,
The multi-viewpoint image compression coding method according to any one of claims 1 to 4, wherein all the portions of the first image and the second image are finally compression-coded. For the first step, the matrix representing the rotational transformation is:

For the second step, the image coordinates of the arbitrary point are normalized image coordinates,
As a result of the rotation transformation, a new depth distance Z _R and a new unit normal vector (n _Rx , n _Ry , n _Rz ), a new relative position (t _Rx , t _Ry , 0) of an arbitrary viewpoint, and Suppose that new image coordinates (x _R ', y _R ') are obtained,
For the sixth step, the disparity vector is:

The primary transformation matrix is

The multi-viewpoint image compression encoding method according to any one of claims 1 to 5, wherein When the optical axis of the first camera and / or the second camera is not parallel to the Z axis,
The camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera are converted using camera internal parameters so that the optical axes of the first camera and the second camera are parallel to the Z axis. The multi-view image compression encoding method according to claim 1, further comprising a step of: Even when three or more cameras are arranged, the depth distance Z and the unit normal vector ( _nx , _ny , _nz ) are constant,
In a seventh step, a prediction error between one base camera block and another camera block is calculated, and a depth distance Z and a unit normal vector (n _x , n _y , _nz ) is derived, The multi-view image compression encoding method according to any one of claims 1 to 7. A first camera and a second camera having an optical axis parallel to the Z axis are arranged at a first relative position (t _x , t _y , t _z ) (where t _z ≠ 0); A multi-viewpoint image compression encoding apparatus using a first image of a first camera and a second image of a second camera that photograph a subject,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. Camera coordinate system coordinate conversion means for converting the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera by applying the same rotational transformation to the camera;
Arbitrary point selection means for determining an arbitrary point on the subject;
First image coordinates (x _R , y _R ) obtained by projecting the arbitrary point onto the first image of the first camera subjected to the rotation transformation, and a second image obtained by applying the rotation transformation to the arbitrary point. Image coordinate calculation means for deriving second image coordinates (x _R ′, y _R ′) projected onto the second image of the camera of
Block selection means for selecting a first block including an arbitrary point for the first image;
A depth distance Z of an arbitrary point and a unit normal vector ( _nx , _ny , _nz ) at an arbitrary point on the surface of the subject are arbitrarily determined, and the depth distance Z _{R with} respect to the same plane subjected to the rotation transformation And parameter determining means for deriving unit normal vectors (n _Rx , n _Ry , n _Rz ),
The first block is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotational transformation is applied, and the first image coordinates are used as the origin and 2 relative positions (t _Rx , t _Ry , 0), depth distance Z _R , unit normal vector (n _Rx , n _Ry , n _Rz ) and second image coordinates (x _R ′, y _R ′). A block matching means for deforming with a primary transformation matrix based on and matching with a block of the second image;
While arbitrarily changing the depth distance Z and the unit normal vector (n _x , n _y , n _z ), the parameter determination means and the block matching means are repeatedly controlled to move and deform the first of the first image Matching control means for searching for a second block of the second image most similar to the block and deriving a depth distance Z and a unit normal vector ( _nx , _ny , _nz );
Prediction error encoding means for deriving and encoding a prediction error between the moved and deformed first block of the first image and the second block of the second image most similar to the first block;
A multi-viewpoint image compression coding apparatus comprising: parameter addition means for adding the derived depth distance Z and unit normal vector ( _nx , _ny , _nz ) to encoded data. The prediction error encoding means uses a prediction error encoding method of MPEG (Moving Picture Experts Group),
The multi-viewpoint image compression encoding apparatus according to claim 9, wherein the first image is a picture to be encoded, and the second image is a reference picture. A first camera and a second camera having an optical axis parallel to the Z axis are arranged at a first relative position (t _x , t _y , t _z ) (where t _z ≠ 0); A multi-viewpoint image compression encoding apparatus using a first image of a first camera and a second image of a second camera that photograph a subject,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. Camera coordinate system coordinate conversion means for converting the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera by applying the same rotational transformation to the camera;
Arbitrary point selection means for determining an arbitrary point on the subject;
First image coordinates (x _R , y _R ) obtained by projecting the arbitrary point onto the first image of the first camera subjected to the rotation transformation, and a second image obtained by applying the rotation transformation to the arbitrary point. Image coordinate calculation means for deriving second image coordinates (x _R ′, y _R ′) projected onto the second image of the camera of
Block selecting means for selecting a second block including an arbitrary point for the second image;
A depth distance Z of an arbitrary point and a unit normal vector ( _nx , _ny , _nz ) at an arbitrary point on the surface of the subject are arbitrarily determined, and the depth distance Z _{R with} respect to the same plane subjected to the rotation transformation And parameter determining means for deriving unit normal vectors (n _Rx , n _Ry , n _Rz ),
The first image is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotational transformation has been applied, and the first image coordinates (x _R , y _R ) as the origin, second relative position (t _Rx , t _Ry , 0), depth distance Z _R , unit normal vector (n _Rx , n _Ry , n _Rz ) and second image coordinates (x _R Block matching means for transforming with a primary transformation matrix based on ', y _R ') to match the second block of the second image;
Depth distance Z and unit normal vector _{(n x, n y, n} z) while changing the optionally repeatedly control parameter determining means and the block matching means, most similar to the second block of the second image and matching control means for deriving a first block of the first image and search, depth distance Z and unit normal vector _{(n x, n y, n} z) and,
And prediction error encoding means for deriving and and encodes the prediction error between the first block of the first image that is most similar to the second block of the second image,
A multi-viewpoint image compression coding apparatus comprising: parameter addition means for adding the derived depth distance Z and unit normal vector ( _nx , _ny , _nz ) to encoded data. The prediction error encoding means uses the MPEG (Moving Picture Experts Group) prediction error encoding method,
12. The multi-viewpoint image compression coding apparatus according to claim 11, wherein the first image is a reference picture, and the second image is an encoding target picture. And the arbitrary point selecting means, and the previous SL image coordinate calculation unit, and said block selecting means, and parameter determining means, a block matching unit, and a matching control unit, and the prediction error encoding unit, the processing of a parameter adding means further it has a block control means for repeating,
In the repetition, the arbitrary point selecting means further determines an arbitrary point on the subject for a part other than the block previously selected by the block selecting means ,
The multi-viewpoint image compression coding apparatus according to any one of claims 9 to 12, wherein all the portions of the first image and the second image are finally compression-coded. For the camera coordinate system coordinate transformation means, the matrix representing the rotation transformation is as follows:

The image coordinate calculation means derives normalized image coordinates as the image coordinates of the arbitrary point,
As a result of the rotation transformation, a new depth distance Z _R and a new unit normal vector (n _Rx , n _Ry , n _Rz ), a new relative position (t _Rx , t _Ry , 0) of an arbitrary viewpoint, and Suppose that new image coordinates (x _R ', y _R ') are obtained,
The block matching means sets the disparity vector as follows:

The primary transformation matrix is

The multi-viewpoint image compression coding apparatus according to any one of claims 9 to 13, When the optical axis of the first camera and / or the second camera is not parallel to the Z axis,
The camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera are converted using camera internal parameters so that the optical axes of the first camera and the second camera are parallel to the Z axis. The multi-view image compression encoding apparatus according to claim 9, wherein the multi-view image compression encoding apparatus is used. Even when three or more cameras are arranged, the depth distance Z and the unit normal vector ( _nx , _ny , _nz ) are constant,
In a seventh step, a prediction error between one base camera block and another camera block is calculated, and a depth distance Z and a unit normal vector (n _x , n _y , The multi-view image compression encoding apparatus according to claim 9, wherein n _z ) is derived. A first camera and a second camera having an optical axis parallel to the Z axis are arranged at a first relative position (t _x , t _y , t _z ) (where t _z ≠ 0); A multi-viewpoint image compression encoding program for causing a computer to function to compress and encode a multi-viewpoint image using the first image of the first camera and the second image of the second camera that photographed the subject. There,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. Camera coordinate system coordinate conversion means for converting the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera by applying the same rotational transformation to the camera;
Arbitrary point selection means for determining an arbitrary point on the subject;
First image coordinates (x _R , y _R ) obtained by projecting the arbitrary point onto the first image of the first camera subjected to the rotation transformation, and a second image obtained by applying the rotation transformation to the arbitrary point. Image coordinate calculation means for deriving second image coordinates (x _R ′, y _R ′) projected onto the second image of the camera of
Block selection means for selecting a first block including an arbitrary point for the first image;
A depth distance Z of an arbitrary point and a unit normal vector ( _nx , _ny , _nz ) at an arbitrary point on the surface of the subject are arbitrarily determined, and the depth distance Z _{R with} respect to the same plane subjected to the rotation transformation And parameter determining means for deriving unit normal vectors (n _Rx , n _Ry , n _Rz ),
The first block is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotational transformation is applied, and the first image coordinates are used as the origin and 2 relative positions (t _Rx , t _Ry , 0), depth distance Z _R , unit normal vector (n _Rx , n _Ry , n _Rz ) and second image coordinates (x _R ′, y _R ′). A block matching means for deforming with a primary transformation matrix based on and matching with a block of the second image;
While arbitrarily changing the depth distance Z and the unit normal vector (n _x , n _y , n _z ), the parameter determination means and the block matching means are repeatedly controlled to move and deform the first of the first image Matching control means for searching for a second block of the second image most similar to the block and deriving a depth distance Z and a unit normal vector ( _nx , _ny , _nz );
Prediction error encoding means for deriving and encoding a prediction error between the moved and deformed first block of the first image and the second block of the second image most similar to the first block;
A multi-viewpoint image compression coding program that causes a computer to function as parameter addition means for adding the derived depth distance Z and unit normal vector (n _x , n _y , n _z ) to encoded data. A first camera and a second camera having an optical axis parallel to the Z axis are arranged at a first relative position (t _x , t _y , t _z ) (where t _z ≠ 0); A multi-viewpoint image compression encoding program for causing a computer to function to compress and encode a multi-viewpoint image using the first image of the first camera and the second image of the second camera that photographed the subject. There,
Using the camera internal parameters, the first camera and the second camera are set so that the optical axes of the first camera and the second camera are perpendicular to the same plane passing through the first camera and the second camera. Camera coordinate system coordinate conversion means for converting the camera coordinate system coordinates of the first camera and the camera coordinate system coordinates of the second camera by applying the same rotational transformation to the camera;
Arbitrary point selection means for determining an arbitrary point on the subject;
First image coordinates (x _R , y _R ) obtained by projecting the arbitrary point onto the first image of the first camera subjected to the rotation transformation, and a second image obtained by applying the rotation transformation to the arbitrary point. Image coordinate calculation means for deriving second image coordinates (x _R ′, y _R ′) projected onto the second image of the camera of
Block selecting means for selecting a second block including an arbitrary point for the second image;
A depth distance Z of an arbitrary point and a unit normal vector ( _nx , _ny , _nz ) at an arbitrary point on the surface of the subject are arbitrarily determined, and the depth distance Z _{R with} respect to the same plane subjected to the rotation transformation And parameter determining means for deriving unit normal vectors (n _Rx , n _Ry , n _Rz ),
The first image is moved by a disparity vector based on the second relative position (t _Rx , t _Ry , 0) and the depth distance Z _R to which the rotational transformation has been applied, and the first image coordinates (x _R , y _R ) as the origin, second relative position (t _Rx , t _Ry , 0), depth distance Z _R , unit normal vector (n _Rx , n _Ry , n _Rz ) and second image coordinates (x _R Block matching means for transforming with a primary transformation matrix based on ', y _R ') to match the second block of the second image;
Depth distance Z and unit normal vector _{(n x, n y, n} z) while changing the optionally repeatedly control parameter determining means and the block matching means, most similar to the second block of the second image and matching control means for deriving a first block of the first image and search, depth distance Z and unit normal vector _{(n x, n y, n} z) and,
And prediction error encoding means for deriving and and encodes the prediction error between the first block of the first image that is most similar to the second block of the second image,
A multi-viewpoint image compression coding program that causes a computer to function as parameter addition means for adding the derived depth distance Z and unit normal vector (n _x , n _y , n _z ) to encoded data.

Images