JPH09275578A - Multi-viewpoint encoding and decoding device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the encoding efficiency of a parallax vector and also to reduce the computational complexity of the parallax vector by estimating the compensation of both movement and parallax when a multi-viewpoint image is encoded. SOLUTION: An image input part 5 inputs a multi-viewpoint stereoscopic image after dividing it into parts images having parallax and a background image having no parallax. Then a movement compensation part 6 estimates the movement compensation based on the moved variable between the frames or fields of the parts images. A parallax compensation part 16 estimates the parallax compensation based on the parallax amount caused between the frames or fields of the parts images. A parallax divider 17 is provided for every parts image to define one of parallax vectors obtained at the part 16 as a global parallax vector and to output a local parallax vector that is defined by subtracting the global parallax vector from other parallax vectors excluding the global one.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a high-efficiency encoding and decoding apparatus for multi-view images, which uses motion compensation or parallax compensation prediction in high-efficiency image encoding and decoding.

[0002]

2. Description of the Related Art In multi-viewpoint image coding, motion compensation, which is generally used in moving image coding, and parallax compensation between images with different viewpoints are combined and coded. A method for improving the coding efficiency is well known. For example, according to a document (WA Shupp, Yasuda, “Data compression of stereo video using parallax compensation and motion compensation”, PCSJ88 (1988), pp.63-64), as shown in FIG. Only motion compensation is performed on the right image. For the left image, one of parallax compensation and motion compensation, whichever has a smaller prediction error, is selected and encoded.

An example of obtaining a motion vector by changing the size of a block when performing motion compensation by block matching in coding a moving image is disclosed in Japanese Patent Laid-Open No. 6-11.
It is disclosed in Japanese Patent No. 3283. In this method,
As shown in FIG. 2, motion compensation is first performed using the small block 71, and a motion vector for the small block 71 (hereinafter,
Abbreviated as small block vector). A plurality of these small blocks 71 are collected to form a large block, motion compensation is performed, and a motion vector for the large block (hereinafter abbreviated as large block vector) is obtained.

If the large block 70 is composed of four small blocks, the small block vector exists within a predetermined range with respect to the large block vector, and the number of small block vectors existing therein is a predetermined threshold value. When it exceeds, the amount of information of the motion vector is reduced by transmitting the large block vector and the difference between the small block vector and the large block vector.

[0005]

However, the conventional method is not efficient because the parallax compensation is performed for all the portions having parallax and the portions having no parallax when performing parallax compensation. In addition, the size of the disparity vector obtained is usually larger than the size of the motion vector obtained in motion compensation, and since the size of the disparity vector is encoded as it is, the efficiency of encoding the disparity vector is not good. There is a point.

Further, in this method, since the motion compensation or the parallax compensation is selectively determined, the parallax vector is not always transmitted. Therefore, when a plurality of objects are three-dimensionally displayed, editing such as changing only the depth of some objects cannot be performed.

On the other hand, in the method of encoding a moving image by using two types of motion vectors, a large block and a small block, the amount of calculation becomes enormous because it is necessary to perform a two-step search over the entire screen. In addition, since the large block is composed of a plurality of small blocks, the two blocks with different reliability are
There is no difference in physical meaning between these vectors, only the types of vectors and their differences are obtained. Furthermore, since these two types of motion vectors cannot have different effects, it is difficult to use for component image editing.

An object of the present invention is to encode a disparity vector by dividing the disparity vector into two stages while realizing highly efficient encoding by using motion compensation and disparity compensation prediction in encoding a multi-view image. Provided is a parallax compensation encoding / decoding device capable of improving efficiency, reducing the amount of calculation when obtaining a parallax vector, and easily editing a three-dimensional component image by using a parallax vector divided into two stages. To do.

[0009]

In order to achieve the above-mentioned object, an encoding apparatus of the present invention divides a multi-view stereoscopic image into at least one parallax component image and at least one parallax-free background image. Input means for inputting, motion compensation means for performing motion compensation prediction using a motion amount between frames or fields of a component image, and parallax compensation means for performing parallax compensation prediction using a parallax amount between frames or fields of a component image. And a parallax vector dividing unit for each component image, wherein the parallax vector dividing unit sets one of the parallax vectors obtained by the parallax compensating unit as a global parallax vector. , The remaining parallax vector minus the global parallax vector is output as a local parallax vector.

Further, in the encoding apparatus of the present invention, a multi-view stereoscopic image of three or more viewpoints is divided into at least one component image having parallax and at least one background image having no parallax, and inputting means, and the component image. For each component image, a motion compensation unit that performs motion compensation prediction using the amount of motion between frames or fields and a parallax compensation unit that performs parallax compensation prediction using the amount of parallax between frames or fields of component images are provided. At the same time, the multi-viewpoint encoding device is provided with one disparity vector division unit and at least one difference unit for each component image, wherein the disparity vector division unit is
One of the parallax vectors obtained between the specific viewpoints is a global parallax vector, and a difference obtained by subtracting the global parallax vector from the remaining parallax vectors is output as a local parallax vector, and the differencer is
The difference between all the parallax vectors obtained between other viewpoints and the global parallax vector is output.

Further, in the present invention, the average of all parallax vectors in the three-dimensional component image may be used as the global parallax vector.

In the present invention, the parallax vector first obtained in the three-dimensional component image may be used as the global parallax vector.

In the present invention, the magnitude of the parallax vector in the three-dimensional component image may be made into a histogram as the global parallax vector, and the parallax vector with the highest frequency of occurrence may be used.

In the present invention, the difference between the global parallax vector of the current frame or field and the global parallax vector of the immediately preceding frame or field may be used as the global parallax vector of the current frame or field.

Further, in the present invention, a difference from the local parallax vector obtained immediately before may be used as the local parallax vector.

In the decoding device of the present invention, the multi-viewpoint includes a motion compensating means for performing motion compensation and a parallax compensating means for performing parallax compensation by using the motion amount and the parallax amount between frames or fields of the multi-view stereoscopic image. In the image decoding device, a parallax vector synthesizing unit that outputs the parallax vector to the parallax compensating unit using the input global parallax vector and the local parallax vector, and a parameter input unit that outputs the movement parameter in the depth direction to the image editing unit. And an image synthesizing unit for synthesizing the input background image and the three-dimensional component image by changing the depth of the three-dimensional component image using the input parameters and outputting the image to the image synthesizing unit. To do.

Further, in the present invention, the image editing means outputs the three-dimensional component images in order from the smallest global parallax vector, and the image synthesizing means outputs the three-dimensional component images sequentially input after the background image is input. The images may be combined by overwriting the images.

[0018]

Embodiments of the present invention will be described below in detail.

FIG. 1 shows a first embodiment of the image coding apparatus of the present invention. The present embodiment is an example of a two-lens system, and an image input to this image encoding device is a background image having no parallax and a three-dimensional component image having parallax. Both the background image and the three-dimensional part image may be either a still image or a moving image, but both are assumed to be moving images here.

Here, the three-dimensional component image is, for example, as shown in FIG. 2, a stereo image 2 obtained by arranging two cameras in parallel and inputting it to a clipping device 3 and clipping only a gaze object. Show. A VOP (Video Object Plane) in the image input unit 5 in FIG. 2 means each of the constituent elements when a composite image is formed by a plurality of component images, and is usually composed of shape data and texture data. However, in order to simplify the description, it is treated as a component image divided into regions of arbitrary shape. The three-dimensional part images obj _n (different parts are distinguished by n = 1, 2, ...) _{Have the} same number of VOP _mn (m = 1, 2,
Different viewpoints are distinguished by ..., and m =
1 and 2), and VOP ₀ is set as the background image 1. At this time, the distance between the two cameras may be the same as human eyes. Further, since the two cameras are arranged in parallel and adjusted so that there is no vertical displacement or distortion, the parallax of the image clipped by the clipping device does not have a vertical component but only a horizontal component. I shall.

First, a background image coding method will be described with reference to FIG. In the coding of moving images, a method called motion compensation predictive coding using block matching in which a coding unit is coded as a block (for example, composed of 16 × 16 pixels) is well known. In this embodiment, the background image coding method is the same as the motion compensation predictive coding method using this block matching. For example, the background image VOP ₀ is motion-compensated by the motion compensating unit 6 using the decoded image of the previous frame or field, and the detected motion vector is the variable length coding unit 10.
Is transmitted to

Thereafter, the subtracter 7 calculates the difference between the background image VOP ₀ and the motion-compensated image, and the difference data is transmitted to the variable length coding unit 10 through the conversion unit 8 and the quantization unit 9. . Further, this difference data is input to the variable length coding unit 10, passes through the dequantization unit 11 and the detransformation unit 12, the motion-compensated images are added by the adder 13, and the background image VOP ₀ is decoded. The image is stored in the frame memory 14. After that, the next input is encoded by this repetition.

Next, the encoding of the three-dimensional part image will be described. The sequence of encoding the three-dimensional component image obj ₁ will be described with reference to FIG. Encoding is performed according to the following procedures (a) to (d).

(A) The left-eye component image 43 of the three-dimensional component image obj ₁ is encoded within a frame or field.

(B) By performing parallax compensation using the decoded image of the component image 43, frame or inter-field coding is performed on the component image 44 for the right eye.

(C) The component image 45 is subjected to frame or inter-field coding by motion compensation using the decoded image of the component image 43.

(D) By performing parallax compensation using the decoded image of the component image 45, the component image 46 for the right eye is subjected to frame or inter-field coding.

That is, the image for the left eye is coded by motion compensation, and the image for the right eye is coded by parallax compensation using the image for the left eye at the same time. After that, encoding is performed by repeating the procedure of (c) and (d). However, when coding by repeating the procedures (c) and (d), instead of the procedures (c) and (d), the coding according to the procedures (a) and (b) is performed in order to prevent the propagation of prediction errors. May be performed at regular intervals.

In FIG. 1, the left-eye image VOP ₁₁ forming the three-dimensional component image obj ₁ is encoded by the same method as that used for encoding the background image. Similarly, three-dimensional parts image o
The right-eye image VOP ₂₁ forming bj ₁ is the left-eye image VOP _21.
Parallax compensation is performed by block matching using the decoded image of OP ₁₁ , and encoding is performed. At this time, the parallax compensation unit 1
The right-eye image VOP ₂₁ and the decoded image of the left-eye image VOP ₁₁ from the adder 15 are input to 6 and the parallax vector is obtained.

To obtain the parallax vector, as shown in FIG. 4, a left-eye component image 31 and a right-eye component image 32 are superposed on the same plane as an image 33, and matching is performed on the target block. By doing. Therefore, one disparity vector is obtained for one block. The calculated disparity vector is input to the disparity vector divider 17, and 1
It is divided into one global parallax vector and a local parallax vector obtained by subtracting the global parallax vector from each parallax vector.

That is, in the three-dimensional part image, the global parallax vector represents the parallax of one predetermined block,
The local parallax vector is the parallax of the other blocks minus the global parallax vector. For example,
The disparity vector v ₁ when the global disparity vector GV as shown in FIG. 5, a disparity vector v ₂ global disparity vector GV minus ones local disparity vector LV ₂ from
Therefore, the parallax vector v ₂ can be expressed as GV + LV ₂ .

Therefore, k (k = 1, 2, ...) Parallax vectors can be represented by one global parallax vector and k−1 local parallax vectors. At this time, the global parallax vector corresponds to the depth (location) of the three-dimensional component image in the three-dimensional space, and the local parallax vector is the local depth distribution (three-dimensional shape) in the three-dimensional component image.
Corresponding to

At this time, the amount of information of the local parallax vector can be reduced by setting the average of all parallax vectors as the global parallax vector. Also, the parallax vector obtained first may be the global parallax vector, and this reduces the amount of calculation when selecting the global parallax vector. Further, the magnitude of the parallax vector is set to a histogram, and the parallax vector that is most frequently generated is set to the global parallax vector, so that the existing position of the three-dimensional component image in the three-dimensional space can be appropriately expressed.

Generally, in comparison with having k disparity vectors, one global disparity vector and k- having a small value are used.
Having one local parallax vector can reduce the amount of information of the parallax vector.

In the stereoscopic image, the change in parallax amount between adjacent blocks in the stereoscopic image is small, but when there are a plurality of objects in the stereoscopic image and these objects are not made into parts, different objects overlap each other. The parallax in a part may change rapidly. In the present invention, since each component is encoded using the three-dimensional component image, such a rapid change in parallax does not occur within one three-dimensional component image. Therefore, the information amount of the local parallax vector can be further reduced by taking the difference between the local parallax vectors of the adjacent blocks.

These vectors are transmitted to the variable length coding unit 10 in FIG. In the subtractor 18, the right-eye image VOP ₂₁
The difference between the image and the parallax-compensated image is obtained, and is transmitted to the variable length encoding unit 10 through the conversion unit 19 and the quantization unit 20. Hereinafter, the following input is repeatedly encoded. Other stereoscopic component images are also encoded in the same manner. The data transmitted to the variable length coding unit 10 is variable length coded there, and multiplexed by the multiplexing unit 21.

Although one background image and n three-dimensional component images are encoded here, the number of background images is not limited to one and may be plural. Further, in FIG. 3, since the global parallax vector does not change so drastically in the time direction, the global parallax vector (for example, the global parallax vector obtained by the left-eye image 45 and the right-eye image 46) is set to 1
The information amount of the global parallax vector can be reduced by taking the difference between the global parallax vectors before the frame or field (for example, the global parallax vector obtained by the left-eye image 43 and the right-eye image 44).

In the present embodiment, the encoding of the two-lens type three-dimensional component image has been described, but the same can be applied to the multi-view three-dimensional component image.

Next, a second embodiment of the present invention will be described. In the present embodiment, the encoding device of the first embodiment can be applied to multi-view stereoscopic images. At this time, the background image is encoded by the same method as in the first embodiment.

FIG. 6 shows a second embodiment of the image coding apparatus according to the present invention. The input image in FIG. 6 is a multi-viewpoint image VOP _mn (m = 1, 2, ..., N = 1, 2, ...).
The image VOP ₁₁ and the image VOP ₂₁ are coded in the same manner as the image VOP ₁₁ and the image VOP ₁₂ of FIG. 1, respectively.
The decoded image VOP ₂₁ is output from the adder 51,
It is input to the parallax compensation unit 53. Furthermore, the global parallax vector obtained using the image VOP ₁₁ and the image VOP ₂₁ is
It is input to the parallax compensation unit 53.

The parallax compensating unit 53 performs parallax compensation on the input image VOP ₃₁ using the decoded image of VOP ₂₁ to obtain a parallax vector. The calculated parallax vector and the global parallax vector calculated using the image VOP ₁₁ and the image VOP ₂₁ are input to the differentiator 54, and the difference between them is output as the local parallax vector. That is, here, it is not necessary to obtain a new global parallax vector.

The fact that global parallax can be shared even with multi-view images will be described. To simplify the description, a three-dimensional part image having three viewpoints will be described.

For example, as shown in FIG. 7, the origin is O ₁ , x
A coordinate system is used in which the axis is the horizontal direction and the z axis is the depth direction.
On this coordinate, the camera group C _m (m = 1, 2, 3) is arranged at a point O so that the optical axis is parallel to the z axis at intervals of a.
_It is placed at the position of _m (m = 1, 2, 3). Similar to the first embodiment, it is assumed that the camera is adjusted so that there is no vertical displacement or distortion.

Further, the point A ₀ is used as a representative point of the three-dimensional part image obj ₁ , and the points A ₀ and O _m are connected to each other to intersect with the virtual image plane of the camera (indicating the image plane taken by the camera). Let A _m (m = 1, 2, 3). Let Q _m (m = 1, 2, 3) be the points where straight lines passing through the point O _m and extending parallel to the z-axis intersect the virtual image plane of the camera. These points Q _m represent the center points of the images by each camera.

Here, the x-coordinates of A ₁ , A ₂ and A ₃ are respectively defined as x ₁ , x ₂ and x ₃ . Since the virtual image plane of the camera and the x-axis are parallel, | A ₁ A ₂ | and | A ₂ A ₃ | are equal. From this and x ₁ <x ₂ <x ₃ , it becomes x ₁ −x ₂ = x ₂ −x ₃ (1).

Since the parallax vector d ₁ in the cameras C ₁ and C ₂ has only a horizontal component, it is a one-dimensional vector, and is the vector Q ₁ A ₁ minus the vector Q ₂ A ₂ . _{d 1 = x 1 - (x} 2 -a 1) becomes (2).

Next, the parallax vector d ₂ in the cameras C ₂ and C ₃ is the vector Q ₃ A ₃ minus the vector Q ₁ A ₂ , and d ₂ = x ₂ − (x ₃ −a) (3) Becomes

Therefore, from equations (1) to (3), d ₁ = d ₂ (4) In this way, the parallax vectors at the representative points between adjacent cameras are all equal. This is not limited to the three-viewpoint stereoscopic image, and the same applies to the case of the number of viewpoints other than three.

Therefore, in this method, since the global parallax vector can be regarded as the parallax vector at the representative point, it is sufficient to have one global parallax vector regardless of the number of viewpoints, and the parallax vector is obtained in a multi-viewpoint image. The amount of calculation at the time and the amount of information of the disparity vector itself can be reduced.

Therefore, in FIG. 6, the value output from the differentiator 54 is the local parallax vector obtained by subtracting the global parallax vector obtained between VOP ₁₁ and VOP ₂₁ from the parallax vector output from the parallax compensator 53. can do. The other VOPs in FIG. 6 are similarly coded.

Next, a third embodiment of the present invention will be described.

FIG. 8 shows an image decoding apparatus according to this embodiment. The decoding device of the present embodiment is for decoding data encoded using the encoding device of FIG. Image V for the left eye that constitutes the background image and the three-dimensional component image obj _1.
The decoding method of OP ₁₁ is the same as the motion compensation prediction decoding using general block matching. For the right-eye image VOP ₂₁ that forms the three-dimensional component image obj ₁ , parallax compensation is performed using the decoded image of the left-eye image VOP ₁₁ to perform decoding.

The global parallax vector and the local parallax vector are input to the parallax vector combiner 81, and normal parallax vectors are combined. The image editing unit 80 uses the parameter input unit 82 to move the component image in parallel to the image plane and the amount of change in the global parallax vector used to move the component image vertically to the image plane. Is entered as a value for editing. The position information of the decoded three-dimensional component image, the global parallax vector, and the representative block having the global parallax vector is also input to the image editing unit 80. The image editing unit 80 edits the three-dimensional part image using these input values. The decoded background image and the edited component image are combined by the image combining unit 83 into one image.

Since the global parallax vector corresponds to the depth (position of existence) of the three-dimensional part image in the three-dimensional space, the part image having a larger global parallax vector exists at a position closer to the camera in the depth direction. In the image editing unit 80, the global parallax vector is output in order from the smallest stereoscopic component image. A background image is first input to the image compositing unit 83, and the sequentially input three-dimensional component images are overwritten on the background image. As a result, when a plurality of component images are overlaid on the left and right images that are initially input, the component image that is closer to the front in the depth direction is written above. By using the global parallax vector in this way, it is possible to correctly represent the overlapping of the component images when synthesizing the three-dimensional component images.

First, the case where the three-dimensional component image is moved in parallel in the horizontal direction by inputting the movement parameter in the x direction will be described.

In FIG. 9, the origin is O ₁ , and the horizontal direction is x.
Take a coordinate system with the axis and the depth direction as the z-axis. On this coordinate, the cameras L and R are arranged at the position of the point O _m (m = 1, 2) so that the optical axis is parallel to the z axis at intervals of a. Further, the point A ₀ (x _A , z _A ) is used as a representative point of the three-dimensional component image obj ₁ , the point A ₀ and the point O _m are connected, and the points intersecting the virtual image plane of the camera are respectively _Am (m = 1, m = 1, 2).

A point where a straight line passing through the point O _m and extending parallel to the z axis intersects with the virtual image plane of the camera is Q _m (m = m).
1, 2). These points Q _m represent the center points of the images by each camera. Further, the distance from the virtual image plane of the camera to the x-axis is w, the point where the z = z _A plane intersects the optical axis of the camera L is S ₁ , and the point where the plane intersecting the optical axis of the camera R is S ₂ (a, z
_a ). Here, the coordinates of A ₁ and A ₂ are (x _a1 ,
w), (x _a2 , w), the one-dimensional vector Q ₁ A ₁ ,
Q ₂ A ₂ becomes x _a1 and x _a2- a, respectively. Where △ O
_{From 1} A ₀ S ₁ ∽ΔO ₁ A ₁ Q ₁ and ΔO ₂ A ₀ S ₂ ∽ΔO ₂ A ₂ Q ₂ , z _A / w = x _A / x _a1 = (x _A −a) / ( x _a2 −a) (5), the x coordinate x _A of the point A ₀ is x _A = a × x _a1 / (x _a1 −x _a2 + a) = a × x _a1 / d _A ( 6) is represented. d _A represents the global disparity vector of the object 100, and d _A = x _a1 − (x _a2 −a) (7).

Here, it is assumed that the moving amount in the x direction of the three-dimensional component image on the left image surface is represented by the moving parameter P _x in the parallel direction. By inputting the parameter P _x , the object 1
If 00 is translated to the position of the object 101, the representative point A ₀ (x _A , z _A ) of the object 100 is the point B ₀ (x _B , z _A ).
Then, the point A ₁ is translated to the point B ₁ (x _a1 + P _x , w).
Assuming that the point where the line segment connecting the representative point B ₀ and the point O ₂ of the object 101 intersects the virtual image plane of the camera is point B ₂ (x _b2 , w), ΔO ₁ O ₂ A ₀ ∽ΔA ₁ A ₂ A ₀ and △ O ₁ O ₂ B ₀ ∽ △
From B ₁ B ₂ B ₀ , since there is a relation of z _A / (z _A −w) = a / (x _a2 −x _a1 ) = a / (x _b2 −x _a1 −P _x ) (8), movement amount x _b2 -x _a2 in the right image plane when the representative point a ₀ of the object 100 has moved to the representative points B ₀ of the object 101 is a _{x b2 -x a2 = P x (} 9).

Therefore, the representative point A ₀ of the object 100 is the object 1
The amount of movement on the left and right image planes when moving to the representative point B _{0 of} 01 is equal to P _x . At this time, similarly to the equations (6) and (7), the x coordinate x _B of the point B ₀ is x _B = a × (x _a1 + P _x ) / d _B (10).

[0060] d _B represents a global disparity vector with the object _{101, d B = (x a1} + P x) - a _{(x b2 -a) (11)} .

From the equations (7), (9) and (11), d _A = d _B (12) and the global parallax vectors d _A and d _B are equal.

At this time, equations (6), (10), and (1
Than 2), and _{x B -x A = P x /} x a1 × x A (13). Therefore, when moving the object in the horizontal direction, the following steps 1) and 2) may be performed.

1) Input the movement parameter P _x in the x direction, and the x coordinate x of the representative point A ₀ of the object 100 on the left image.
_a1 is moved in parallel to the position of the x coordinate x _a1 + P _x of the representative point B ₀ on the left image.

2) Representative point A ₀ of the object 100 on the right image
X coordinate x _a2 of the representative point B ₀ on the left image x _a2 +
Move parallel to P _x position.

As a result, the object 100 moves to the position of the object 101, and the moving amount in the three-dimensional space at this time is calculated by the equation (1
From 3), P _x / x _a1 × x _A.

Next, a case where the three-dimensional component image is moved in the depth direction (direction perpendicular to the image plane) by inputting the movement parameter in the z direction will be described with reference to FIG. However, the coordinate system and the object 100 used in FIG.
Is the same as in FIG. Movement parameter P in z direction
_It is assumed that the distance between the camera and the object after the movement is P _z times the distance before the movement due to _z . In FIG. 10, when the parameter P _z is input and the object 100 moves to the position of the object 102 on the straight line passing through the points O ₁ and A ₀ , the object 10
The representative point A ₀ (x _A , z _A ) of ₀ moves to the point C ₀ (x _C , z _C ) but the point A ₁ does not move. At this time, z _C = P _z × z _A (14) The point where the line segment connecting the points O ₂ and C ₀ intersects the virtual image plane of the camera is C ₂ , and the point where the z = z _C plane intersects the z axis is T ₁ . The coordinates of C ₂ are respectively (x _c2 , w), and the global parallax vector of the point C ₀ after moving the point A _{0 of the} object 100 on a straight line passing through the points O ₁ and A ₀ is d _C.

Further, x _C becomes x _C = a × x _a1 / d _C (15) in the same manner as in the case of obtaining the equation (6). Here, d _C = (x _a1 −x _c2 + a) (16). From ΔO ₁ A ₀ S ₁ ∽ ΔO ₁ C ₀ T ₁ , there is a relation of x _C / x _A = z _C / z _A (17). Therefore, in equation (17), equation (6) and equation (6) 15) is substituted to obtain z _C , z _C = (d _A / d _C ) × z _A (18)

Therefore, from the equations (14) and (17), the global parallax vector d _C , the x coordinate x _C of the point C ₀ and the coordinate x _c2 of the point C ₂ are: d _C = d _A / P _z (19) x _C = P _z × a × x _a1 / d _A (20) x _c2 = x _a1 + a − d _A / P _z (21). Therefore, when moving the object 100 to the object 102,
The representative point A ₁ of the object 100 on the left image is not moved, and the point A ₂ on the right image is moved by x _c2 −x _a2 .

Here, the object 102 is moved by − (x _C −x _A ) to be an object 103. Representative point C _{0 of the} object 102
The moved point is defined as the representative point D ₀ of the object 103. Further, the point where the straight line connecting the points D ₀ and O ₁ intersects the virtual image plane of the camera is D ₁ (x _d1 , w). The x-coordinate x _d1 of the point D ₁ is expressed by the formula (6) and the formula (19) as follows: x _d1 = x _a1 / P _z (22) Therefore, when moving the object 102 to the object 103,
Point B ₁ on the left image and point B ₂ on the right image x
Only move _a1- x _d1 . Therefore, when the object 100 is moved in the depth direction (the position of the object 103), the following steps 3) and 4) are performed.

3) The movement parameter P _z in the z direction is input, the global parallax d _C is calculated by the equation (19), and the point A1 on the left image of the three-dimensional component image is horizontally (x _d1 −
x _a1 ) Move.

4) From the equations (21) and (22), x _c2
And the values of x _d1 are obtained, and the point A ₂ on the right image of the three-dimensional part image is moved (x _c2 −x _a2 + x _a1 −x _d1 ).

Therefore, by changing the parallax vector to two levels, that is, the global parallax vector and the local parallax vector, the movement parameter P _z in the z direction is input, the global parallax vector is changed, and the parallax vector in the depth direction of each component image is changed. The position can be changed easily. Further, by combining the movement in the horizontal direction (procedures 1 and 2)) and the movement in the depth direction (procedures 3) and 4)), the object can be moved to an arbitrary position in the three-dimensional space.

Since parallax is not related to the movement in the vertical direction, it may be simply moved in parallel on the screen.

Further, in the present embodiment, the two-lens type three-dimensional component image is used, but the same applies to the multi-view three-dimensional component image.

[0075]

As described above, according to the coding apparatus of the present invention, for all parallax vectors used when coding a three-dimensional component image, one of the parallax vectors is the global parallax vector ( (Corresponding to the depth of the stereoscopic component image in the three-dimensional space), and the difference between this global parallax vector and another parallax vector is taken as the local parallax vector (corresponding to the local depth distribution in the stereoscopic component image). Since the disparity vector is divided into two, the information amount of the disparity vector can be reduced and the coding efficiency can be improved. In the present invention, two types of vectors, a global parallax vector and a local parallax vector, are used. However, since the search for obtaining these can be performed only once, the calculation amount can be reduced as compared with the case of performing the search twice. .

According to the encoding device of the present invention, when a stereoscopic image is divided into a background image and a stereoscopic component image and encoded, a portion having no parallax with respect to images of all viewpoints is encoded as one background image. By doing so, the amount of information of the background image can be reduced, and the coding efficiency can be improved.

According to the encoding apparatus of the present invention, by performing the encoding separately for the three-dimensional component image cut out from the three-dimensional image and the background image, a sudden change in parallax does not occur in the three-dimensional image, and matching is performed. Reliability is improved, and even when the information amount of the local parallax vector is reduced by taking the difference between the local parallax vectors at the adjacent positions, a large absolute amount does not occur.

According to the encoding apparatus of the present invention, the information amount of the local parallax vector can be reduced by setting the average of all parallax vectors in the three-dimensional component image as the global parallax vector.

According to the encoding apparatus of the present invention, the amount of calculation for selecting a global parallax vector is reduced by using the parallax vector first obtained in the three-dimensional component image as the global parallax vector.

According to the encoding apparatus of the present invention, the size of the disparity vector in the three-dimensional part image is set as a histogram, and the disparity vector with the highest frequency of occurrence is set as the global disparity vector. The depth of the image can be appropriately expressed.

According to the encoding apparatus of the present invention, the information amount of the global parallax vector can be reduced by taking the difference between the global parallax vector of the current frame or field and the global parallax vector of the immediately preceding frame or field. it can.

According to the encoding device of the present invention, since one global parallax vector is required for one multi-view stereoscopic component image in the spatial direction, the calculation amount and information amount of the global parallax vector can be reduced. , The encoding efficiency can be improved.

In the decoding apparatus of the present invention, when a plurality of component images overlap each other when synthesizing images of multiple viewpoints, a component image having a larger global parallax is overwritten so that the overlap is correctly represented. A viewpoint stereoscopic image can be decoded.

In the decoding apparatus of the present invention, a movement parameter in the depth direction is input and the value of the global parallax vector of the three-dimensional component image is changed, so that the three-dimensional component image having an arbitrary depth can be easily decoded.

[Brief description of drawings]

FIG. 1 is a block diagram of an encoding device that encodes a background image and a plurality of three-dimensional component images, which is an embodiment of the present invention.

FIG. 2 is a diagram illustrating cutting out a three-dimensional component image from an image input by a camera.

FIG. 3 is a diagram illustrating an order in which a three-dimensional component image is encoded.

FIG. 4 is a diagram illustrating an example of calculating a disparity vector.

[Fig. 5] Fig. 5 is a diagram illustrating that a parallax vector is represented by a global parallax vector and a local parallax vector.

[Fig. 6] Fig. 6 is a configuration diagram of an encoding device when encoding a multi-view stereoscopic component image, which is another embodiment of the present invention.

[Fig. 7] Fig. 7 is a diagram for describing that a global parallax vector is commonly used in a spatial direction when encoding a multi-view stereoscopic component image.

FIG. 8 is a configuration diagram of a decoding device that decodes a background image and a plurality of three-dimensional component images, which is an embodiment of the present invention.

FIG. 9 is a diagram illustrating parallel movement of a three-dimensional component image.

FIG. 10 is a diagram illustrating movement of a three-dimensional component image in the depth direction.

FIG. 11 is an explanatory diagram of a conventional example regarding compression of a multi-viewpoint image.

FIG. 12 is an explanatory diagram of a conventional example for obtaining a two-step motion vector.

[Explanation of symbols]

 1 Background Image 2 Original Image 3 Clipping Device 43, 44, 45, 46 Three-dimensional Component Image 5 Image Input Unit 6 Motion Compensation Unit 7, 18 Subtractor 8, 19 Transforming Unit 9, 20 Quantizing Unit 10 Variable Length Encoding Unit 11 Inverse quantizer 12 Inverse transformer 13, 15, 51 Adder 14 Frame memory 16, 53 Parallax compensator 17, 52 Parallax vector divider 21 Multiplexer 31 Left-eye stereoscopic component image 32 Right-eye stereoscopic component image 33 Composite image 54 difference device 60 comparator 70 large block 71 small block 80 editing unit 81 disparity vector combiner 82 external input unit 83 image combining unit 100, 101, 102, 103 object

Claims (9)

[Claims] 1. A multi-view stereoscopic image is divided into at least one parallax image having parallax and at least one background image having no parallax, input means is used, and a motion amount between frames or fields of the component image is used. A multi-view coding apparatus including, for each component image, a motion compensation unit that performs motion compensation prediction, a parallax compensation unit that performs parallax compensation prediction using a parallax amount between frames or fields of component images, and a parallax vector division unit. The parallax vector dividing means outputs one of the parallax vectors obtained by the parallax compensating means as a global parallax vector, and subtracts the global parallax vector from the remaining parallax vectors as a local parallax vector. A multi-view image encoding device characterized by: 2. A multi-view stereoscopic image of three or more viewpoints, which is divided into at least one component image having parallax and at least one background image having no parallax, and inputting means, and movement between frames or fields of component images. For each component image, a motion compensation unit that performs motion compensation prediction using the amount and a parallax compensation unit that performs parallax compensation prediction using the parallax amount between the frames or fields of the component images are provided for each component image. A multi-viewpoint encoding device including one disparity vector division unit and at least one difference unit, wherein the disparity vector division unit sets one of the disparity vectors obtained between specific viewpoints as a global disparity vector. , The remaining disparity vector minus the global disparity vector is output as the local disparity vector, and the difference A multi-view image coding device, wherein the device outputs the differences between all the parallax vectors obtained between other viewpoints and the global parallax vector. 3. The multi-view image encoding device according to claim 1, wherein the parallax vector dividing unit outputs the average of all parallax vectors in the stereoscopic component image as a global parallax vector. 4. The multi-viewpoint image coding device according to claim 1, wherein the parallax vector dividing unit outputs the parallax vector first obtained in the three-dimensional component image as a global parallax vector. 5. The parallax vector dividing means according to claim 1 or 2, wherein the magnitude of the parallax vector in the three-dimensional component image is made into a histogram as a global parallax vector, and the parallax vector having the highest frequency of occurrence is output. Multi-view image coding device. 6. The parallax vector dividing means according to claim 1 or 2 outputs the difference between the global parallax vector of the current frame or field and the global parallax vector of the immediately preceding frame or field as the global parallax vector of the current frame or field. A multi-view image encoding device characterized by: 7. The multi-view image encoding device according to claim 1 or 2, wherein the parallax vector division means outputs a difference between the local parallax vector and the local parallax vector obtained immediately before as the local parallax vector. 8. A multi-view image decoding apparatus comprising: a motion compensating means for performing motion compensation and a parallax compensating means for performing parallax compensation by using a motion amount and a parallax amount between frames or fields of a multi-view stereoscopic image. Parallax vector synthesizing means for outputting the parallax vector to the parallax compensating means by using the global parallax vector and the local parallax vector, parameter input means for outputting the movement parameter in the depth direction to the image editing means, and the input parameter A multi-viewpoint image decoding apparatus characterized by including an image editing means for changing the depth of a three-dimensional component image using . 9. The multi-view image decoding device according to claim 8, wherein the image editing means outputs the three-dimensional component images in order from the smallest global parallax vector, and the image synthesizing means sequentially inputs the background image. A multi-view image decoding device characterized by synthesizing images by overwriting an input three-dimensional component image on a background image.