JP3194258B2 - Image coding method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for encoding a plurality of images obtained by photographing with a plurality of cameras having slightly different viewpoints.

[0002]

2. Description of the Related Art In the prior art, an image photographed by one camera is encoded, or one of a plurality of images photographed by a plurality of cameras is selected. It was common to encode. When only one camera is used, camera operations such as panning and zooming are performed on the transmission side in order to change the subject to be encoded or to enlarge a part of the image. When a plurality of cameras are used, the image supplied to the encoding device is switched according to the intention of the transmission side or the reception side. Further, when encoding a plurality of images at the same time, each image is separately processed by an independent encoding device.

[0003]

When an image is encoded and transmitted in a video conference system or the like, there is a case where it is desired to simultaneously encode a wide subject range that cannot be covered by one camera. In this case, as shown in FIG. 4 (a), a plurality of cameras may be used on the transmitting side to obtain a plurality of images by slightly different viewpoint positions or line-of-sight directions to obtain a plurality of images, all of which may be encoded. The receiving side can decode and display the plurality of images simultaneously or selectively as necessary. By doing so, it is possible to avoid the inconvenience associated with camera operations such as camera switching and zooming / panning that have conventionally been performed for switching images to be transmitted. Also, as shown in FIG. 4B, if a plurality of images obtained by simultaneously photographing the same subject from different viewpoints are encoded, on the receiving side, each observer can view an image closer to his own monitor on his own monitor. Can be selected and displayed. This makes it possible to realize more natural and realistic image communication in consideration of the position of the subject on the transmitting side and the direction of the line of sight of the observer on the receiving side.

When encoding a plurality of images obtained by photographing with a plurality of cameras having different viewpoints as described above, the conventional encoding method processes each image separately and independently, Since the correlation existing in the above is not used, the total code amount increases in proportion to the increase in the number of images to be coded. Therefore, there have been problems such as an increase in communication cost and a limitation in the number of images that can be transmitted simultaneously due to restrictions on the communication line capacity.

An object of the present invention is to encode a plurality of images obtained by photographing with a plurality of cameras each having a slightly different viewpoint, and to use the correlation existing between the images to code the entire image. It is an object of the present invention to provide an image encoding method with a reduced amount.

[0006]

According to a first image encoding method of the present invention, in encoding a plurality of input images obtained by photographing with a plurality of cameras having slightly different viewpoints,
Means for generating an encoded signal of the selected image and a locally reproduced image for one representative image selected from the plurality of input images, and for each of the other plurality of input images, Detecting a parallax parameter describing a displacement of the input image caused by parallax when the representative image is used as a reference image,
Generating a predicted image from the local reproduction image using the disparity parameter, detecting a difference signal between the input image and the predicted image, and encoding the difference signal and the disparity parameter. Features.

According to a second image encoding method of the present invention, in the detection of the parallax parameter, a corresponding position on a reference image is detected for each pixel of an input image or for each small block including a plurality of pixels. A difference between coordinate positions is set as a disparity vector, and a disparity parameter is estimated from the disparity vector using a least squares error method.

According to a third image encoding method of the present invention, in detecting the parallax parameters and generating a predicted image using the parallax parameters, an input image is constituted by a plurality of object regions having different distances from a camera. Means for detecting a disparity vector for each pixel or for each small block composed of a plurality of pixels; means for dividing the entire image area into a plurality of areas using the disparity vector; and for each of the divided areas, The apparatus further comprises means for estimating a disparity parameter from a disparity vector, and means for generating a predicted image from the locally reproduced image using the detected disparity parameter for each of the divided areas.

[0009]

The first image coding method of the present invention will be described with reference to FIG. In the block diagram of FIG.
Although the case where there are three of the input images 102 and 103 has been described, when the number of input images further increases, it is only necessary to add blocks for performing the same processing as for the input images 102 or 103. First, 101 is set as a representative image from a plurality of input images.
It is assumed that is selected. The input image 101 is encoded by the encoding circuit 11 by a method such as a motion-compensated inter-frame prediction encoding method, and the reproduced image 110 is encoded by the decoding circuit 16 on the receiving side.
Is decrypted. At the same time, the encoding circuit 11
A local reproduction image 105, which is the same image as 0, is generated and output.

On the other hand, for the other input images 102 and 103, encoding is performed for each input image using the correlation between the images when the representative image 101 is used as a reference image. First, when encoding the input image 102, the parallax parameter detection circuit 12 detects a parallax parameter 106 describing the displacement between the two images from the input image 102 and the reference image 101. The operation of the parallax parameter detection circuit 12 will be described below. Since the positional relationship between the subject and the camera is different between the input image 102 and the reference image 101 captured by a camera having a slightly different viewpoint, the coordinate positions of the corresponding pixels are different between the two images. The displacement of the coordinate position can be described as a function of the position on the input image 102. As an example of this function, when the entire subject can be approximated by a two-dimensional plane, the vector (v _x , v _y ) indicating the displacement of the coordinate position of the corresponding pixel between the two images is expressed by the equation (1) ( It can be described in the form of 2). v _x = ax + by + (gx + hy) x + e (1) v _y = cx + dy + (gx + hy) y + f (2)

In equations (1) and (2), (v _x , v _y ) is the displacement amount in the x-axis (horizontal) direction and the y-axis (vertical) direction at the position (x, y) on the input image 102. Show. This is called a disparity vector. Eight coefficients (a, b, c, d, e, f, g, h) characterizing equations (1) and (2)
Is referred to as a parallax parameter. FIG. 5 shows equation (1).
The relationship between the disparity vector described in (2) and the disparity parameter will be described using a specific example.
The case where the y (vertical) axis of the dimensional plane is perpendicular to the optical axis of the camera is shown. FIG. 5A shows a case in which the viewpoint position in the input image 102 is shifted to the left from that of the reference image 101, and the disparity vectors are all rightward and have the same size. However, the hatched portion is an image portion newly appearing on the input image 102 due to a change in the viewpoint position.
Since there is no correlation with the reference image 101, a disparity vector cannot be detected. For the disparity vector as shown in FIG. 5A, the only effective disparity parameter in equations (1) and (2) is e. FIG. 5B shows a case where the line of sight of the input image 102 is more oblique to the subject than in the case of the reference image 101, and it can be seen from the disparity vector that the image is displaced in a fan shape. In this case, the parallax parameter g is effective. Further, FIG. 5C shows a case where the viewpoint position of the input image 102 is closer to the subject than when the reference image 101 is, or a case where a zooming operation is performed, and the disparity vector is radially detected. The effective parallax parameters in this case are a,
d. In addition, when the camera rotates around the optical axis, the parameters b and c are valid, and when the camera tilts and moves up and down, the parameters h and f are valid.
That is, it can be seen that the equations (1) and (2) can describe the displacement amount of the coordinate position on the image due to various changes in the viewpoint position. Equations (1) and (2) are functions when the entire subject is approximated by a two-dimensional plane. When the entire subject is approximated by a curved surface, a higher-order function related to the coordinate position (x, y) is used. I just need. Also, the parallax parameter detection circuit 1 of FIG.
In 2, the input image 101 is used as a reference image for accurate parallax detection, but the local reproduction image 105 can be used as a reference image.

Next, in FIG. 1, a predicted image generation circuit 13 uses the detected parallax parameters 106 to convert a locally reproduced image 105 into a predicted image 10
7 is generated. The detected parallax parameter is expressed by equation (1).
By substituting into (2), the displacement amount of the position of the corresponding pixel between the images can be obtained by calculation. So input image 1
A coordinate position where the displacement amount is compensated is obtained for each pixel of 02, and a pixel value is read from the corresponding coordinate position of the local reproduced image 105 to be used as a predicted value to generate a predicted image 107. Further, the subtracter 14 calculates a difference between the predicted image 107 and the input image 102 and outputs the result as a predicted difference image 108.

The encoding circuit 15 encodes and transmits the predicted difference image 108 and the disparity parameter 106, and the decoding circuit 17 on the receiving side decodes the reproduced difference image 111 and the disparity parameter 112. The predicted image generation circuit 18 generates a predicted image 113 from the reproduced image 110 using the parallax parameters 112, and outputs a reproduced image 114 by adding the reproduced difference image 111 to the predicted image 113 by the adder 19.

In the image coding method described above, the input image 101 selected as the representative image is coded in the same manner as in the conventional method, but the second input image 102 is input image 1
Since only the component that is obtained by subtracting the correlation existing with 01 is encoded, it is possible to reduce the transmission code amount as a whole. In addition, by using the same method as the input image 102 for the third input image 103, it is possible to realize efficient encoding by subtracting the correlation with the input image 101.

According to a second method of the present invention, a parallax vector indicating a displacement of a coordinate position with respect to a reference image is detected for each pixel or small block of an input image, and the least square error method of the parallax vector is used. This is a method of estimating a parallax parameter that characterizes a function that describes the amount of displacement in the entire image. In the present invention, it is assumed that the entire subject can be approximated by a two-dimensional plane, and the case where the function can be described by equations (1) and (2) is targeted. The principle of this parallax parameter estimation method is shown in FIG.
This will be described with reference to FIG. First, the reference image 201 and the input image 202 are supplied to the disparity vector detection circuit 21. Here, as the reference image 201, the input image 101 selected as the representative image as shown in FIG. 1 may be used, or the local reproduction image 105 obtained by encoding and decoding the input image 101 may be used. In the disparity vector detection circuit 21, the input image 2
02 for each pixel or each small block.
The corresponding coordinate position above is detected, and the displacement amount of the coordinate position is defined as the disparity vector of the pixel or the small block. The displacement amount can be detected by a block matching method, a luminance gradient method, or the like. The value 203 of the disparity vector (v _x , v _y ) detected for each pixel or small block is supplied to the disparity parameter estimation circuit 22 together with the value 204 of the coordinate position (x, y) of the pixel or small block.

[0016] Next a number in the parallax parameter estimation circuit 22 in within an input image 202 detected disparity vector (v ⁱ _x,
v ⁱ _y ) and the value 204 of the coordinate position (x ⁱ , y ⁱ ) are subjected to the least squares error method for equations (1) and (2), and the parallax parameters (a, b, c) , D,
e, f, g, h) are estimated. Now the N Found _{^{v i x, v i y,}} x i, wherein with respect to y ⁱ (1)
The parallax parameters (a, b, c, d, e, f,
g, h), the mean square error MSE of the measured values of v _x and v _{y with} the actually measured values is given by equations (3) and (4).

(Equation 1)

Here, the equations (3) and (4) are replaced by parameters a,
By differentiating each of b, c, d, e, f, g, and h and setting the result equal to 0, the value of the parameter that minimizes the mean square error MSE can be obtained. By using the above-described method, it is possible to know the parallax parameter describing the displacement between images from only the input image and the reference image by a simple method. Also, if the fluctuation of the viewpoint position and the line-of-sight direction is known to some extent, the parallax parameters to be detected are also limited, so that the calculation for the detection can be omitted.
As shown in FIG. 5, if the y (vertical) axis of the two-dimensional plane of the subject is perpendicular to the camera optical axis and the rotation about the camera optical axis can be ignored, the parameters b, Simplification can be achieved by setting c and h to all 0. When estimating the disparity parameter from the disparity vector as shown in FIG. 5, the hatched portions in FIGS. 5A and 5B are not used for calculating the parameter estimation since there is no correlation between the images. When the parallax vector is detected by block matching, the hatched portion can be easily determined because there is no correlation between images and a matching error increases.

In the third image encoding method of the present invention, detection of a parallax parameter and generation of a predicted image are performed for each of a plurality of subject regions in the image. The principle of the processing for each subject area will be described with reference to FIGS. In an image composed of a plurality of subjects at greatly different distances from the camera, different parallax parameters may be detected for each subject.
FIG. 6 is a diagram for explaining that, in an image obtained by photographing two subjects having greatly different distances from the camera with two cameras having different viewpoints, a detected disparity vector differs between subjects. The image (b) photographed from the viewpoint B with respect to the image (a) photographed from the viewpoint A is set as a reference image. At this time, the subject C closer to the camera in both images is almost at the center of the images. Therefore, the disparity vector on the subject C becomes almost zero. On the other hand, for the subject D far from the camera, the viewpoint B shoots the left side more.
A rightward disparity vector on the subject D is detected. That is, since the disparity vectors are different between subjects whose distances to the camera are significantly different, it can be seen that the corresponding disparity parameters are also different. Therefore, the image is divided into a plurality of regions by regarding the portion where the disparity vector changes discontinuously as the boundary of the subject. Next, a parallax parameter may be estimated for each of the divided regions using a parallax vector detected in the region.

FIG. 3 is a block diagram showing an embodiment for realizing a predicted image generation method for detecting a parallax parameter and generating a predicted image for each subject area. First, the reference image 301 and the input image 302 are supplied to the disparity vector detection circuit 31, and the disparity vector 303 is provided for each pixel or each small block.
Is detected and output together with the detected coordinate position 304. Next, the area division circuit 32 generates area division information 305 for dividing the input image 302 into a plurality of areas from the supplied disparity vector 303. Here, the input image 30
When 2 is composed of a plurality of subjects, the fact that the detected parallax vector is discontinuous at the boundary between the subjects is used. That is, if the difference between the values between adjacent disparity vectors is larger than a predetermined threshold, the detected coordinate position of the disparity vector is set as the region boundary point. This processing is performed on the disparity vector detected in the entire image, and the detected area boundary points are continuously connected to draw a boundary line, thereby obtaining a result of dividing the entire image into a plurality of areas. Based on this result, area division information 305 indicating to which of the divided areas each pixel or each small block constituting the input image 302 belongs is output.

Next, the parallax parameter estimation circuit 33 estimates a parallax parameter 306 for each region from the detected parallax vector 303 with reference to the region division information 305. The estimation of the parallax parameter can be realized by using the least square error method shown in the second method of the present invention. In the predicted image generation circuit 34, the area division information 305
To determine which region belongs to each pixel or each small block of the input image 302, select a parallax parameter, and calculate a corresponding coordinate position on the reference image 301 from the selected parameter. . Then, a predicted image 308 is generated by reading the value of a pixel from the corresponding coordinate position on the local reproduction image 307 obtained by code-decoding the reference image 301 and using the pixel value as the predicted value of the pixel or the small block. The subtracter 35 subtracts the predicted image 308 from the input image 302 and outputs a predicted difference image 309.
Also, a plurality of parallax parameters 306 and area division information 305
Is also output as information to be encoded.

[0021]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described with reference to the drawings.
FIG. 1 is a block diagram of an embodiment for realizing a first image encoding method according to the present invention. In FIG. 1, an encoding circuit 1
1 is an encoded signal 104 obtained by encoding the input image 101;
A local reproduction image 105 obtained by decoding and reproducing this is output. In the parallax parameter detection circuit 12, the input images 101, 1
02, a parallax parameter 106 is detected. The predicted image generation circuit 13 generates a predicted image 107 from the locally reproduced image 105 using the parallax parameter 106, and
In step 4, a difference between the input image 102 and the predicted image 107 is obtained, and a predicted difference image 108 is output. The encoding circuit 15 outputs an encoded signal 109 obtained by encoding the prediction difference image 108 and the disparity parameter 106. The decoding circuit 17 decodes the encoded signal 109 and outputs a prediction difference image 111 and a disparity parameter 112. The predicted image generation circuit 18 generates a predicted image 113 from the reproduced image 110 using the parallax parameters 112, and the adder 19 adds the predicted difference image 111 to the predicted image 113 to output a reproduced image 114.

FIG. 2 is a block diagram of an embodiment for realizing a parameter detection system using the least square error method in the second system of the present invention. In FIG. 2, the disparity vector detection circuit 21 detects a disparity vector 203 for each pixel or each small block from the reference image 201 and the input image 202, and outputs the detected disparity vector 203 together with the coordinate position 204 of the pixel or the small block. The parallax parameter estimation circuit 22 estimates and outputs a parallax parameter 205 from the parallax vector 203 and its detected coordinate position 204 using the least squares error method.

FIG. 3 is a block diagram of an embodiment for realizing a predicted image generation method for detecting a parallax parameter and generating a predicted image for each subject region in the third image coding method according to the present invention. In FIG. 3, the disparity vector detection circuit 31 detects a disparity vector 303 for each pixel or each small block from the reference image 301 and the input image 302, and outputs the detected disparity vector together with the coordinate position 304 of the pixel or the small block. Next, the region division circuit 32 detects discontinuity of the disparity vector 303 and outputs region division information 305 for dividing the input image 302 into a plurality of regions. The parallax parameter estimating circuit 33 estimates and outputs a parallax parameter 306 from the parallax vector 303 for each divided region with reference to the region division information 305. The parallax parameter 306 is selected for each pixel or each small block with reference to the predicted image generation circuit 34 area division information 305, and the local reproduction image 307 is selected using the selected parallax parameter 306.
To generate and output a prediction image 308. The subtracter 35 subtracts the predicted image 308 from the input image 302 and outputs a predicted difference image 309.

[0024]

As described above, according to the present invention, in encoding a plurality of images taken by a plurality of cameras having slightly different viewpoints, one image is selected and encoded as a representative image. When encoding the remaining images, a predicted image is generated from the selected representative image, and only the difference image from the predicted image is encoded, which is effective in reducing the code amount of the plurality of images as a whole. . In addition, when generating a predicted image from a reference image, image conversion using a parallax parameter is performed, thereby achieving high prediction efficiency.

[Brief description of the drawings]

FIG. 1 is a basic block diagram of a first embodiment according to the present invention.

FIG. 2 is a basic block diagram of a second embodiment according to the present invention.

FIG. 3 is a basic block diagram of a third embodiment according to the present invention.

FIG. 4 is a diagram illustrating a relationship between a plurality of images obtained by photographing with a plurality of cameras each having a slightly different viewpoint.

FIG. 5 is a diagram illustrating an example of a disparity vector detected between two images captured by cameras having different viewpoints.

FIG. 6 is a diagram illustrating that different parallax vectors are detected for each subject when a plurality of subjects at different distances from the camera are photographed from different viewpoints.

[Explanation of symbols]

 11, 15 Encoding circuit 12 Parallax parameter detection circuit 13, 18, 34 Predictive image generation circuit 14, 35 Subtractor 16, 17 Decoding circuit 19 Adder 21, 31, Disparity vector detection circuit 22, 33 Disparity parameter estimation circuit 32 Area Dividing circuits 101, 102, 103, 202, 302 Input images 110, 109 Coded signals 105, 307 Locally reproduced images 106, 112, 205, 306 Parallax parameters 107, 113, 308 Predicted images 108, 111, 309 Predicted difference images 104 , 114 Reconstructed image 201, 301 Reference image 203, 303 Parallax vector 204, 304 Parallax vector detection coordinate position 305 Area division information

Claims (2)

(57) [Claims] 1. An image pickup method using a plurality of cameras each having a slightly different viewpoint.
Encode multiple input images obtained by shadingImage sign
It is a conversion method,  For a representative image selected from the plurality of input images,
Means for generating an encoded signal of the image and a locally reproduced image.
And for each of the plurality of other input images,
Input caused by parallax when the representative image is used as the reference image
Describe the displacement of the imageConsists of eight coefficientsParallax parameters
The local reproduction using the parallax parameters.
Generating a prediction image from the image, and calculating the input image and the prediction signal;
And the difference signal and the parallax parameter are detected.
Means for generating an encoded signal with the data.
The encoding method of the image. 2. The method of claim 1, further comprising detecting the parallax parameter and detecting the parallax.
Of prediction image using parametersGeneration  If the input image is in multiple subject areas with different distances from the camera
When configured, small pixels consisting of pixels or multiple pixels
Means for detecting a disparity vector for each lock, and using the disparity vector to convert the entire image area into a plurality of areas
Means for dividing, a disparity parameter from the disparity vector for each of the divided areas.
Means for estimating a meter; and the detected parallax parameter for each of the divided areas.
Means for generating a predicted image from the locally reproduced image using
To do with2. The image according to claim 1, wherein the image is characterized by:
Encoding scheme.