JP4944046B2 - Video encoding method, decoding method, encoding device, decoding device, program thereof, and computer-readable recording medium - Google Patents

Description

  The present invention is a method for generating distance information and a video signal of another viewpoint using a known video signal and distance information in a multi-view image and a multi-view video. The present invention also relates to a technique for encoding and decoding a multi-view image and a multi-view video using the same.

  A multi-view image is a plurality of images obtained by photographing the same subject and background with a plurality of cameras, and a multi-view video (multi-view video) is a moving image. Hereinafter, a moving image captured by one camera is referred to as a “two-dimensional moving image”, and a two-dimensional moving image group in which the same subject and background are captured is referred to as a multi-viewpoint moving image.

  The two-dimensional moving image has a strong correlation in the time direction, and the encoding efficiency is improved by using the correlation. On the other hand, in multi-view images and multi-view images, if the cameras are synchronized, the images of each camera corresponding to the same time are taken from different positions of the subject and background in the same state. There is a strong correlation. In the encoding of a multi-view image or a multi-view video, the encoding efficiency can be increased by using this correlation.

  First, a description will be given of a conventional technique related to a two-dimensional video encoding technique. H., an international encoding standard. In many conventional two-dimensional video coding systems such as H.264, MPEG-2, and MPEG-4, high-efficiency coding is performed using techniques such as motion compensation, orthogonal transformation, quantization, and entropy coding. I do. A technique called motion compensation is a method that uses temporal correlation between frames.

  H. The details of the motion compensation technique used in H.264 are described in the following Non-Patent Document 1, and the outline thereof will be described below. H. In H.264 motion compensation, a frame to be encoded is divided into blocks of various sizes, a frame that has already been encoded called a reference frame is selected for each block, and vector information indicating corresponding points called a motion vector is used. , Predict video. There are seven types of block division allowed at this time: 16 × 16, 16 × 8, 8 × 16, 8 × 8, 8 × 4, 4 × 8, and 4 × 4, and the direction and size of the movement of the subject. It is now possible to predict the video corresponding to the difference in small units. As a result, the residual of the encoding target represented by the difference between the predicted image and the original image becomes small, and high encoding efficiency is achieved.

  Next, a conventional multi-view image and multi-view video encoding method will be described. The difference between the multi-view image encoding method and the multi-view image encoding method is that the multi-view image has a correlation in the time direction in addition to the correlation between cameras. However, the method using the correlation between cameras can be the same in either case. Therefore, here, a method used in encoding multi-view video will be described.

  For multi-view video coding, in order to use the correlation between cameras, a multi-view video is encoded with high efficiency by "parallax compensation" in which motion compensation is applied to different camera images at the same time. Has traditionally existed. Here, the parallax is a difference between positions at which the same position on the subject is projected on the image planes of cameras arranged at different positions.

  A conceptual diagram of the parallax generated between the cameras is shown in FIG. In this conceptual diagram, the image plane of a camera with parallel optical axes is viewed vertically. In this way, the position where the same position on the subject is projected on the image plane of different cameras is generally called a corresponding point. In the disparity compensation, each pixel value of the encoding target frame is predicted from the reference frame based on the correspondence relationship, and the prediction residual and the disparity information indicating the correspondence relationship are encoded.

  In many methods, parallax is expressed as a vector on the image plane. For example, Non-Patent Document 2 includes a mechanism for performing parallax compensation in units of blocks. However, parallax in units of blocks is expressed by a two-dimensional vector, that is, by two parameters (x component and y component). That is, in this method, disparity information composed of two parameters and a prediction residual are encoded.

  On the other hand, in Patent Document 1 (video encoding method, video decoding method, video encoding program, video decoding program and computer-readable recording medium on which these programs are recorded), the camera parameters are used for encoding, and epipolar geometry is used. By expressing the disparity vector as one-dimensional information based on the constraint, the prediction information is efficiently encoded.

  A conceptual diagram of the epipolar geometric constraint is shown in FIG. According to the epipolar geometric constraint, in two cameras (camera 1 and camera 2), a point on the other image corresponding to a point on one image is constrained on a straight line called an epipolar line. In the method of Patent Document 1, in order to indicate the position on the epipolar line, the parallax with respect to all the encoding target frames is expressed by one parameter, the distance from the camera that captures the reference frame to the subject.

  Non-Patent Document 3 describes a technique for generating a parallax compensation image while assuming continuity between adjacent pixels that can be used in the implementation of the present invention.

Non-Patent Document 4 describes a method of encoding while using a parallax compensated image (composite image) as a reference image for an encoding target image and adding vector information indicating a prediction destination.
JP 2007-036800 A ITU-T Rec. H.264 / ISO / IEC 11496-10, “Advanced Video Coding”, Final Committee Draft, Document JVT-E022, September 2002. Hideaki Kimata and Masaki Kitahara, “Preliminary results on multiple view video coding (3DAV)”, document M10976 MPEG Redmond Meeting, July, 2004. Shinya Shimizu, Masaki Kitahara, Hideaki Kimata, Kazuto Kamikura and Yoshiyuki Yashima. “View Scalable Multiview Video Coding using 3-D Warping with Depth Map.” IEEE Transactions on Circuits and Systems for Video Technology, Vol.17, No.11, pp .1485-1495, 2007. Masaki Kitahara, Hideaki Kimata, Shinya Shimizu, Kazuto Kamikura and Yoshiyuki Yashima. “Multi-view Video Coding using View Interpolation and Reference Picture Selection.” IEEE International Conference on Multimedia and Expo, pp.97-100, 2006.

  According to the conventional multi-view video encoding method, when the camera parameter is known, the epipolar geometric constraint is used to determine the distance from the camera to the subject with respect to the reference frame regardless of the number of cameras. Only by encoding the dimension information, it is possible to realize the parallax compensation for the encoding target frames of all the cameras, and it is possible to efficiently encode the parallax information.

  However, when the distance between the camera that captured the reference frame and the camera of the encoding target frame is increased, the quality of the predicted video is degraded due to the effects of illumination and occlusion. As a result, many prediction residual signals must be encoded, and efficient encoding cannot be realized.

  As a similar problem, in normal video encoding, there is a problem that prediction efficiency decreases due to an increase in the time interval between the reference frame and the encoding target frame. In general video coding, this problem is addressed by setting different reference frames for each frame to be encoded.

  Therefore, as a technique for avoiding the problem with respect to the decrease in the prediction quality due to the increase in the camera interval, a method of using a different camera for each camera as a reference frame can be easily considered.

  However, in the conventional multi-view video coding method, since it is necessary to encode the distance to the subject with respect to the reference frame, when using different reference frames for each camera, multiple distance information is encoded. Need arises. For this reason, encoding efficiency will fall.

  The present invention has been made in view of such circumstances, and even when a different reference camera is used for each camera in multi-view video encoding, by generating distance information from known distance information and video signals, An object of the present invention is to realize efficient parallax compensation without encoding distance information of a plurality of cameras, thereby achieving higher encoding efficiency than before.

  In order to solve the above-mentioned problem, in the present invention, when a frame shot from a viewpoint different from the viewpoint given distance information is used as a reference frame, as a process in encoding and decoding, The given distance information is converted into distance information for another viewpoint. As a result, even when video (images) taken by various cameras is used as a reference frame, there is no need to transmit multiple distance information, and a camera with a similar video signal is used as a reference destination for each camera. Will be able to. That is, the prediction efficiency can be improved without changing the code amount, and as a result, efficient encoding can be realized.

  Since this conversion is performed according to the physical phenomenon when an object in real space is photographed by the camera, if the projection conversion by the camera can be sufficiently modeled, the conversion can be realized with very high accuracy. it can.

  In addition, there is a region where only one of the images captured by different cameras is captured. The above conversion is to convert distance information given to an image taken with a certain camera into distance information about an image taken with a different viewpoint. Therefore, effective distance information cannot be given to an area not photographed by the camera before conversion. Therefore, in the present invention, in addition to the above conversion, a means for generating distance information for such a region is provided.

  Specifically, the distance information is generated assuming that the continuous pixels on the conversion destination image have the distance from the same camera to the subject. Since this is a process based on the fact that the subject exists continuously in the real space, it is possible to generate distance information with high accuracy. In addition, since continuity does not hold when the subjects are different, if there are a plurality of subjects in an area where distance information is to be generated, this means may reduce the accuracy of distance information generation. Therefore, the subject is extracted by performing edge extraction and area division using color information from the conversion destination image, and distance information for the unknown region is generated using distance information given on the same subject nearby. Thus, it is possible to generate distance information with higher accuracy. In the case of a landscape that exists at a sufficiently long distance from the camera, even different subjects can be generated as having the same distance information. This is because sampling when taken by the camera is discrete.

  In addition, when distance information about the conversion destination camera at another time is obtained, it is possible to generate distance information of an unknown area at the current time using the distance information. Since this means is based on the fact that the subject cannot move instantaneously in real space, it is possible to generate highly accurate distance information. As the simplest method, there is means for copying distance information of the same pixel position. This is effective when the positional relationship between the subject and the camera does not change. For example, when a fixed camera is used and shooting is performed using a background with little change in time. In addition, even if the subject has a temporal change, the subject's movement is extracted using the image at the time when the distance information is known and the image at the current time, and the unknown region is extracted using the distance information of the corresponding pixel. It is possible to generate highly accurate distance information by generating the distance information. In addition, when finding a corresponding point using an image signal, it is possible to generate distance information with higher accuracy by using a factorization method even when the camera includes motion.

  Note that the distance information generating means using the continuity in the spatial direction introduced here and the distance information generating means using the continuity in the time direction may be used in combination. By using the continuity of two different axes, it is possible to generate distance information with higher accuracy.

  When correcting the distance information when generating or accumulating the distance information, using the condition that the subject in the area where the distance information was unknown cannot be viewed from the conversion source camera, the incorrect distance information May be avoided. In other words, when the generated distance information is first converted into distance information for the camera to which the distance information is given, if the distance information before the conversion exists at the edge of the image, the corresponding points on the converted image are displayed on the screen. If the distance information before the conversion exists in the occlusion area, the converted pixel exists on the input distance information indicating that the pixel is closer to the camera.

  The distance information generated by the above processing may be used as the past (future) distance information used when generating the distance information in consideration of continuity in time. Is obtained, the distance information can be estimated using stereo matching or the like and updated to a more accurate one.

  For example, if there is a camera (camera 1) to which distance information is given, a camera (camera 2) that captures a video as a reference frame, and a camera (camera 3) that captures a video to be encoded, encoding is performed. Since only the images of the camera 1 and the camera 2 are obtained at a certain time, the distance information for the camera 2 for the area not captured by the camera 1 is the spatial and temporal continuity of the subject. It can only be generated using However, a decoded image of the camera 3 is obtained in addition to the images of the cameras 1 and 2 for the past time in the encoding order. Therefore, even if the image is not captured by the camera 1, if the image is captured by the camera 3, the corresponding points can be calculated using stereo matching with the image captured by the camera 2, so continuity is assumed. Thus, it is possible to generate more accurate information than the distance information generated in this way.

  According to the present invention, it is possible to realize parallax compensation using a different reference camera for each camera while preventing a large increase in information necessary for parallax compensation. Highly efficient encoding can be realized.

  Hereinafter, the present invention will be described in detail according to embodiments. In the embodiment described below, it is assumed that a multi-view video captured by three cameras is encoded. Here, it is assumed that the video of the camera A is encoded without performing parallax compensation, and information regarding the distance from the camera to the subject photographed is encoded in addition to the video. In addition, the video of the camera B is encoded using the encoded distance information of the camera A while using the camera A as a reference camera and using parallax compensation. The camera C is encoded using the encoded distance information of the camera A using the camera B as a reference camera and using parallax compensation. Since the conventional method can be used as it is for the encoding of the camera A and the camera B, a method for encoding the video of the camera C will be described in the present embodiment. FIG. 3 shows a conceptual diagram of a camera configuration used in this embodiment.

  First, the first embodiment (hereinafter referred to as the first embodiment) will be described. FIG. 4 shows a configuration diagram of a video encoding apparatus according to Embodiment 1 of the present invention. As illustrated in FIG. 4, the video encoding device 100 according to the first embodiment stores an encoding target image input unit 101 that inputs an original image of a camera C to be encoded, and an input encoding target image. An encoding target image memory 102, a reference camera image input unit 103 that inputs a decoded image of the camera B, which is a reference image for parallax compensation, a reference image memory 104 that stores an input reference image, and parallax compensation A distance information input unit 105 that inputs information about the distance from the camera to the subject in the camera A used when generating an image, and a distance information conversion unit that converts the distance information about the camera A into distance information about the camera B that is a reference camera. 106, a background distance information generation unit 107 that generates background distance information for an area for which effective distance information cannot be obtained due to occlusion, etc. A distance information memory 108 for storing distance information, a parallax compensation image generation unit 109 for generating a parallax compensation image for the encoding target image from the reference image and the distance information, and image encoding for actually encoding the encoding target image Unit 110 and a decoded image memory 111 for storing an image obtained by decoding an encoded image.

  FIG. 5 shows a processing flow executed by the video encoding apparatus 100 configured as described above. The processing executed by the video encoding device 100 according to the first embodiment will be described in detail according to this processing flow.

  First, the image of the camera C to be encoded is input by the encoding target image input unit 101 and stored in the encoding target image memory 102 [step S11]. In addition, an image obtained by encoding and decoding an image shot at the same time as an encoding target image of the camera B serving as a reference camera is input from the reference camera image input unit 103 and stored in the reference image memory 104. Further, information regarding the distance from the camera to the subject in the camera A at the same time as the encoding target image is input from the distance information input unit 105. Here, it is not necessary to input all the information at the same time. The information may be input before the encoding target image is input, or may be input after the encoding target image is input. In the latter case, the processing after step S14 for generating the parallax compensation image is not performed until all the information at the same time as the encoding target image is obtained.

  The distance information input here does not need to be information that accurately represents the distance from the camera to the subject, and may be any information that can calculate the parallax between the cameras with this information. Also, if the same information as obtained on the decoding device side is used, it is not necessary to generate coding distortion called drift in the decoded image. Therefore, if the distance information is encoded with distortion and transmitted to the decoding device side, the distance information obtained as a result of decoding the encoded distance information may be used. Of course, if drift distortion is allowed, the distance information input here may be different from that obtained on the decoding device side.

  The input distance information is converted from distance information for the camera A to distance information for the camera B by the distance information conversion unit 106 [step S12]. In the simplest conversion method, conversion is performed according to the following calculation formula (1).

Here, A, R, and t represent the internal parameter matrix, rotation matrix, and translation vector of the camera, respectively. The subscript indicates which camera information the parameter is. A and R are 3 × 3 matrices, and t is a three-dimensional vector. Since the camera parameters can be expressed by various methods, it is assumed that the above calculation formula uses an expression in which the correspondence relationship between the image coordinate m and the world coordinate M is obtained by the formula M ^* = RA ⁻¹ m ^* + t. Note that the symbol * attached to the right shoulders of M and m represents homogeneous coordinates that allow scalar multiplication. d _x (a, b) represents the distance from the camera to the subject in the pixel (a, b) of the image taken by the camera X.

  In the conversion, after initializing the distance information buffer of the camera B, the pixel position and distance information in the camera B are calculated for all the pixels of the camera A using the expression (1), and the distance information corresponding to the pixel position is calculated. This is done by storing the obtained distance information in the buffer. When a value other than the initial value is already stored in the corresponding area when storing, distance information indicating that it is closer to the camera B is selected and stored according to the context of the camera indicated by the distance information.

  In this conversion, since a plurality of pixels of the camera A are associated with one pixel of the camera B, a large number of pixels for which effective distance information cannot be obtained may occur. However, distance information cannot be obtained in areas other than those actually captured by camera A but not captured by camera B. Therefore, a method for obtaining conversion distance information in the camera B in consideration of continuity between pixels in the camera A may be used. However, in order to eliminate the occurrence of drift distortion, the distance information conversion method here and the distance information conversion method performed on the decoding side need to be the same.

  As a conversion method considering continuity, for example, it is assumed that the subject is continuous in the real space if the distance information is within a certain range with respect to adjacent pixels on the image of camera A. Then, after obtaining the corresponding pixels on the image of the camera B, a method of converting the obtained pixels into distance information in the camera B while interpolating between the obtained pixels using the distance information given to the respective pixels is provided. is there.

  Here, since the camera A and the camera B are different cameras, there is an area that is captured by the camera B but not captured by the camera A. Therefore, effective distance information is not obtained in an area on the image of the camera B corresponding to such an area.

  Therefore, next, distance information in an area where there is no such effective value is generated [step S13]. When drift distortion is not generated, it is necessary to use the same method on the decoding side, but any method may be used.

  For example, since the subject is continuous in real space, there is a method of copying distance information of adjacent pixels. If the object in the area is a three-dimensional simple wall or a background that is far enough from the camera, the distance from the camera to the subject between adjacent pixels is almost equal. Information generation has high accuracy. Even when there are a plurality of objects in the area, the subject is discriminated by extracting an edge or dividing the area from the decoded image of the corresponding camera B stored in the reference image memory 104. , It is possible to realize more accurate generation of distance information by copying from adjacent pixels whose distance information is already known on the same subject.

  It is also possible to generate distance information using temporal correlation. When shooting an object that is stationary in the background or that has a sufficiently long camera distance, the distance information does not change with time, so the past distance information is stored in the distance information memory 108, and the distance information is stored. Distance information can be generated by copying a value stored in an unknown area. If the subject in the background is not sufficiently far from the camera and changes over time, the movement of the subject is obtained using the reference image at the corresponding time, and the distance stored in the distance information memory 108 More accurate distance information can be generated by translating and copying the information. Even in this case, since the subject in the image can be distinguished from the edge information and color information of the image, it is possible to generate highly reliable distance information by using such information.

  When the position or orientation of the camera B is changed, it is necessary to change the distance information on the distance information memory 108 accordingly. If the camera motion is explicitly given as a change in camera parameter, it is assumed that there is a camera B ′ having a past camera parameter and a camera B having a current camera parameter, and is given to the camera B ′. The obtained distance information may be converted into distance information for the camera B by the same method as in step S12. On the other hand, if camera motion is not given as a change in camera parameters, matching is performed between a reference image at a time for which distance information is desired and a reference image at a time for which distance information is known. The camera motion can be acquired while using it to separate the subject motion.

  As the past distance information stored in the distance information memory 108, the actually used distance information may be stored as it is, and stereo matching is performed between the reference image and the decoded image to make it more correct. You may use what was corrected so that it might become. Although not input in this embodiment, more accurate distance information may be obtained again using the decoded image of the camera A.

  In addition, when correcting the distance information when generating the distance information or when accumulating the distance information, an incorrect distance information is used under the condition that the subject in the area where the distance information is unknown cannot be viewed from the camera A. Can be prevented from being generated. That is, when the distance information for the camera B generated (corrected) is converted into the distance information for the camera A using the expression in which the camera A and the camera B in the expression (1) are interchanged, the image before conversion is the end of the image. In the case of the distance information, the corresponding pixel is obtained outside the image, and in the occlusion area, the distance information indicating that the converted pixel is closer to the camera must be present on the input distance information. By limiting the value of the distance information after generation (correction) using this condition, it is possible to prevent the generation of distance information that is erroneous to some extent.

  The generation of the distance information is completed, and the distance information for which valid values are obtained for all the pixels is stored in the distance information memory 108. If the encoding target image, the reference image, and the distance information at the same time are stored in the encoding target image memory 102, the reference image memory 104, and the distance information memory 108, respectively, the disparity compensation image generation unit is used using the information. A parallax compensation image is generated at 109 [step S14].

  Any method may be used as a method for generating a parallax compensation image as long as it is performed using distance information and an image signal. For example, according to the following formula (2) in which the camera A in the formula (1) is changed to the camera B and the camera B is changed to the camera C, the pixel of the camera C corresponding to the pixel of the camera B is determined. Can be generated as the pixel value of the parallax compensation image at the corresponding pixel position.

  The alphabet representing coordinates is changed for each camera, but the meanings of the other symbols are the same as in equation (1). Further, since the distance information of the camera B is not obtained by the equation (1) but is corrected by the background distance information generation unit 107, a dash (′) symbol is added to distinguish the information.

  As in Non-Patent Document 3, a parallax compensation image may be generated while assuming continuity between adjacent pixels. In this case, unlike the distance information conversion, the value interpolated between the corresponding pixels is the pixel value. Of course, once converted into the distance information in the camera C and the parallax information in each pixel of the image captured by the camera C is acquired, the parallax compensation image may be generated. There is also a method of generating a parallax compensation image by generating a three-dimensional polygon object from given distance information by regarding each pixel as a vertex and projecting it to the camera C.

  If the process for generating the parallax compensation image performed here does not cause drift distortion in the decoded image, it is necessary to use the same process as that on the decoding side.

  When the parallax compensation image is generated, the image to be encoded is encoded by the image encoding unit 110 [step S15]. The bit stream generated as a result of encoding is an output of the video encoding device 100. Note that any coding method can be used as long as the coding method uses a parallax compensation image.

  For example, the simplest method is a method in which the generated disparity compensation image is directly encoded as a predicted image candidate, but the disparity compensation image (composite image) is referred to the encoding target image as in Non-Patent Document 4. Using as an image, encoding while adding vector information indicating a prediction destination, or taking a difference between an encoding target image and a parallax compensation image as in Patent Document 1, and predictively encoding the difference image Thus, there is a method of encoding the encoding target image. In addition to video prediction using a parallax compensated image, motion compensation, parallax compensation using a vector, intra prediction, and the like may be used in combination.

  The encoded image to be encoded is decoded and stored in the decoded image memory 111 [step S16]. The accumulated decoded image is used to correct the distance information stored in the distance information memory 108 [step S17]. This correction is performed in order to predict more accurate information when generating distance information in another frame. For example, by performing stereo matching between a reference image that is an image of camera B and a decoded image that is an image of camera C, the image is generated assuming continuity in the process of step S13. The distance information is corrected to more reliable distance information based on the fact that the images were actually taken by a plurality of cameras. That is, a corresponding point search is performed between the reference camera image and the decoded image by a disparity search unit (not shown), and the distance information stored in the distance information memory 108 is corrected based on the obtained disparity information.

  In the present invention, distance information for an area for which distance information is not given as input is generated using distance information of spatially adjacent areas or distance information of temporally adjacent areas. This is based on the fact that the subject is continuous in real space and the fact that the subject cannot move instantaneously, so that it is possible to generate accurate distance information to some extent.

  Of course, since unknown information is generated, it may be very difficult to generate distance information at the edge of the image. In the worst case, something far from the actual distance information is generated. However, if the image encoding unit has multiple prediction modes and encodes while selecting the optimum one as in a general video encoding device, encoding is performed even if incorrect distance information is generated. It does not reduce efficiency. This is because in a region where the distance information is unknown, efficient prediction cannot be performed in the prediction mode using the parallax compensation image. Therefore, in the conventional method, another mode is selected and encoding is performed. That is, even if the distance information is incorrect, only a different mode is selected as in the conventional method, and the information to be encoded does not increase.

  Next, a second example (hereinafter referred to as Example 2) will be described. FIG. 6 shows a configuration diagram of a video decoding apparatus according to Embodiment 2 of the present invention. As illustrated in FIG. 6, the video decoding device 200 according to the second embodiment includes an encoded data input unit 201 that inputs encoded data of the camera C to be decoded, and encoded data that stores the input encoded data. A memory 202, a reference camera image input unit 203 that inputs a decoded image of the camera B, which is a reference image for parallax compensation, a reference image memory 204 that stores the input reference image, and a parallax compensation image are generated. A distance information input unit 205 that inputs information about the distance from the camera to the subject in the camera A used at the time, a distance information conversion unit 206 that converts the distance information about the camera A into distance information about the camera B as a reference camera, and an occlusion A background distance information generation unit 207 that generates background distance information for an area for which effective distance information cannot be obtained, and the like. A distance information memory 208 for storing information, a parallax compensation image generation unit 209 for generating a parallax compensation image for the decoding target image from the reference image and the distance information, and an image decoding unit for actually decoding the decoding target image from the encoded data 210 and a decoded image memory 211 for storing the decoded image.

  FIG. 7 shows a processing flow executed by the video decoding apparatus 200 configured as described above. Processing executed by the video decoding apparatus 200 according to the second embodiment will be described according to this processing flow.

  First, encoded data obtained by encoding an image of the camera C to be decoded is input by the encoded data input unit 201 and stored in the encoded data memory 202 [step S21]. Further, the decoded image of the camera B serving as the reference camera is input from the reference camera image input unit 203 and stored in the reference image memory 204. Further, information regarding the distance from the camera to the subject in the camera A is input from the distance information input unit 205. Note that it is not necessary to input all the information at the same time, it may be input before the encoded data of the decoding target image is input, or may be input after the encoded data of the decoding target image is input. It does not matter. However, the processing in steps S23 and S24 shown below is not executed until the distance information and the reference image at the same time are aligned, and the processing in steps S25 and S26 is performed until the encoded data, the reference image and the distance image at the same time are aligned. Not executed.

  As in the first embodiment, the distance information input here does not need to be information that accurately represents the distance from the camera to the subject, and the accuracy with which the parallax between the cameras is obtained using this information. Anything that can be calculated with However, in order to prevent the occurrence of drift distortion, the same distance information as that used in the video encoding device must be input.

  The input distance information is converted from distance information for the camera A to distance information for the camera B by the distance information conversion unit 206 [step S22]. As the simplest method, conversion can be performed according to the above-described calculation formula (1), and the detailed procedure in this case is already described in the processing part of step S12 in the first embodiment. Note that any method may be used for the conversion, but in order to suppress the occurrence of drift distortion, it is necessary to use the same method as that used in the video encoding apparatus.

  Next, distance information in a region on the reference image having no valid value is generated in the converted distance information [step S23]. Again, in order not to generate drift distortion, it is necessary to use the same method as the video encoding device, but any method may be used.

  For example, since the subject is continuous in the real space, the distance information is considered to have a spatial correlation. Therefore, a method of copying and generating adjacent known distance information, or filtering from neighboring distance information There is a method of generating by processing. There is also a method of estimating from the distance information of the proximity time stored in the distance information memory 208 because the subject does not move instantaneously. In this case, there are a method of copying distance information at the same position assuming that the subject does not move, and a method of extracting the motion of the subject from the reference image stored in the reference image memory 204 and generating the distance information by motion compensation. . As described in the first embodiment, when the position and orientation of the camera B are changed, it is necessary to change the distance information on the distance information memory 208 accordingly.

  Note that the past distance information stored in the distance information memory 208 may store the actually used distance information as it is, and as described in the present embodiment, between the reference image and the decoded image. You can use stereo matching that has been modified to be more correct. Although not input in this embodiment, more accurate distance information may be obtained again using the decoded image of the camera A. However, in order not to cause drift distortion, it is necessary to obtain distance information to be stored using the same processing as that performed by the video encoding device.

  The generation of the distance information is completed, and the distance information for which valid values are obtained for all the pixels is once stored in the distance information memory 208. If the encoded data of the decoding target image, the reference image at the same time, and the distance information are stored in the encoded data memory 202, the reference image memory 204, and the distance information memory 208, respectively, the disparity compensation is performed using these information. The image generation unit 209 generates a parallax compensation image [step S24].

  As in the first embodiment, any method may be used as a method for generating a parallax compensation image. However, in order to prevent the occurrence of drift distortion, it is necessary to use the same method as that used in the video encoding device.

  When the parallax compensated image is generated, the encoded data is decoded by the image decoding unit 210 using this image, and a decoded image is obtained [step S25]. The decoded image is stored in the decoded image memory 211 and is output from the video decoding device 200. In this embodiment, the images are output in the order of decoding. However, buffering may be performed in the decoded image memory 211 and the images may be output in the order of shooting. Here, it is necessary to use a method that can correctly decode the encoded data generated by using the encoding method used in the video encoding device.

  The accumulated decoded image is used to correct the distance information stored in the distance information memory 208 [step S26]. This correction is performed in order to predict more accurate information when generating distance information in another frame. For example, by performing stereo matching between a reference image that is an image of camera B and a decoded image that is an image of camera C, the image is generated assuming continuity in the process of step S23. The distance information is corrected to more reliable distance information based on the fact that the images were actually taken by a plurality of cameras. That is, a corresponding point search is performed between the reference camera image and the decoded image by a parallax search unit (not shown), and the distance information stored in the distance information memory 208 is corrected based on the obtained parallax information. In this case, the same processing as that of the video encoding device needs to be used in order to prevent drift distortion.

  In the embodiment described here, the video encoding device and the video decoding device have been described. However, the present invention can also be applied to encoding and decoding of multi-viewpoint images taken by a plurality of cameras. In that case, the continuity in the time direction cannot be used, and only the continuity in the spatial direction is used to generate the distance information of the area for which effective distance information cannot be obtained. Also, a process for generating a disparity compensation image using distance information conversion / distance information generation in an invalid area is extracted, and other information is obtained from the input video (camera A video in the above embodiment) and distance information for the input video. It is also possible to use for the process which synthesize | combines the image | video (camera B image | video in the said Example) image | photographed with this camera. Even in this case, by updating the generated distance information for generating the distance information at another time (corresponding to the processing steps S17 and S26 in the above embodiment), higher quality video composition can be performed. .

  The processing described above can be realized by a computer and a software program, and the program can be provided by being recorded on a computer-readable recording medium or can be provided through a network.

  In the above embodiments, the video encoding device and the video decoding device have been mainly described, but the video encoding method and the video encoding method according to the present invention are performed according to the steps corresponding to the operations of the respective units of the video encoding device and the video decoding device. A video decoding method can be realized.

  The characteristics of the video encoding method and video decoding method according to this embodiment are listed as follows.

  (1) In the video encoding method according to the present embodiment, an encoding target image in a multi-view image or multi-view video captured by a plurality of cameras including one reference viewpoint camera is transmitted from the reference viewpoint camera to the subject. A video encoding method that encodes by performing image prediction between cameras using distance information representing the distance of the image, and is used for prediction of an image signal between cameras when encoding an encoding target image. A reference camera image setting step for setting a reference camera image obtained by decoding an encoded camera image, and the reference viewpoint camera image and the reference camera from the reference information for the reference viewpoint camera image captured by the reference viewpoint camera. Based on the correspondence with the image, the distance information variable is converted into distance information representing the distance from the camera to the subject in the reference camera image. A parallax-compensated image generating step for generating a parallax-compensated image for the encoding target image from the converted distance information and the reference camera image, image prediction is performed using the parallax-compensated image, and the code And an image encoding step for encoding the encoding target image.

  (2) In the video encoding method of (1), distance information is obtained for a region of the reference camera image for which a correspondence relationship with the reference viewpoint camera image has not been obtained when the distance information is converted. A distance information correction step for correcting the converted distance information by generating the distance information, wherein the corrected distance information is used in the parallax compensation image generation step.

  (3) In the video encoding method of (2) above, in the distance information correction step, the distance information of the area for which the correspondence relationship is not obtained is converted into continuity with the distance information in the peripheral area of the area. It is assumed that it is generated based on distance information in the peripheral area of the area.

  (4) In the video encoding method of (2), an area dividing step of detecting an area where the same subject is photographed in the reference camera image and dividing the reference camera image by the detected area. And the distance information correcting step generates distance information on the assumption that continuity with distance information in surrounding areas belonging to the same division is assumed in the area dividing step.

  (5) Further, in the video encoding method of (2), there is a distance information storage step for storing the distance information used in the parallax compensation image generation step, and the distance information correction step stores the distance information. The distance information is generated using the distance information.

  (6) In the video encoding method of (2), a distance information storage step for storing distance information used in the parallax compensation image generation step, and a reference camera image used in the parallax compensation image generation step. A reference camera image accumulation step for accumulating, and a motion search step for searching for corresponding points between the accumulated reference camera image and the reference camera image. In the distance information correction step, the corresponding point search is performed. Using the temporal movement information of the subject obtained in the step, distance information at the current time is generated from the accumulated distance information.

  (7) In the video encoding method of (5) or (6), an image decoding step for decoding the image encoded in the image encoding step, the decoded image, the reference camera image, A parallax search step for searching for corresponding points between, and a distance information correction step for correcting the distance information used in the parallax compensation image generation step from the parallax information obtained in the parallax search step, In the distance information storage step, the distance information corrected in the distance information correction step is stored.

  (8) In the video encoding method of any one of (2) to (7), in the distance information correction step, when the generated distance information is converted into distance information for the reference viewpoint camera, The generation is performed so as not to be effective distance information.

  (9) In the video encoding method according to any one of (1) to (8), in the image encoding step, the encoding target image is encoded using the parallax compensation image as a predicted image candidate. It is characterized by becoming.

  (10) In the video encoding method according to any one of (1) to (8), in the image encoding step, encoding is performed using the parallax compensation image as a reference image candidate used for video prediction. The target image is encoded.

  (11) In the video encoding method according to any one of (1) to (8), in the image encoding step, a difference image between the parallax compensation image and the encoding target image is generated, and the difference image The encoding target image is encoded by encoding.

  (12) In the video decoding method according to the present embodiment, encoded data obtained by encoding a multi-view image or a multi-view video captured by a plurality of cameras including one reference viewpoint camera is transmitted from the reference viewpoint camera to the subject. This is a video decoding method that decodes by performing image prediction between cameras using distance information that represents the distance up to, and was used to predict the image signal between cameras when encoding the decoding target image. Correspondence relationship between the reference viewpoint camera image and the reference camera image from the reference camera image setting step for setting the reference camera image that has already been decoded and the reference distance camera image captured by the reference viewpoint camera with the distance information. A distance information conversion step for converting the reference camera image into distance information representing the distance from the camera to the subject based on A disparity compensation image generation step for generating a disparity compensation image for the decoding target image from the separation information and the reference camera image, and an image for performing image prediction using the disparity compensation image and decoding the decoding target image from the encoded data And a decoding step.

  (13) Also, in the video decoding method of (12), distance information is applied to an area of the reference camera image for which a correspondence relationship with the reference viewpoint camera image has not been obtained when the distance information is converted. It has a distance information correction step for correcting the converted distance information by generating, and the corrected distance information is used in the parallax compensation image generation step.

  (14) Also, in the video decoding method of (13), in the distance information correction step, the distance information of the area for which the correspondence relationship has not been obtained is assumed to be continuous with the distance information in the peripheral area of the area. Then, it is generated based on the distance information in the peripheral area of the area.

  (15) The video decoding method according to (13) further includes a region dividing step of detecting a region where the same subject is photographed in the reference camera image and dividing the reference camera image by the detected region. In the distance information correction step, the distance information is generated by assuming continuity with the distance information in the peripheral area belonging to the same division in the area division step.

  (16) The video decoding method of (13) further includes a distance information storage step for storing distance information used in the parallax compensation image generation step, and the distance information stored in the distance information correction step is stored in the distance information storage step. The distance information is generated using the information.

  (17) In the video decoding method of (13), a distance information accumulation step for accumulating distance information used in the parallax compensation image generation step, and a reference camera image used in the parallax compensation image generation step. A reference camera image storage step for storing; and a motion search step for searching for a corresponding point between the stored reference camera image and the reference camera image. In the distance information correcting step, the corresponding point searching step The distance information at the current time is generated from the accumulated distance information using the temporal movement information of the subject obtained in the above.

  (18) In the video decoding method of (16) or (17), a parallax search step for searching for corresponding points between the image decoded in the image decoding step and the reference camera image, and the parallax search step A distance information correction step for correcting the distance information used in the parallax compensation image generation step from the obtained parallax information, and the distance information correction step stores the distance information corrected in the distance information correction step. It is characterized by doing.

  (19) In any one of the video decoding methods of (13) to (18), in the distance information correction step, an effective distance when the generated distance information is converted into distance information for the reference viewpoint camera. It is characterized by generating so as not to become information.

  (20) In any one of the video decoding methods according to (12) to (19), in the image decoding step, the decoding target image is decoded from the encoded data while using the parallax compensation image as a predicted image candidate. It is characterized by that.

  (21) In the video decoding method according to any one of (12) to (19), in the image decoding step, a decoding target is encoded from encoded data while using the parallax compensated image as a reference image candidate used for video prediction. It is characterized by decoding an image.

  (22) In any one of the video decoding methods of (12) to (19), in the image decoding step, a difference image between the parallax compensation image and a decoding target image is decoded from encoded data, The decoding target image is decoded by adding the parallax compensation images.

  The effects of this embodiment will be described in comparison with the prior art.

  The conventional technique (conventional technique 1) that encodes information necessary for parallax compensation with respect to an encoding target image as described in Non-Patent Document 2 has a reference camera that is optimal for prediction for each encoding target image. While there is an advantage that it can be selected, there is a disadvantage that the amount of additional information (the amount of information necessary for parallax compensation) becomes enormous.

  Further, the conventional technique (conventional technique 2) in which a reference camera as described in Patent Document 1 is shared and distance information necessary for parallax compensation is encoded with respect to a reference image has a very small amount of additional information. However, since a reference camera optimal for prediction cannot be selected for each image to be encoded, there is a drawback in that the video prediction capability by parallax compensation decreases when the camera interval increases.

In contrast, the present embodiment has the following effects.
[Effect 1] With the same amount of additional information as in the prior art 2, it is possible to perform video prediction by parallax compensation by selecting a reference camera optimal for prediction for each encoding target image.
[Effect 2] Without increasing the amount of additional information, it is possible to provide video prediction based on parallax compensation in an occlusion region that could not be predicted at all by the conventional method.
[Effect 3] It becomes possible to generate distance information (information necessary for parallax compensation) having high reliability with respect to accuracy.

  The reason why the effect 1 is obtained is that the present embodiment includes means for converting distance information input during encoding and decoding into distance information for the reference camera. The principle used when the camera captures an object and projects it in two dimensions is used. This is processing performed in step S12 in FIG. 5 and step S22 in FIG. For example, conversion processing using equation (1) or the like corresponds to this processing.

  The reason why the above effect 2 can be obtained is that, in the present embodiment, when distance information is converted, it is necessary to perform parallax compensation in an occlusion area or an image edge area where effective distance information cannot be obtained. This is for providing means for generating distance information. There are areas where only one of the images captured by different cameras is captured. In the conversion of distance information, distance information given to an image shot with a camera is converted into distance information for an image shot with a different viewpoint. Effective distance information cannot be given to an area that is not photographed by the camera before conversion. In such a case, in the present embodiment, the distance information is generated based on the fact that the object is spatially continuous in the real space and cannot be moved or changed instantaneously in time. Therefore, a parallax compensation image can be generated using the generated distance information. This process is performed in step S13 in FIG. 5 and step S23 in FIG. In general, distance information is higher in spatial and temporal continuity than image signal, so continuity of distance information is assumed rather than generating an image signal assuming continuity on the generated parallax compensation image. It is possible to generate a high-quality predicted image by performing generation and performing parallax compensation using the generated image.

  The reason why the third effect is obtained is that, in the present embodiment, the distance information generated after encoding is updated to distance information indicating an accurate corresponding point using a reference image and a local decoded (decoded) image. It is for providing a means. The generated distance information is accurate to some extent but contains errors. Since the same local decoding image can be obtained for both encoding and decoding, updating the distance information using this enables accurate distance information to be used in frames that are encoded later in the encoding order. It becomes. For the update, a stereo matching method for estimating distance information from an image signal can be used. This is processing performed in step S17 of FIG. 5 and step S26 of FIG.

  The embodiments of the present invention have been described above with reference to the drawings. However, the above embodiments are merely examples of the present invention, and it is clear that the present invention is not limited to the above embodiments. . Accordingly, additions, omissions, substitutions, and other modifications of the components may be made without departing from the spirit and scope of the present invention.

It is a conceptual diagram of the parallax which arises between cameras. It is a conceptual diagram of epipolar geometric constraint. It is a conceptual diagram of the camera structural example in an Example. It is a figure which shows the video coding apparatus of Example 1 of this invention. 4 is a video encoding flowchart according to the first exemplary embodiment. It is a figure which shows the video decoding apparatus of Example 2 of this invention. 10 is a video decoding flowchart according to the second embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Video coding apparatus 101 Encoding object image input part 102 Encoding object image memory 103 Reference camera image input part 104 Reference image memory 105 Distance information input part 106 Distance information conversion part 107 Background distance information generation part 108 Distance information memory 109 Parallax Compensation image generation unit 110 Image encoding unit 111 Decoded image memory 200 Video decoding device 201 Encoded data input unit 202 Encoded data memory 203 Reference camera image input unit 204 Reference image memory 205 Distance information input unit 206 Distance information conversion unit 207 Background Distance information generation unit 208 Distance information memory 209 Parallax compensated image generation unit 210 Image decoding unit 211 Decoded image memory

Claims (24)

The encoding target image in the multi-view image or a multi-viewpoint video images taken by the plurality of cameras, including one reference view camera, using the distance information representing the distance from the reference view camera to the object, between the camera A video encoding method for encoding by performing image prediction,
Reference camera image setting for setting a reference camera image obtained by decoding a camera image different from the camera image of the reference viewpoint camera, which has already been encoded, and is used to predict an image signal between cameras when encoding an encoding target image Steps,
Represents the distance from the camera to the subject in the reference camera image based on the correspondence between the reference viewpoint camera image and the reference camera image from the distance information for the reference viewpoint camera image captured by the reference viewpoint camera A distance information conversion step for converting into distance information;
A disparity compensation image generation step for generating a disparity compensation image for the encoding target image from the converted distance information and the reference camera image;
A video encoding method comprising: an image encoding step of performing image prediction using the parallax compensation image and encoding the encoding target image. The video encoding method according to claim 1, wherein
Distance information for correcting the converted distance information by generating distance information for the region of the reference camera image for which the correspondence with the reference viewpoint camera image was not obtained during the conversion of the distance information A correction step,
In the parallax compensation image generation step, the corrected distance information is used. The video encoding method according to claim 2,
In the distance information correction step, the distance information of the area where the correspondence relationship is not obtained is assumed to be continuous with the distance information in the surrounding area of the area, and based on the distance information in the surrounding area of the area. A video encoding method comprising generating the video encoding method. The video encoding method according to claim 2,
A distance information storage step for storing the distance information used in the parallax compensation image generation step;
In the distance information correction step, the distance information is generated using the accumulated distance information. The video encoding method according to claim 4, wherein
An image decoding step for decoding the image encoded in the image encoding step;
A parallax search step of searching for corresponding points between the decoded image and the reference camera image;
A distance information correction step for correcting the distance information used in the parallax compensation image generation step from the parallax information obtained in the parallax search step;
In the distance information storage step, the distance information corrected in the distance information correction step is stored. The encoded data obtained by encoding a multi-view image or a multi-view video captured by a plurality of cameras including one reference viewpoint camera is used as distance information representing the distance from the reference viewpoint camera to the subject. A video decoding method for decoding by performing image prediction,
A reference camera image setting step for setting a reference camera image different from the camera image of the base viewpoint camera that has already been decoded and used for prediction of an image signal between cameras when encoding a decoding target image;
Represents the distance from the camera to the subject in the reference camera image based on the correspondence between the reference viewpoint camera image and the reference camera image from the distance information for the reference viewpoint camera image captured by the reference viewpoint camera A distance information conversion step for converting into distance information;
A disparity compensation image generation step for generating a disparity compensation image for the decoding target image from the converted distance information and the reference camera image;
A video decoding method comprising: an image decoding step of performing image prediction using the parallax-compensated image and decoding a decoding target image from encoded data. The video decoding method according to claim 6,
Distance information for correcting the converted distance information by generating distance information for the region of the reference camera image for which the correspondence with the reference viewpoint camera image was not obtained during the conversion of the distance information A correction step,
In the parallax compensation image generation step, the corrected distance information is used. The video decoding method according to claim 7,
In the distance information correction step, the distance information of the area where the correspondence relationship is not obtained is assumed to be continuous with the distance information in the surrounding area of the area, and based on the distance information in the surrounding area of the area. And a video decoding method. The video decoding method according to claim 7,
A distance information storage step for storing the distance information used in the parallax compensation image generation step;
In the distance information correcting step, the distance information is generated using the accumulated distance information. The video decoding method according to claim 9, wherein
A parallax search step for searching for corresponding points between the image decoded in the image decoding step and the reference camera image;
A distance information correction step for correcting the distance information used in the parallax compensation image generation step from the parallax information obtained in the parallax search step;
In the distance information storage step, the distance information corrected in the distance information correction step is stored. A multi-view image captured by a plurality of cameras including one reference viewpoint camera or an encoding target image in a multi-view moving image is obtained by using distance information representing the distance from the reference viewpoint camera to the subject. A video encoding device for encoding by performing prediction,
Reference camera image setting for setting a reference camera image obtained by decoding a camera image different from the camera image of the reference viewpoint camera, which has already been encoded, and is used to predict an image signal between cameras when encoding an encoding target image Means,
Represents the distance from the camera to the subject in the reference camera image based on the correspondence between the reference viewpoint camera image and the reference camera image from the distance information for the reference viewpoint camera image captured by the reference viewpoint camera Distance information converting means for converting into distance information;
A disparity compensation image generating means for generating a disparity compensation image for the encoding target image from the converted distance information and the reference camera image;
An image encoding apparatus comprising: an image encoding unit that performs image prediction using the parallax-compensated image and encodes the encoding target image. The video encoding device according to claim 11, wherein
Distance information for correcting the converted distance information by generating distance information for the region of the reference camera image for which the correspondence with the reference viewpoint camera image was not obtained during the conversion of the distance information With correction means,
The video encoding apparatus, wherein the parallax compensation image generating means uses the corrected distance information. The video encoding device according to claim 12,
The distance information correcting means is based on the distance information in the peripheral area of the area assuming the continuity of the distance information of the area where the correspondence relationship is not obtained with the distance information in the peripheral area of the area. A video encoding device characterized by generating. The video encoding device according to claim 12,
Distance information storage means for storing distance information used by the parallax compensation image generation means;
The video encoding apparatus according to claim 1, wherein the distance information correcting means generates distance information using the accumulated distance information. The video encoding device according to claim 14, wherein
Image decoding means for decoding the image encoded by the image encoding means;
Disparity search means for searching for corresponding points between the decoded image and the reference camera image;
Distance information correction means for correcting distance information used by the parallax compensation image generation means from the parallax information obtained by the parallax search means;
The distance information storage means stores the distance information corrected by the distance information correction means. The encoded data obtained by encoding a multi-view image or a multi-view video captured by a plurality of cameras including one reference viewpoint camera is used as distance information representing the distance from the reference viewpoint camera to the subject. A video decoding device that decodes by performing image prediction of
Reference camera image setting means for setting a reference camera image different from the camera image of the base viewpoint camera that has already been decoded and used for prediction of an image signal between cameras when encoding a decoding target image;
Represents the distance from the camera to the subject in the reference camera image based on the correspondence between the reference viewpoint camera image and the reference camera image from the distance information for the reference viewpoint camera image captured by the reference viewpoint camera Distance information converting means for converting into distance information;
A disparity compensation image generating means for generating a disparity compensation image for the decoding target image from the converted distance information and the reference camera image;
A video decoding apparatus comprising: an image decoding unit that performs image prediction using the parallax-compensated image and decodes a decoding target image from encoded data. The video decoding device according to claim 16, wherein
Distance information for correcting the converted distance information by generating distance information for the region of the reference camera image for which the correspondence with the reference viewpoint camera image was not obtained during the conversion of the distance information With correction means,
The video decoding device, wherein the parallax compensation image generation means uses the corrected distance information. The video decoding device according to claim 17,
The distance information correcting means is based on the distance information in the peripheral area of the area assuming the continuity of the distance information of the area where the correspondence relationship is not obtained with the distance information in the peripheral area of the area. A video decoding device characterized by generating the video decoding device. The video decoding device according to claim 17,
Distance information storage means for storing distance information used by the parallax compensation image generation means;
The distance information correction means generates distance information using the accumulated distance information. The video decoding device according to claim 19,
Disparity search means for searching for corresponding points between the image decoded by the image decoding means and the reference camera image;
Distance information correction means for correcting distance information used by the parallax compensation image generation means from the parallax information obtained by the parallax search means;
The distance information storage means stores the distance information corrected by the distance information correction means.   A video encoding program for causing a computer to execute the video encoding method according to any one of claims 1 to 5.   A computer-readable recording medium on which a video encoding program for causing a computer to execute the video encoding method according to any one of claims 1 to 5 is recorded.   A video decoding program for causing a computer to execute the video decoding method according to any one of claims 6 to 10.   A computer-readable recording medium on which a video decoding program for causing a computer to execute the video decoding method according to any one of claims 6 to 10 is recorded.

Images