WO2007129840A1

WO2007129840A1 - Method and apparatus for encoding multi-view moving pictures

Info

Publication number: WO2007129840A1
Application number: PCT/KR2007/002207
Authority: WO
Inventors: Suk-Hee Cho; Hyoung-Jin Kwon; Nam-Ho Hur; Jin-Woong Kim; Soo-In Lee
Original assignee: Electronics And Telecommunications Research Institute
Priority date: 2006-05-04
Filing date: 2007-05-04
Publication date: 2007-11-15
Also published as: EP2014096A1; KR20070108092A; KR101348276B1; EP2014096A4

Abstract

Provided are a method and an apparatus for improving resolution in comparison with lowering resolution of an original image and increasing a compression rate in comparison with directly compressing an original image based on information of the original image. The method includes the steps of subtracting a pixel value of a reference image from a pixel value of an image to be encoded to thereby create a residue signal, lowering resolution by down-sampling the residue signal, and encoding the down-sampled residue signal.

Description

DESCRIPTION

METHOD AND APPARATUS FOR ENCODING MULTI-VIEW MOVING

PICTURES

TECHNICAL FIELD

The present invention relates to a method and apparatus for encoding a multi-view moving picture; and, more particularly, to efficiently increasing a compression rate by down-sampling and encoding a multi- view moving picture without deteriorating a viewer' s sense of stereoscope.

BACKGROUND ART

It is required to increase a compression rate in order to provide a multi-view stereoscope contents service while using infrastructure of conventional transmission network. Generally, as the number of viewpoints increases by one, the quantity of data becomes double. Thus, encoding methods for reducing the quantity of data have been suggested. They are represented by a method using correlation between views. However, a coding gain due to correlation between views is not large due to an occlusion area which is generated by change of a region seen according to the location of a camera and difficulty in estimation of an actual disparity vector. An application service having an enough bandwidth may adopt an encoder using correlation between views. When the conventional transmission infrastructure whose bandwidth is limited is used, it is required to increase a compression rate in comparison with a conventional stereo and a multi-view encoder using only correlation between views .

According to a binocular suppression theory of a human visual system, when an image of different resolution is inputted to each eye, an image of a higher resolution affects more in finally recognizing a 3 dimension (3D) than an image of a relatively lower resolution. Therefore, it is not required to equally encode the resolution of left/right images. Although the right image is allocated in low bits, there is no difference in 3D effect of a stereo image. A simple method for increasing a compression rate of a right image includes a method of changing a quantization parameter

(QP) and roughing a quantization level in a quantization process, and a method of encoding an image by reducing the resolution of the image. The former is simple, but has a difficulty in finding a proper QP value since a QP value where a coding error is noticeably seen according to details of the image. Since the latter lowers resolution of an original image, there is a loss and information of the original image is not used in encoding.

Fig. 1 shows a principle of conventional stereo encoding. Referring to Fig. 1, when a left image performs encoding, the left image refers to the left image for backward compatibility with a conventional 2D moving picture service. Resolution of a right image is lowered in a horizontal direction by half through down- sampling. The right image refers to the right and left images for encoding. That is, an image of a viewpoint, which is a reference, is encoded based on correlation information on a time axis, and images of other viewpoints are encoded based on correlation information on time and space axes. As shown in Fig. 1, since the left image down-sampled according to the resolution of the right image should be referred to, a loss is generated from down-sampling in the referred left image.

Fig. 2 shows a conventional stereo encoding apparatus and realizes a concept of Fig. 1. A left image is encoded according to a conventional video encoding method such as H.2βx series and MPEG-x series. A right image is encoded according to the conventional encoding method in a time axis. A part for referring to the left image and predicting disparity is added in a space axis. As described above, when the right image is encoded by referring to the left image, a process for down-sampling the left image according to the resolution of the right image is added and a loss is generated from the process.

Since a recovery image referred to in motion and disparity prediction is also used after down-sampling when a right image whose resolution is lowered by performing down-sampling as a pre-process is used as an input in encoding of the conventional stereo moving pictures, a loss is generated from down-sampling in the original image and the recovery image.

DISCLOSURE TECHNICAL PROBLEM

An embodiment of the present invention is directed to efficiently increasing a compression rate by down- sampling and encoding a multi-view moving picture without deteriorating a viewer's sense of stereoscope.

Another embodiment of the present invention is directed to efficiently transmitting multi-view moving picture data to a limited transmission band. Another embodiment of the present invention is directed to preventing a loss from generating due to down-sampling by directly using resolution of a reference image used in motion and disparity prediction without reducing the resolution of the reference image. Other objects and advantages of the present invention can be understood by the following description, and become apparent with reference to the embodiments of the present invention. Also, it is obvious to those skilled in the art of the present invention that the objects and advantages of the present invention can be realized by the means as claimed and combinations thereof,

TECHNICAL SOLUTION In accordance with an aspect of the present invention, there is provided a method for encoding a multi-view moving picture, including the steps of subtracting a pixel value of a reference image from a pixel value of an image to be encoded to thereby create a residue signal, lowering resolution by down-sampling the residue signal, and encoding the down-sampled residue signal .

In accordance with another aspect of the present invention, there is provided an apparatus for encoding a multi-view moving picture, including a means for subtracting a pixel value of a reference image from a pixel value of an image to be encoded to thereby create a residue signal, a means for lowering resolution by down- sampling the residue signal, and a means for encoding the down-sampled residue signal.

In accordance with another aspect of the present invention, there is provided a method for decoding a multi-view moving picture including an encoded residue signal, includes the steps of decoding the encoded residue signal, recovering original resolution by up- sampling the decoded residue signal, and recovering a present image by summing a residue signal of recovered resolution and a previous image.

In accordance with another aspect of the present invention, there is provided an apparatus for decoding a multi-view moving picture including an encoded residue signal, includes a means for decoding the encoded residue signal, a means for recovering original resolution by up- sampling the decoded residue signal, a means for recovering a present image by summing a residue signal of recovered resolution and a previous image.

ADVANTAGEOUS EFFECTS

When an image size is down-sampled, the present invention performs encoding by lowering resolution of a residue signal, which is a difference value not between an original image and a recovery image, but between a reference image and a present image, after motion and disparity prediction. In disparity prediction, a block referred to in the left image does not have a loss caused by down-sampling since the original image is encoded without change of resolution. Therefore, in the block of the encoded right image, both of an original block and a reference block do not have a loss caused by down- sampling and only a loss caused by down-sampling of a residue signal showing a difference between pixel values of two blocks is generated. Accordingly, the present invention can efficiently increase a compression rate without changing a viewer' s sense of stereoscope and transmit multi-view moving picture data through a limited transmission band.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 describes a principle of conventional stereo encoding.

Fig. 2 shows a conventional stereo encoding apparatus .

Fig. 3 is a flowchart describing a multi-view moving picture encoding method according to the present invention.

Fig. 4 shows a multi-view moving picture encoding apparatus according to the present invention.

Fig. 5 is a flowchart showing a method for decoding encoded a multi-view moving picture including a residue signal and a motion vector or a disparity vector according to the present invention.

Fig. 6 shows a multi-view moving picture decoding apparatus according to the present invention.

BEST MODE FOR THE INVENTION

The preferred embodiments of the present invention will be described in detail hereinafter with reference to the attached drawings. The same reference numbers in drawings denote the same or similar constituent element or signal.

A motion vector or a disparity vector of an image to be currently encoded, i.e., a present image, is calculated with respect to a reference image at step S302.

Resolution is lowered by down-sampling a residue signal, which is a pixel value difference of a present image and a reference image and encoding is performed at step S304. The resolution of the residue signal is lowered in interframe prediction of an image of another viewpoint with respect to an image of the reference viewpoint after motion and disparity prediction. Encoding is performed through Discrete Cosine Transform (DCT), quantization, and entropy encoding. Presence of down-sampling or a rate is determined in a view of rate- distortion optimization. H.264 determines a coding mode of a block in a view of rate-distortion optimization. H.264 means acquiring a cost function value reflecting distortion, which shows a difference between an actually required bit and an original image, when coding is performed as a specific mode, and determining a mode of the smallest cost function value as a coding mode of the block. A cost function value reflecting distortion, which shows a difference between an actually required bit and an original image, when a present frame block is encoded for each down-sampling rate, is calculated by applying the H.264 in determination of the down-sampling rate. A sampling rate is determined when the cost function value is the smallest. For example, when 1 means compressing resolution of a residue signal without reducing the resolution of the residue signal and 1/2 means reducing the resolution of the residue signal by half in a horizontal direction. An original resolution is recovered by decoding and up-sampling the encoded residue signal at step S306. The present image is recovered at step S308 by summing a block of the recovered residue signal and a block of the reference image which is determined according to a motion vector or a disparity vector. Deblocking is performed in a block boundary of the recovered present image and a next reference image is stored at step S310. The present invention does not perform down-sampling on the reference image, but performs motion and disparity prediction directly onto an image of the same size. Therefore, since exact prediction is possible in the motion and disparity prediction in comparison with the conventional method shown in Fig. 2, a block to be encoded may bring a reference block that a motion or disparity-predicted location designates without loss.

When a residue signal created after motion and disparity prediction is encoded, the procedure is the same as a conventional stereo codec. When an image size is down-sampled in order to increase a compression rate as shown in Fig. 2, the present invention does not perform down-sampling on an original image or a recovery image, but performs down-sampling on a residue signal, which is a difference value between a reference block and a present block after motion and disparity prediction. In the conventional method shown in Fig. 2, since an image down-sampling a right image sequence as a pre- process is inputted in the video encoder, the original image is not used and the recovery image used when the left image is referred to is also used after down- sampling. Therefore, a loss caused by down-sampling in the original image and the recovery image is generated. However, since the present invention uses the reference image in the motion and disparity prediction, a loss caused by down-sampling in the original image and the recovery image is not generated. Compressibility is improved by applying encoding after down-sampling to the residue signal as shown in Fig. 2.

Therefore, the present invention has a merit as follows in comparison with the conventional method of Fig, 2. First, a loss caused by down-sampling is small. Second, although a size of a motion vector is two times as large as the vector of the conventional method, a similar quantity of bits are generated because the motion vector is encoded into Differential pulse code modulation (DPCM) . Since the residue signal is also encoded after down-sampling in both present and conventional methods, a similar quantity of bits are generated. Precisely, the present invention is better than the conventional method only in the motion and disparity prediction. That is, the present invention is not useful for an intra mode for prediction in the inside of the frame, but is useful an inter mode for prediction between frames. Since the intra mode of the present invention does not perform down-sampling, the present invention generally has larger quantity of bits in comparison with the conventional method.

When the decoder or the encoder creates a recovery image, the down-sampled and encoded residue signal performs up-sampling on the decoded residue signal into a residue signal of the original size and performs motion/displacement compensation by being added with the reference block that the motion and disparity vector designates. Therefore, calculation complexity for down- sampling and up-sampling is added to the present invention in comparison with the codec of the conventional method.

To take an example, when encoding is performed on the basis of a 16x16 macro block unit, motion prediction is performed based on the sizes of 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, and 4x4 in the inter frame prediction of conventional H.26x series and MPEG-x series encoding, and DCT is performed on the basis of 4x4. When prediction is performed as the 16x16 size, encoding is performed through DCT and quantization of 1 motion vector, a brightness component residue signal of 16 4x4 blocks, and a color difference component residue signal of 8 4x4 blocks. When prediction is performed as an 8x8 size, the motion vector becomes 4. However, the residue signal performs encoding through DCT and quantization of the brightness component residue signal of 16 4x4 blocks, and the color difference component residue signal of 8 4x4 blocks, just as the 16x16 size prediction.

However, in the inter frame prediction of the present invention, when the residue signal is down- sampled by half in a horizontal or vertical direction and prediction is performed as a 16x16 size, encoding is performed through DCT and quantization of 1 motion and disparity vector, a brightness component residue signal of 8 4x4 blocks, and a color difference component residue signal of 4 4x4 blocks. That is, the number of the motion and disparity vectors in the present invention is the same as the number of the motion and disparity vectors in the conventional encoding method. However, the number of the residue signals is reduced by half in the present invention. When prediction is performed as an 8x8 size, the motion and disparity vector becomes 4. However, the residue signal performs encoding through DCT and quantization of the brightness component residue signal of 16 4x4 blocks and the color difference component residue signal of 8 4x4 blocks, just as the 16x16 size prediction of the present invention.

When the residue signal is down-sampled by half in a horizontal and vertical direction, prediction is performed as a 16x16 size and encoding is performed through DCT and quantization of the brightness component residue signal of 1 motion vector and 4 4x4 blocks, and the color difference component residue signal of 2 4x4 blocks. That is, the number of the motion and disparity vectors in the present invention is the same as the number of the motion and disparity vectors in the conventional encoding method. However, the residue signals are reduced by a fourth.

Fig. 4 shows a multi-view moving picture encoding apparatus according to the present invention. Since the multi-view moving picture encoding apparatus additionally includes a down-sampling unit 408 for down-sampling the residue signal and a up-sampling unit 416 for up-sampling a recovered residue signal in right image encoding, a configuration of the multi-view moving picture encoding apparatus is different from the encoding apparatus of Fig. 2.

A left image sequence is encoded on a time axis through a video encoder 402. The recovered left image frame created in the video encoder 402 is stored in the reference frame buffer 404.

A motion/disparity predicting unit 422 performs motion or disparity prediction on a right image sequence according to kinds of images referred to in the reference frame buffer 404. A motion vector or a disparity vector created in the motion/disparity predicting unit 422 is provided to a motion/disparity compensating unit 418 and an entropy encoding unit 412. The motion/disparity compensating unit 418 reads the reference image of the reference frame buffer 404 designated by the motion vector or the disparity vector, and provides the reference image to a subtractor 406 and an adder 420. The subtractor 406 subtracts a reference image from an original right image, creates a residue signal and provides the residue signal to the down-sampling unit 408. The down-sampling unit 408 performs down-sampling on the provided residue signal, lowers resolution and provides the residue signal to a DCT/quantizing unit 410. The DCT/quantizing unit 410 performs DCT and quantization on the residue signal of low resolution, and provides the created result to the entropy encoding unit 412 and an inverse quantization/IDCT unit 414. The entropy encoding unit 412 performs entropy encoding on the residue signal provided from the DCT/quantizing unit 410, and the motion vector or the disparity vector provided from the motion/disparity predicting unit, and creates an encoding stream on the down-sampled right image. The inverse quantization/IDCT unit 414 performs inverse quantization and IDCT on the residue signal on which DCT and quantization are performed, and decodes the residue signal. The up-sampling unit 416 recovers the original resolution of the decoded residue signal and provides the residue signal to the adder 420.

Compared with the encoding apparatus of Fig. 2, a loss caused by down-sampling is not generated by performing motion and disparity prediction of the right image without lowering resolution of the reference image. When a reference block is brought by performing motion prediction in the recovered right image and prediction encoding is also performed on the brought reference block, a loss caused by down-sampling and up-sampling of the residue signal is generated. However, the loss caused by down-sampling and up-sampling of the residue signal is smaller than the loss of the method of Fig. 2. Since the method of Fig. 2 performs up-sampling on the recovered image as a post-process, there is a loss in the residue signal and the signal of the reference block. However, the present invention has a loss only in the residue signal. The merit of the present invention is noticeable particularly in disparity prediction. Since a block referred to in the left image in disparity prediction is encoded without change of resolution of the original image, there is no loss caused by down-sampling. Therefore, in the block of the encoded right image, both of the original block and the reference block do not have a loss caused by down-sampling and a loss caused by down- sampling of the residue signal showing a difference between pixel values of two blocks is generated.

Fig. 5 is a flowchart showing a method for decoding an encoded multi-view moving picture including a residue signal and a motion vector or a disparity vector according to the present invention.

The encoded residue signal is decoded through entropy decoding, inverse quantization and IDCT at step S502. The original resolution of the residue signal is recovered at step S504 by up-sampling decoded low resolution residue signal. A present image is recovered at step S506 by summing a block of the residue signal with recovered resolution and a block of a previous image determined according to the motion vector or the disparity vector. That is, a block is recovered by performing motion or disparity compensation according to the encoding method selected for the recovered residue signal. Deblocking is performed in a block boundary of the recovered present image through loop filtering at step S508.

Fig. 6 shows a multi-view moving picture decoding apparatus according to the present invention. An encoding stream of a left image is recovered through a video decoder 602 and the recovered left image is provided to a motion/disparity compensating unit 614 through a switch 616. In motion compensation, the recovered right image is provided to the motion/disparity compensating unit 614 through the switch 616. In disparity compensation, the recovered left image is provided to the motion/disparity compensating unit 614 through the switch 616.

An encoding stream on the down-sampled right image created through the encoding apparatus of Fig. 4 is decoded through an entropy decoding unit 604 and an inverse quantization/IDCT unit 606, recovers original resolution through a up-sampling unit 608, and provided to an adder 610. Deblocking is performed in a block boundary of the right image recovered through the adder 610 through a loop filter 612.

Since the left image is referred to as a time axis, the decoder does not perform disparity compensation, but performs motion compensation on a time axis. The left image is recovered according to the same method as the conventional video decoder. The right image sequentially performs entropy encoding, inverse quantization and IDCT and recovers a residue signal with lowered resolution. The residue signal is recovered through a up-sampling process of recovering original resolution. A recovering process that the recovered residue signal performs motion compensation or disparity compensation according to a selected encoding method is executed. Recovering the right image is completed by performing loop filtering for deblocking in a block boundary such as H.264. The recovered left image and the right image are stored as a reference frame for a frame to be decoded later.

ADVANTAGEOUS EFFECTS MODE FOR INVENTION As described above, the technology of the present invention can be realized as a program and stored in a computer-readable recording medium, such as CD-ROM, RAM, ROM, floppy disk, hard disk and magneto-optical disk. Since the process can be easily implemented by those skilled in the art of the present invention, further description will not be provided herein.

While the present invention has been described with respect to certain preferred embodiments, it will be apparent to those skilled in the art that various changes and modifications may be made without departing from the scope of the invention as defined in the following claims,

INDUSTRIAL APPLICABILITY

When an image size is down-sampled, the present invention performs encoding by lowering resolution of the residue signal, which is a difference value between not an original image and a recovery image, but a reference image and a present image, after motion and disparity prediction. In disparity prediction, a block referred to in the left image does not have a loss caused by down- sampling since the original image is encoded without change of resolution. Therefore, in the block of the encoded right image, both of an original block and a reference block do not have a loss caused by down- sampling and only a loss caused by down-sampling of a residue signal showing a difference between pixel values of two blocks is generated. Accordingly, the present invention can efficiently increase a compression rate without change of a viewer' s sense of stereoscope and transmit multi-view moving picture data through a limited transmission band.

Claims

WHAT IS CLAIMED IS:

1. A method for encoding a multi-view moving picture, comprising the steps of: subtracting a pixel value of a reference image from a pixel value of a present image to be encoded to thereby create a residue signal; lowering resolution by down-sampling the residue signal; and encoding the down-sampled residue signal.

2. The method of claim 1, further comprising the steps of: calculating a motion vector or a disparity vector of the present image with respect to the reference image; decoding and up-sampling the encoded residue signal to thereby recover original resolution; and recovering the present image by summing a block of the recovered residue signal and a block of the reference image determined based on the motion vector or the disparity vector.

3. The method of claim 1, wherein a rate of the down-sampling is determined as a case that a cost function value is a minimum when a block of the present image is encoded and the cost function value according to distortion, which shows a difference between an actually required bit and an original image, is calculated.

4. The method of claim 1, wherein the step of reducing the resolution of the residue signal is performed after motion and disparity prediction in interframe prediction of an image of another viewpoint with respect to an image of a reference viewpoint .

5. The method of claim 1, wherein the encoding is performed by at least one among Discrete Cosine Transform (DCT) , quantization, and entropy encoding.

6. The method of claim 1, further comprising the step of: storing the recovered present image as a next reference image.

7. The method of claim 1, further comprising the step of: deblocking in a block boundary of the recovered present image.

8. An apparatus for encoding a multi-view moving picture, comprising: a means for subtracting a pixel value of a reference image from a pixel value of a present image to be encoded to thereby create a residue signal; a means for lowering resolution by down-sampling the residue signal; and a means for encoding the down-sampled residue signal.

9. The apparatus of claim 8, further comprising: a means for calculating a motion vector or a disparity vector on the reference image of the present image with respect to the reference image; a means for recovering original resolution by decoding and up-sampling the encoded residue signal; and a means for recovering the present image by summing a block of the recovered residue signal and a block of the reference image determined based on the motion vector or the disparity vector.

10. The apparatus of claim 8, wherein a rate of the down-sampling is determined as a case that a cost function value is a minimum when a block of the present image is encoded and a cost function value according to distortion, which shows a difference between an actually required bit and an original image, is calculated.

11. The apparatus of claim 8, wherein encoding is performed by at least one among Discrete Cosine Transform (DCT), quantization, and entropy encoding.

12. The apparatus of claim 8, further comprising: a means for storing the recovered present image as a next reference image.

13. The apparatus of claim 8, further comprising: a means for deblocking the recovered present image in a block boundary.

14. A method for decoding a multi-view moving picture including an encoded residue signal, comprising the steps of: decoding the encoded residue signal; recovering original resolution by up-sampling the decoded residue signal; and recovering a present image by summing a residue signal of recovered resolution and a previous image.

15. The method of claim 14, wherein encoding is performed by at least one among Discrete Cosine Transform

(DCT), quantization, and entropy encoding.

16. The method of claim 14, further comprising the step of: deblocking the recovered present image in a block boundary.

17. An apparatus for decoding a multi-view moving picture including an encoded residue signal, comprising: a means for decoding the encoded residue signal; a means for recovering original resolution by up- sampling the decoded residue signal; and a means for recovering a present image by summing a residue signal of recovered resolution and a previous image .