WO2008023917A1

WO2008023917A1 - Method and apparatus for encoding/decoding stereoscopic images/videos

Info

Publication number: WO2008023917A1
Application number: PCT/KR2007/003985
Authority: WO
Inventors: Sung Moon Chun; Gwang Hoon Park; Doug Young Suh; Kyu Heon Kim; Yong Hyub Oh; Tae Sup Jung
Original assignee: Enhanced Chip Technology Inc.; Industry Academic Cooperation Foundation Of Kyunghee University
Priority date: 2006-08-25
Filing date: 2007-08-21
Publication date: 2008-02-28

Abstract

Provided are a method and an apparatus for encoding a stereoscopic image including a left image and a right image and a method and an apparatus for decoding data encoded by the encoding method and apparatus. In the method of encoding a stereoscopic image, a first merged combined image is generated from odd or even vertical lines of the left image and even or odd vertical lines of the right image and an adaptive encoding process is performed by the use of an MxN merged combined block and an MxN separate combined block of the first merged combined image. Accordingly, it is possible to select image data encoded by a method with the higher compression efficiency and to transmit the selected image data to a decoding apparatus. In the method, block encoding mode information indicating the selected encoding method is transmitted to the decoding apparatus along with the encoded image data.

Description

METHOD AND APPARATUS FOR ENCODING/DECODING STEREOSCOPIC IMAGES/VIDEOS

Technical Field

[1] The present invention relates to encoding and decoding an image, and more particularly, to a method and an apparatus for encoding a stereoscopic image and a method and an apparatus for decoding stereoscopic image data encoded by the encoding method and apparatus. Background Art

[2] A binocular stereoscopic image (hereinafter, referred to as stereoscopic image) includes a pair of a left image and a right image acquired by photographing a subject with a left camera and a right camera which are spaced by a constant distance from each other. Since the left image and the right image are acquired by photographing the same subject but are different from each other in a viewpoint, image information thereof may be slightly different from each other depending on a surface characteristic of the subject, a position of a light source, and the like. The difference in image information between the left image and the right image acquired from the same subject is referred to as disparity.

[3] The stereoscopic images mean images acquired by the use of the left camera and the right camera, respectively, but include three-dimensional images acquired by applying a predetermined transform algorithm to a monoscopic image in a broad sense. The stereoscopic images are generally used to give a three-dimensional effect to a subject to be displayed.

[4] There are various methods of giving a three-dimensional effect to an image which is reproduced by a flat display apparatus such as a liquid crystal display (LCD) device and a plasma display panel (PDP) device by the use of a stereoscopic image. One is a method using a barrier type display apparatus. The barrier type display apparatus attracts attention as a next-generation display apparatus, since it can display both a monoscopic image and a stereoscopic image.

[5] In the barrier type display apparatus, a barrier polarizing plate formed of a polarizing film or a polarizing glass is attached or provided to a front surface of a flat display apparatus. A line-type barrier pattern is formed in the barrier polarizing plate. The barrier pattern allows a left eye to see only a left image portion of the displayed image and allows a right eye to see only a right image portion of the displayed image. Such barrier patterns can be basically classified into a vertical line type and a horizontal line type. The barrier patterns can be classified into a straight line shape, a saw-teeth shape, a diagonal line shape, depending on minute shapes of the vertical lines or the horizontal lines. The minute line shapes of the barrier patterns cause differences in the three-dimensional effect between images.

[6] In recent years, such barrier type display apparatuses are attractively used to display three-dimensional images on display units of 3-D televisions or mobile electronic devices such as mobile phones, PMP, and DMB receivers. However, a new and efficient encoding method different from the known method of encoding a monoscopic image is required for displaying a three-dimensional image by the use of the barrier type display apparatus. This is because a left image and a right image are both required for displaying a stereoscopic image.

[7] The multi-view profile (MVP) of MPEG-2 is known as a method of encoding a stereoscopic image. In the MVP of MPEG-2, one of the left and right images, for example, the left image, is encoded as a base layer by the same method as encoding a monoscopic image and only correlation information of the other image, for example, the right image, with the left image is encoded as an enhancement layer. This method is an encoding method basically using temporal scalability. Such a method of encoding a stereoscopic image using temporal scalability is also defined in MPEG-4.

[8] In the method of encoding a stereoscopic image using temporal scalability, the left image and the right image should be synchronized with each other. In the method of forming the base layer and the enhancement layer into individual bit streams and transmitting the individual bit streams, the synchronization at the time of display is not easy. A method of forming the encoded bit streams output from the base layer and the enhancement layer into a single bit stream and transmitting the single bit stream is known as one method of solving the problem with synchronization. The method has a problem in that a particular multiplexing process for forming a single bit stream is necessary. When differences in image information such as brightness, chrominance, and/or RGB values are caused between the left and right images depending on lighting directions or camera characteristics, a data amount of the correlation information increases to reduce a compression rate. Disclosure of Invention Technical Problem

[9] Therefore, there is need for a still image or video encoding method that can solve the problem with synchronization of the left and right images, that can use the known methods of encoding a stereoscopic image without any change, and that has a high compression rate even when differences in pixel values between the left and right images are caused depending on the lighting directions and the camera characteristics.

[10] An object of the invention is to provide methods and apparatuses for encoding and decoding a stereoscopic image, which can simplify the configuration of a decoding apparatus including a barrier type display and which can prevent the compression rate from decreasing due to differences in image information between the left and right images.

[11] Another object of the invention is to provide methods and apparatuses for encoding and decoding a stereoscopic image, which can prevent a color blurring phenomenon from occurring even at the time of encoding a merged combined image by the use of MPEG-I, MPEG-2, MPEG-4, H.264/AVC, or JPEG which is a known method of encoding a monoscopic image and can enhance the compression rate. Technical Solution

[12] According to an aspect of the invention, there is provided a method of encoding a stereoscopic image including a left image and a right image, the method including the steps of: generating a first merged combined image from odd or even vertical lines of the left image and even or odd vertical lines of the right image; and performing an adaptive encoding process by the use of an MxN merged combined block of the first merged combined image and an MxN separate combined block corresponding to the MxN merged combined block.

[13] According to another aspect of the invention, there is provided a method of encoding a stereoscopic image including a left image and a right image in the unit of block and constructing block header data including information on the encoding in the unit of block. Here, the block header data includes block encoding mode information indicating an encoding method for an MxN block of a merged combined image which is generated from odd or even vertical lines of the left image and even or odd vertical lines of the right image, and the block encoding mode information indicates a first merging method of encoding a merged combined block or a first separation method of encoding a separate combined block.

[14] According to still another aspect of the invention, there is provided a method of encoding a stereoscopic image including a left image and a right image in the unit of block and constructing block header data including information on the encoding in the unit of block. Here, the block header data includes motion estimation mode information indicating a motion estimation method for an MxN block of a merged combined image which is generated from odd or even vertical lines of the left image and even or odd vertical lines of the right image, and the motion estimation mode information indicates a second merging method of performing a motion estimation process by the use of a merged combined block or a second separation method of encoding a separate combined block.

[15] According to still another aspect of the invention, there is provided a method of decoding a stereoscopic image, the method including the steps of: decoding block encoding mode information indicating an encoding method for image data of an MxN block of a merged combined image, from an encoded bit stream of the merged combined image which is generated from odd or even vertical lines of a left image and even or odd vertical lines of a right image; and adaptively decoding the image data of the MxN block on the basis of the block encoding mode information.

[16] In the above-mentioned aspect of the invention, the block encoding mode information indicates a merging method of encoding an MxN merged combined block or a separation method of encoding an MxN separate combined block. In this case, the image data of the MxN block is encoded using 4:2:2 as a sampling frequency ratio when the block encoding mode information indicates the merging method, and the image data of the MxN block is encoded using 4:2:0 as the sampling frequency ratio when the block encoding mode information indicates the separation method.

[17] According to still another aspect of the invention, there is provided a method of decoding a stereoscopic image, the method including the steps of: decoding motion estimation mode information indicating a motion estimation method for image data of an MxN block of a merged combined image, from an encoded bit stream of the merged combined image which is generated from odd or even vertical lines of a left image and even or odd vertical lines of a right image; and adaptively decoding the image data of the MxN block on the basis of the motion estimation mode information.

[18] According to still another aspect of the invention, there is provided a method of decoding a stereoscopic image, the method including the steps of: decoding block encoding mode information indicating an encoding method for image data of an MxN block of a merged combined image, from an encoded bit stream of the merged combined image which is generated from all vertical lines of a left image and all vertical lines of a right image; and adaptively decoding the image data of the MxN block on the basis of the block encoding mode information.

[19] According to still another aspect of the invention, there is provided a method of encoding a stereoscopic image including a left image and a right image, the method including the steps of: generating a first merged combined image from odd or even vertical lines of the left image and even or odd vertical lines of the right image; adaptively encoding an MxN merged combined block of the first merged combined image in the unit of a block to generate first encoding data and encoding an MxN separate combined block corresponding to the MxN merged combined block in the unit of a block to generate second encoding data; and outputting one encoding data selected from the first encoding data and the second encoding data and block encoding mode information indicating an encoding method for the selected encoding data.

[20] In the above-mentioned aspect of the invention, the method may further include, before performing the encoding process in the unit of a block: performing a first motion estimation process of estimating a motion using the MxN merged combined block and a second motion estimation process of estimating a motion using the MxN separate combined block, respectively; selecting one of the first motion estimation process and the second motion estimation process; and making a motion compensation in accordance with the selected motion estimation process. In this case, motion estimation mode information indicating the selected motion estimation process may be output in the step of selecting the motion estimation process

[21] According to still another aspect of the invention, there is provided an apparatus for encoding a stereoscopic image including a left image and a right image, the apparatus including the steps of: a merged combined image generating unit for generating a first merged combined image from odd or even vertical lines of the left image and even or odd vertical lines of the right image; a block encoding unit for adaptively encoding an MxN merged combined block of the first merged combined image in the unit of a block to generate first encoding data and encoding an MxN separate combined block corresponding to the MxN merged combined block in the unit of a block to generate second encoding data; and a first selection unit for selecting one of the first encoding data and the second encoding data and outputting block encoding mode information indicating an encoding method for the selected encoding data.

[22] According to still another aspect of the invention, there is provided an apparatus for decoding a stereoscopic image, the apparatus including: a demultiplexing unit for separating image data of an MxN block of a merged combined image and block encoding mode information indicating an encoding method for the image data of the MxN block, from a bit stream obtained by encoding the merged combined image which is generated from odd or even vertical lines of a left image and even or odd vertical lines of a right image; and a block decoding unit for decoding the block encoding mode information and adaptively decoding the image data of the MxN block in accordance with the decoded block encoding mode information.

[23] According to still another aspect of the invention, there is provided a method of encoding a stereoscopic image including a left image and a right image, the method comprising the steps of: generating a first merged combined image from odd or even horizontal lines of the left image and even or odd horizontal lines of the right image; and performing an adaptive encoding process by the use of an MxN merged combined block of the first merged combined image and an MxN separate combined block corresponding to the MxN merged combined block.

Advantageous Effects

[24] In the above-mentioned technical solution, the stereoscopic image is formed with the left and right cameras and is then displayed by a display apparatus fitted with a polarizing barrier panel. Accordingly, by transmitting additional information per unit block at the time of displaying a three-dimensional image, it is possible to remove overlapping data in space which cannot be solved by the known video codec and thus to enhance the compression rate of data by the use of a verified method. [25] By transmitting additional color information to effectively remove a color blur phenomenon resulting from a difference in color information which cannot be solved by a known video codec, it is possible to greatly enhance the color quality of a stereoscopic image.

Brief Description of the Drawings [26] Fig. 1 is a diagram illustrating a separated combined image generated from a left image and a right image. [27] Fig. 2 is a diagram illustrating a method of generating a merged combined image from odd vertical lines of a left image and even vertical lines of a right image. [28] Fig. 3 is a diagram illustrating an encoding process in the unit of block using a merging method in which a sampling frequency ratio of brightness Y and two chrominance signals Cb and Cr is 4:2:0. [29] Fig. 4 is a diagram illustrating an encoding method according to a first embodiment of the invention. [30] Figs. 5 and 6 are diagrams illustrating a sampling frequency ratio of brightness Y and chrominance signals Cb and Cr in performing an encoding process in the unit of block using a merging method and a separation method according to the first embodiment of the invention. [31] Fig. 7 is a diagram illustrating an encoding method according to a second embodiment of the invention. [32] Figs. 8 and 9 are block diagrams illustrating a configuration of an encoding apparatus according to an embodiment of the invention. [33] Figs. 10 and 11 are block diagrams illustrating a configuration of a decoding apparatus according to an embodiment of the invention. [34] [35] Hereinafter, exemplary embodiments of the invention will be described in detail with reference to the accompanying drawings. The following embodiments are intended to explain technical spirits of the invention and thus the technical spirits of the invention is not limited to the exemplary embodiments.

Best Mode for Carrying Out the Invention [36] A method using a "separate combined image" is known as a method of encoding a stereoscopic image, which can solve a problem associated with the synchronization between left and right images and can use known methods of encoding a monoscopic image without any change. The separate combined image means a combined image obtained by reducing left and right images into images with a 1/2 resolution and then combining the reduced left and right images in a side-by-side format or a top-down format. For example, as shown in Fig. 1, the separate combined image 12 is an image obtained by extracting only odd vertical lines from a 320x240 left image 10a acquired from a left camera to form a 160x240 reduced left image 12a, extracting only even vertical lines from a 320x240 right image 10b acquired from a right camera to form a 160x240 reduced right image 12b, and arranging the reduced left and right images in the side-by- side format.

[37] In this method, at the time of encoding a stereoscopic image, a separate combined image is formed from left and right images and is encoded by treating the separate combined image as one image. Accordingly, since a single encoded stream is output as a result of encoding the separate combined image, the problem with the synchronization between the left and right images dose not occur.

[38] However, when the separate combined image 12 is encoded in the unit of block by the use of the known methods, two independent reduce images 12a and 12b are encoded respectively. Accordingly, correlation information between the left and right images in the separate combined image cannot be used. As a result, there is a problem in that the amount of data increases, thereby reducing compression efficiency. In addition, a process of separating the left and right images again and forming a merged combined image should be added for a display.

[39] In another method of encoding a stereoscopic image, a merged combined image can be encoded without any change. As shown in Fig. 2, the merged combined image means an image 22 in which odd vertical lines extracted from a left image 20a and even vertical lines extracted from a right image 20b are alternately arranged. A merged combined image may be formed by extracting even vertical lines from the left image 20a and odd vertical lines from the right image 20b, or extracting horizontal lines from both the left and right images 20a and 20b.

[40] The method of encoding a merged combined image has an advantage that the encoding efficiency can be improved slightly because the encoding is performed on one merged image. In addition, since an image format used in the barrier type display is encoded without any change, the method has an advantage that the configuration of a decoding apparatus can be simplified.

[41] In the known encoding method, the encoding process is performed in the unit of

MxN block (where M and N are a natural number of 2 or more and preferably are an even number of 4 or more). The MxN unit block of the merged combined image has a shape of a merged combined block in which pixel lines of the left image and pixel lines of the right image are arranged adjacent to each other. In the merged combined block, no difference in image information may occur in one unit block, but a great difference in image information may occur between the adjacent lines in some cases. In the latter case, when the encoding process is performed in the unit of block, the encoding efficiency may decrease due to the difference in image information between the adjacent lines.

[42] When the merged combined image is encoded using 4:2:0 which is most widely used as a sampling frequency ratio of brightness Y and chrominance signals Cb and Cr, chrominance information cannot be displayed to correspond to the original image in some pixel positions, thereby causing a color blurring phenomenon. This will be now described in more detail with reference to Fig. 3.

[43] Fig. 3 shows a 16x16 merged combined block for the brightness Y and 8x8 merged combined blocks for the chrominance signals Cb and Cr. In the 4:2:0 method, since the chrominance information is expressed by only 1/4 of bright information, the 16x16 merged combined block for the brightness Y corresponds to the 8x8 merged combined blocks for the chrominance signals Cb and Cr. In other words, in the 4:2:0 method, four brightness pixels are expressed by one chrominance pixel. For example, a first brightness (Y) block 40a including four pixels is expressed by first chrominance (Cb, Cr) pixels 40b and 40c and a second brightness block 42a including other four pixels is expressed by second chrominance pixels 42b and 42c.

[44] In this case, the first chrominance (Cb, Cr) pixels 40b and 40c and the second chrominance (Cb, Cr) pixels 42b and 42c have an average value or a mean value of the pixels of the first vertical line which are pixels extracted from the left image and the pixels of the second vertical line which are pixels extracted from the right image. As a result, when the merged combined block is encoded without any change, inaccurate chrominance values may be used for the pixels, thereby making it difficult to accurately reproduce the original colors of a subject and causing the color blurring phenomenon.

[45] Fig. 4 is a diagram illustrating an encoding method according to a first embodiment of the invention. In Fig. 4, reference numeral 50 denotes a merged combined image that corresponds to reference numeral 12 in Fig. 3 and that is constructed by extracting odd vertical lines of the left image 10a and even vertical lines of the right image 10b. As described above, the merged combined image 50 may have a structure in which the odd vertical lines of the left image and the even vertical lines of the right image are alternately arranged. However, it is obvious to those skilled in the art that embodiments of the invention to be described later can be applied to a case that the merged combined image has a structure in which horizontal lines of the left image and horizontal lines of the right image are alternately arranged. It will be described below only that the merged combined image has a structure in which the vertical lines of the left image and the vertical lines of the right image are arranged alternately.

[46] It is assumed in Fig. 4 that a macro block A is a block in which the left image and the right image are similar to each other in image characteristic and thus hardly have a difference in image information therebetween. It is also assumed that a macro block B is a block in which the left image and the right image are different from each other in image characteristic and thus have a light or large difference in image information therebetween. Reference numerals 52a and 52b denote blocks obtained by deforming the macro blocks A and B of the merged combined image 50 for the encoding according to the embodiment of the invention, where reference numeral 52a denotes a merged combined block showing the macro block A without any change and reference numeral 52b denotes a separate combined block in which the pixel lines of the left image and the pixel lines of the right image in the macro block B are separated side by side and arranged.

[47] According to this embodiment, unit blocks are adaptively encoded depending on image characteristics of the encoding unit blocks of the merged combined image 50. Here, "adaptively encoded" means that image data are encoded by the use of one of two or more encoding methods or that image data having a more excellent compression rate and the like are selected from data encoded by the use of two or more encoding methods and only the selected image data are transmitted to a decoding apparatus.

[48] More specifically, in an encoding method according to an example of this embodiment, a macro block which is an encoding unit block is encoded in the form of the merged combined block 52a (hereinafter, referred to as "merging method") and/or is encoded in the form of the separate combined block 52b (hereinafter, referred to as "separation method"). Of course, when the encoding process is performed using the separation method, a left 8x16 block and a right 8x16 block are encoded independently of each other. In this case, the left 8x16 block and the right 8x16 block may be independently encoded by the use of the known method of encoding a monoscopic image, or the left 8x16 block may be encoded by the use of the known method, only disparity information of the right 8x16 block on the left 8x16 block may be encoded, and then the encoded data may be transmitted to the decoding apparatus.

[49] A criterion for determining by what method of the merging method and the separation method an encoding process should be performed or determining what data of data encoded by the merging method and the separation method should be transmitted to the decoding apparatus can be determined freely. For example, a method of performing an encoding process by the use of the above-mentioned two methods, comparing compression rates such as generated bit rates thereof, and selecting image data encoded by a method having the more excellent encoding ability may be used. In this case, the criterion is the compression rate, but the invention is not limited to it.

[50] For example, in Fig. 4, since the left and right images of the macro block A are similar to each other in image characteristic, it is more effective to encode the macro block A using the merging method. On the other hand, since the left and right images of the macro block B are slightly different from each other in image characteristic, it is more effective to encode the macro block B using the separation method. In this case, in this embodiment, regarding the macro block A, data encoded using the merging method having a more excellent compression rate of the merging method and the separation method are selected and transmitted to the decoding apparatus. Regarding the macro block B, data encoded using the separation method having a more excellent compression rate of the merging method and the separation method are selected and transmitted to the decoding apparatus.

[51] According to this embodiment of the invention, the merged combined macro block

54a can be encoded in the unit of smaller block. For example, the encoding may be performed in the unit of 16x8 block, 8x16 block, 8x8 block, 8x4 block, 4x8 block, or 4x4 block, instead of the unit of macro block. The separate combined macro block 54b can be encoded in the unit of smaller block. For example, the encoding may be performed in the unit of 8x8 block, 8x4 block, 4x8 block, or 4x4 block, instead of the unit of 8x16 block.

[52] In this way, only a bit stream having a higher compression rate among the bit streams encoded by one method having more excellent encoding efficiency among the merging method and the separation method or encoded by the two methods is selected and transmitted to the decoding apparatus. Accordingly, the decoding apparatus need know what method has been used to encode the input bit stream. In the invention, one bit of block encoding mode information is assigned per unit block as information indicating an encoding method. For example, "0" can be assigned to a block encoded by the merging method, " 1 " can be assigned to a block encoded by the separation method, and vice versa. The one bit information is included in a block header and transmitted to a decoding apparatus.

[53] The encoding method according to the embodiment of the invention can be applied to an encoding process in the unit of a block. In the known video encoding method, an encoding process in the unit of a block is performed in a motion compensation process using a motion estimation, a transform process such as a discrete cosine transform (DCT), and a quantization process. For example, a 16x16 unit block is often used in the motion estimation process, and 16x8 and 8x4 unit blocks are often used therein. 16x16 and 8x8 unit blocks are representatively used in the transform process and the quantization process. [54] Accordingly, the adaptive encoding method according to this embodiment of the invention can be used in the motion estimation process and/or the transform process and the quantization process. When the motion estimation process is performed by the use of the separation method according to this embodiment of the invention, two motion vectors are transmitted per unit block.

[55] A sampling frequency ratio of the brightness Y and the chrominance signals Cb and

Cr according to a first embodiment of the invention will be described now. Similarly to the known encoding method, when the encoding process is performed by the use of the separation method, original colors of a subject can be easily reproduced even with the sampling frequency ratio set. However, at the time of using the merging method, the sampling frequency ratio at which the number of chrominance pixels to be extracted is double the number of chrominance pixels to be extracted at the time of using the separation method is preferably used. For example, when the sampling frequency ratio is 4:2:0 in case of the separation method, the sampling frequency ratio may be 4:2:2 in case of the merging method. This example will be described more specifically with reference to Figs. 5 and 6. However, it is obvious to those skilled in the art that the same sampling frequency ratio as being used in the separation method can be used in the merging method.

[56] Fig. 5 is a diagram illustrating an example where the sampling frequency ratio used in the separation method is 4:2:0. Referring to Fig. 5, in the separation method, one chrominance pixel of the chrominance separate combined blocks 62 and 64 is extracted per four brightness pixels of a 16x16 brightness separate combined block 60 to perform an encoding process, similarly to the known encoding methods. That is, in the separation method, one 8x8 Cb separate combined block 62 and one 8x8 Cr separate combined block 64 correspond to the 16x16 brightness separated combined block 60. In this case, four pixels 60a extracted from the left image in the brightness separate combined block 60 correspond to the pixels 62a and 64a extracted from the left images in the chrominance separate combined blocks 62 and 64 and four pixels 60b extracted from the right image in the brightness separate combined block 60 correspond to the pixels 62b and 64b extracted from the right images in the chrominance separate combined blocks 62 and 64. Accordingly, the color blurring phenomenon does not occur and it is thus possible to accurately reproduce the original color of a subject.

[57] Fig. 6 is a diagram illustrating an example where the sampling frequency ratio used in the merging method is 4:2:2 and the number of chrominance pixels to be extracted is double that of the separation method in the side-by-side direction. That is, in the merging method, since one chrominance pixel is extracted per two brightness pixels adjacent to each other in the top-down direction, two 8x8 Cb blocks 72 and two 8x8 Cr blocks 74 correspond to a 16x16 brightness merged combined block 70. Here, as shown in Fig. 6, two 8x8 Cb blocks 72 include two different 8x8 blocks in which the pixels extracted from the left image and the pixels extracted from the right image are distinguished and arranged, or two 8x8 Cb merged combined blocks. As shown in Fig. 6, two 8x8 Cr blocks 74 also include two different 8x8 blocks in which the pixels extracted from the left image and the pixels extracted from the right image are distinguished and arranged, or two 8x8 Cr merged combined blocks.

[58] In this case, two pixels extracted from the left image in the brightness separate combined block 70 correspond to the pixels 72a and 74a extracted from the left images in the chrominance blocks 72 and 74. Two pixels extracted from the right image in the brightness separated combined block 70 correspond to the pixels 72b and 74b extracted from the right images in the chrominance blocks 72 and 74. According to this embodiment of the invention, in the merging method, by extracting double the chrominance information in the separation method, the color blurring phenomenon does not occur even when using the merging method, thereby accurately reproducing the original colors of a subject. Mode for the Invention

[59] A second embodiment of the invention will be described now. In the second embodiment of the invention, an encoding process may be performed by the use of all image information on the left image and the right image input from the left and right cameras. Fig. 7 is a diagram illustrating the second embodiment of the invention.

[60] Referring to Fig. 7, a merged combined image 80 has a size larger two times in the side-by-side direction than the merged combined image 50 shown in Fig. 4. As described above, the merged combined image 50 shown in Fig. 4 is an image obtained by arranging odd (or even) vertical lines of the left image in odd (or even) vertical lines of the merged combined image 50 and arranging even (or odd) vertical lines of the right image in even (or odd) vertical lines of the merged combined image 50. On the contrary, the merged combined image 80 shown in Fig. 7 is an image obtained by arranging all the vertical lines of the left image in odd (or even) vertical lines of the merged combined image 80 and arranging all the vertical lines of the right image in the even (or odd) vertical lines of the merged combined image 80.

[61] The second embodiment is similar to the first embodiment, except that the encoding process is performed using the merged combined image 80 shown in Fig. 7 and using the same method as the merged combined image 50 shown in Fig. 4. Accordingly, all the details described in the first embodiment are true in this embodiment. Hereinafter, only differences from the first embodiment will be described.

[62] In this embodiment, the encoding process is performed in the unit of a macro block by the use of the merging method as denoted by reference numeral 82 and the encoding process is performed in the unit of a macro block by the use of the separation method as denoted by reference numeral 84. Then, data encoded by a method with the higher compression rate of the two encoding methods is selected and transmitted to a decoding apparatus along with information on the selected encoding method. Since the macro blocks 82 and 84 extracted from the merged combined image 80 shown in Fig. 7 have a very high correlation property between the left image and the right image regardless of the merging method or the separation method, data of the right image may be transmitted to the decoding apparatus by converting disparity information between the left image and the right image.

[63] In this embodiment, since all the information on the left image and the right image is transmitted to the decoding apparatus, there is a disadvantage that the amount of data to be transmitted increases, but there is an advantage that an image with higher image quality can be displayed. This embodiment is applicable to a display apparatus requiring all the information on the left image and the right image, as well as the barrier type display apparatus.

[64] A third embodiment of the invention will be described now. In the third embodiment of the invention, the first and second embodiments of the invention are both applied to perform an encoding process and then data encoded by an encoding method with the highest compression rate is selected and transmitted to a decoding apparatus.

[65] According to the third embodiment, the merged combined image 50 shown in Fig. 4 and the merged combined image 80 shown in Fig. 7 are all constructed from the left image and the right image. Then, the merged combined images 50 and 80 are encoded by the use of the merging method and the separation method, data encoded by the method with the highest encoding efficiency is selected, and the selected data is transmitted to the decoding apparatus along with information indicating the encoding method for the selected data.

[66] Here, the information indicating the encoding method can be expressed using a two- bit fixed length code indicating the encoding method. For example, the information indicating the encoding methods of the merged combined image 50 using the merging method and the separation method may be expressed by codes "00" and "01" and the information on the encoding methods of the merged combined image 80 using the merging method and the separation method may be expressed by codes " 10" and "11." However, the information indicating the encoding method may be expressed by a variable length code or by an Exp-Golomb code.

[67] Apparatuses for encoding and decoding a stereoscopic image according to embodiments of the invention will be described now.

[68] Fig. 8 is a block diagram illustrating a partial configuration of an apparatus for encoding a stereoscopic image according to a fourth embodiment of the invention and Fig. 9 is a block diagram illustrating a partial configuration of an apparatus for encoding a stereoscopic image according to a fifth embodiment of the invention. Although only an encoding process on a P-picture and a B -picture including a motion estimation process and a motion compensation process will be described, it is obvious to those skilled in the art that the encoding apparatus according to the embodiments of the invention can be applied to an encoding process on an I-picture.

[69] Referring to Fig. 8, the apparatus 100 for encoding a stereoscopic image includes a motion estimation unit 102, a motion compensation unit 104, a unit block encoder 106, a multiplexer 108, a memory 110, a first adder 112, and a second adder 114.

[70] The motion estimation unit 102 makes a motion estimation process by the use of the input left and right images and the reconstructed image of the previous frame stored in the memory 110, that is, the merged combined image of the previous frame. The motion estimation process is performed in the unit of a predetermined sized block by the use of one method of the first to third embodiments of the invention. As described above, according to this embodiment, the motion estimation process can be performed using the merging method and the separation method. Accordingly, the motion estimation unit 102 generates information on a motion estimation mode indicating what method is used to perform the motion estimation and outputs the generated motion estimation mode information to the multiplexer 108 along with calculated motion vectors. As described above, when the motion estimation process is performed using the separation method, two motion vectors are generated every unit block. Accordingly, two motion vectors are output every unit block.

[71] The motion compensation unit 104 performs the motion estimation process by the use of the information input from the motion estimation unit 102 and the reconstructed image of the previous frame stored in the memory 110. The estimated image resulting from the motion estimation process is output to the first adder 112.

[72] The first adder 112 generates an error image by subtracting the estimated image input from the motion compensation unit 104 and the current image and outputs the generated error image.

[73] The unit block encoder 106 performs an encoding process such as a transform process and a quantization process on the error image input from the first adder 112. The unit block encoder 106 performs the encoding process in the unit of a predetermined sized block by the use of one method of the first to third embodiments of the invention. As described above, according to this embodiment, it is possible to perform the encoding process by the use of the merging method and the separation method. Accordingly, the unit block encoder 106 generates block encoding mode information indicating what method is used to perform the encoding process at the time of performing the transform and quantization processes and outputs the generated block encoding mode information to the multiplexer 108 along with a calculated quantization coefficient. Of course, before being input to the multiplexer 108, it is obvious to those skilled in the art that the quantization coefficient is subjected to a usual entropy encoding process.

[74] The multiplexer 108 multiplexes the motion estimation mode information and/or the block encoding mode information and all the information having been subjected to the entropy encoding process by the unit block encoder 106. The bit stream multiplexed by the multiplexer 108 is transmitted to a decoding apparatus or stored in a storage medium.

[75] On the other hand, the quantization coefficient encoded by the unit block encoder

106 is subjected to an inverse quantization process and an inverse transform (not shown) and then is stored in the memory 110 for the purpose of encoding a next frame to be input.

[76] Referring to Fig. 9, the apparatus for encoding a stereoscopic image according to the fifth embodiment of the invention further includes a chrominance information encoder 120 in addition to the encoding apparatus 100 according to the fourth embodiment. Only differences from the encoding apparatus 100 according to the fourth embodiment will be described now.

[77] The chrominance information encoder 120 is a means for encoding unit block color information of a right image when the encoding process is performed using the merging method by the unit block encoder 106. The unit block color information of the right image encoded by the chrominance information encoder 120 along with other information is also input to the multiplexer 108 after being subjected to the entropy encoding process. The multiplexer 108 forms a bit stream including the additional information and transmits the formed bit stream to the decoding apparatus.

[78] The chrominance information encoder 120 is shown as a constituent element independent of the unit block encoder 106 in the figure, but may be a constituent element incorporated into the unit block encoder 106. In this case, the unit block encoder 106 not only encodes the unit block color information of the left image but also encodes the unit block color information of the right image.

[79] According to this embodiment, the chrominance information of the left image is encoded by the unit block encoder 106 and the chrominance information of the right image is encoded by the chrominance information encoder 120. On the contrary, the chrominance information of the right image may be encoded by the unit block encoder 106 and the chrominance information of the left image may be encoded by the chrominance information encoder 120. In some embodiments, the unit block encoder 106 may encode the chrominance information of the left image and the chrominance information of the right image by a half unit block and the chrominance information encoder 120 may encode the chrominance information of the left image and the chrominance information of the right image by the other half unit block.

[80] Fig. 10 is a block diagram illustrating a partial configuration of an apparatus for decoding a stereoscopic image according to a sixth embodiment of the invention and Fig. 11 is a block diagram illustrating a partial configuration of an apparatus for decoding a stereoscopic image according to a seventh embodiment of the invention. Similarly to the encoding apparatuses, although only a decoding process on a P-picture and a B-picture including a motion estimation process and a motion compensation process will be described, it is obvious to those skilled in the art that the decoding apparatus according to the embodiments of the invention can be applied to a decoding process on an I-picture.

[81] Referring to Fig. 10, the apparatus 200 for decoding a stereoscopic image includes a demultiplexer 202, a unit block decoder 204, a motion compensation unit 206, a memory 208, and an adder 210.

[82] The demultiplexer 202 separates the input bit stream into the encoded image data and the additional information. The encoded image data is transmitted to the unit block decoder 204 after being subjected to an entropy decoding process and the additional information, which includes block decoding mode information, motion vector information, and motion estimation mode information, is transmitted to proper units.

[83] The unit block decoder 204 performs a decoding process such as an inverse quantization process and an inverse transform process on the image data having been subjected to the entropy decoding process and input from the demultiplexer 202. In the inverse quantization process and the inverse transform process, referring to the block encoding mode information included in a block header or the like, it is determined what method of the merging method and the separation method is used to encode the image data and then an adaptive decoding process is performed depending on the determination result.

[84] The motion compensation unit 206 makes a motion compensation by the use of the motion vectors and the motion estimation mode information output from the demultiplexer 202 and the reconstructed image data of the previous frame stored in the memory 208. At this time, the motion compensation is made using the merging method or the separation method on the basis of the motion estimation mode information and the estimated image is output to the adder 210.

[85] The adder 210 adds the estimated image data output from the motion compensation unit 206 to the encoded differential image data input from the unit block decoder 204 to reconstruct the current image, that is, the merged combined image. The reconstructed merged combined image is output to a display apparatus and is stored in the memory 208.

[86] Referring to Fig. 11, the apparatus 200' for decoding a stereoscopic image according to the seventh embodiment of the invention includes a chrominance information decoder 220 in addition to the decoding apparatus 200 according to the sixth embodiment. Only differences from the decoding apparatus 200 according to the sixth embodiment will be described now.

[87] The chrominance information decoder 120 decodes the encoded chrominance data output from the demultiplexer 202. The decoding process of the chrominance information decoder 120 is performed to correspond to the encoding process of the encoding apparatus 100'. The information decoded by the chrominance information decoder 120 is used to reconstruct the merged combined image as an output image along with the information decoded by the unit block decoder 204.

[88] The test result of the encoding method according to an embodiment of the invention will be compared with the test result using the known video codec.

[89] The test conditions are as follows.

[90] 1. The compression rates in accordance with the International Standard for a stereoscopic video are compared. By comparing the compression rates of images corresponding to 20% of the entire test images, the whole compression rate could be estimated.

[91] 2. The compression rates of nine images having a 320x240 size are compared by the use of the I-picture encoding in MPEG-4. In the following test results, the header information having 1 to 2 bits per unit block is not included in the number of generated bits and the test is made by the use of DCT data.

[92] 3. An amount of generated bits of a merged combined image, an amount of generated bits of a separate combined image, an amount of bits selectively calculated by dividing an 8x8 block into 8x4 blocks as described in the first embodiment, and an amount of bits selectively calculated by dividing a 16x8 block into 8x8 blocks as described in the second embodiment are compared with each other.

[93] 4. The objective image quality (PSNR) was calculated in the test. However, since an average error thereof was 0.4 dB or less, the PSNR was not particularly described. In the quantization in accordance with the H-263 standard, Q=IO was set as a quantization step size.

[94] 5. In the 8x4 block encoding, the bits (1200 bits) indicating the number of blocks should be contained as additional bits and used for comparison. In the 16x8 block encoding, the bits (600 bits) indicating the number of blocks should be contained as additional bits and used for comparison.

[95]

[96] Test Result [97] 1. It can be seen from Table 1 that an excellent effect is obtained from a still image (JPEG) when an encoding process is performed using the encoding method according to the invention in the International Standard for a stereoscopic video. Particularly, when a 16x8 block is encoded according to the invention, it can be seen that an effect of reducing the generated data by 5% to 30% is exhibited, except for one singular image among nine images. When an image hardly has the effect of reducing the generated data, it means that the left image and the right image are greatly different from each other in brightness and color. Accordingly, such an image is preferably excluded from the test target.

[98] Table 1

[99] [100] 2. When an 8x4 block is encoded according to the invention, a data amount greater than that of the separate combined image LR is generated. It also means that it is an image having a great difference between the left image and the right image. It can be seen that an enhancement in compression rate by 5% to 30% is obtained from the other images of which the left and right images have brightness similar to each other.

[101] 3. It is predicted that the compression rate for a video can be enhanced by 40% in maximum by selectively adding a motion estimation method. [102]

Industrial Applicability

[103] The invention can be usefully applied to the industries associated with apparatuses for encoding and/or decoding a stereoscopic image, display apparatuses including the apparatuses, mobile apparatuses such as mobile phones, and the like.

Claims

Claims [1] A method of encoding a stereoscopic image including a left image and a right image, the method comprising the steps of: (a) generating a first merged combined image from odd or even vertical lines of the left image and even or odd vertical lines of the right image; and (b) performing an adaptive encoding process by the use of an MxN merged combined block of the first merged combined image and an MxN separate combined block corresponding to the MxN merged combined block. [2] The method according to claim 1, wherein the step of (b) includes the steps of:

(b-1) encoding the MxN merged combined block by the use of a merging method and encoding the MxN separate combined block by the use of a separation method; and

(b-2) comparing encoded data by the merging method and encoded data by the separation method with each other and selecting and outputting only the encoded data by one method of the two methods.

[3] The method according to claim 2, wherein block encoding mode information and/or motion estimation mode information representing the selected encoding method are both output in the step of (b-2).

[4] The method according to claim 3, wherein a motion estimation process and/or a transform process are performed in performing an encoding process in the step of (b-1).

[5] The method according to claim 4, wherein the motion estimation process is performed in performing an encoding process in the step of (b-1), and wherein at the time of performing the motion estimation process by the use of the separation method, a motion estimation process is performed in the unit of an M/ 2xN block or a smaller block to calculate and output two or more motion vectors in the step of (b-1).

[6] The method according to claim 2, wherein 4:2:2 is used as a sampling frequency ratio in the encoding process using the merging method and 4:2:0 is used as the sampling frequency ratio in the encoding process using the separation method.

[7] The method according to claim 1, wherein a second merged combined image is further generated from all the vertical lines of the left image and all the vertical lines of the right image in the step of (a), and wherein an adaptive encoding process is performed by the use of the MxN merged combined block and the MxN separate combined block and a 2MxN merged combined block and a 2MxN separate combined block obtained from the 2MxN merged combined image in the step of (b).

[8] A method of encoding a stereoscopic image including a left image and a right image in the unit of block and constructing block header data including information on the encoding in the unit of block, wherein the block header data includes block encoding mode information indicating an encoding method for an MxN block of a merged combined image which is generated from odd or even vertical lines of the left image and even or odd vertical lines of the right image, and wherein the block encoding mode information indicates a first merging method of encoding a merged combined block or a first separation method of encoding a separate combined block.

[9] A method of encoding a stereoscopic image including a left image and a right image in the unit of block and constructing block header data including information on the encoding in the unit of block, wherein the block header data includes motion estimation mode information indicating a motion estimation method for an MxN block of a merged combined image which is generated from odd or even vertical lines of the left image and even or odd vertical lines of the right image, and wherein the motion estimation mode information indicates a second merging method of performing a motion estimation process by the use of a merged combined block or a second separation method of encoding a separate combined block.

[10] A method of decoding a stereoscopic image, the method comprising the steps of:

(a) decoding block encoding mode information indicating an encoding method fo r image data of an MxN block of a merged combined image, from an encoded bit stream of the merged combined image which is generated from odd or even vertical lines of a left image and even or odd vertical lines of a right image; and

(b) adaptively decoding the image data of the MxN block on the basis of the block encoding mode information.

[11] The method according to claim 10, wherein the block encoding mode information indicates a merging method of encoding an MxN merged combined block or a separation method of encoding an MxN separate combined block.

[12] The method according to claim 11, wherein the image data of the MxN block is encoded using 4:2:2 as a sampling frequency ratio when the block encoding mode information indicates the merging method, and the image data of the MxN block is encoded using 4:2:0 as the sampling frequency ratio when the block encoding mode information indicates the separation method.

[13] A method of decoding a stereoscopic image, the method comprising the steps of:

(a) decoding motion estimation mode information indicating a motion estimation method for image data of an MxN block of a merged combined image, from an encoded bit stream of the merged combined image which is generated from odd or even vertical lines of a left image and even or odd vertical lines of a right image; and

(b) adaptively decoding the image data of the MxN block on the basis of the motion estimation mode information.

[14] The method according to claim 13, wherein the motion estimation mode information is information indicating the separation method, and wherein the image data of the MxN is reconstructed using two or more motion vectors in the step of (b).

[15] A method of decoding a stereoscopic image, the method comprising the steps of:

(a) decoding block encoding mode information indicating an encoding method for image data of an MxN block of a merged combined image, from an encoded bit stream of the merged combined image which is generated from all vertical lines of a left image and all vertical lines of a right image; and

[16] A method of encoding a stereoscopic image including a left image and a right image, the method comprising the steps of:

(a) generating a first merged combined image from odd or even vertical lines of the left image and even or odd vertical lines of the right image;

(b) adaptively encoding an MxN merged combined block of the first merged combined image in the unit of a block to generate first encoding data and encoding an MxN separate combined block corresponding to the MxN merged combined block in the unit of a block to generate second encoding data; and

(c) outputting one encoding data selected from the first encoding data and the second encoding data and block encoding mode information indicating an encoding method for the selected encoding data.

[17] The method according to claim 16, further comprising the steps of, before the step of (b): performing a first motion estimation process of estimating a motion using the

MxN merged combined block and a second motion estimation process of estimating a motion using the MxN separate combined block, respectively; selecting one motion estimation process of the first motion estimation process and the second motion estimation process; and making a motion compensation in accordance with the selected motion estimation process.

[18] The method according to claim 17, wherein motion estimation mode information indicating the selected motion estimation process is output in the step of selecting the motion estimation process.

[19] An apparatus for encoding a stereoscopic image including a left image and a right image, the apparatus comprising the steps of:

(a) a merged combined image generating unit for generating a first merged combined image from odd or even vertical lines of the left image and even or odd vertical lines of the right image;

(b) a block encoding unit for adaptively encoding an MxN merged combined block of the first merged combined image in the unit of a block to generate first encoding data and encoding an MxN separate combined block corresponding to the MxN merged combined block in the unit of a block to generate second encoding data; and

(c) a first selection unit for selecting one of the first encoding data and the second encoding data and outputting block encoding mode information indicating an encoding method for the selected encoding data.

[20] The apparatus according to claim 19, further comprising: a motion estimation unit for performing a first motion estimation process of estimating a motion using the MxN merged combined block and a second motion estimation process of estimating a motion using the MxN separate combined block, respectively; a second selection unit for selecting one motion estimation process of the first motion estimation process and the second motion estimation process; and a motion compensation unit for making a motion compensation in accordance with the selected motion estimation process.

[21] The apparatus according to claim 20, wherein the second selection unit outputs motion estimation mode information indicating the selected motion estimation process.

[22] An apparatus for decoding a stereoscopic image, the apparatus comprising: a demultiplexing unit for separating image data of an MxN block of a merged combined image and block encoding mode information indicating an encoding method for the image data of the MxN block, from a bit stream obtained by encoding the merged combined image which is generated from odd or even vertical lines of a left image and even or odd vertical lines of a right image; and a block decoding unit for decoding the block encoding mode information and adaptively decoding the image data of the MxN block in accordance with the decoded block encoding mode information.

[23] The apparatus according to claim 22, wherein the block encoding mode information indicates a merging method of encoding an MxN merged combined block or a separation method of encoding an MxN separate combined block.

[24] The apparatus according to claim 22, wherein the demultiplexing unit further separates motion estimation mode information indicating indicates a motion estimation method for the image data of the MxN block, and wherein the apparatus further comprises a motion compensation unit for performing a motion compensation process in accordance with the motion estimation mode information.

[25] A method of encoding a stereoscopic image including a left image and a right image, the method comprising the steps of:

(a) generating a first merged combined image from odd or even horizontal lines of the left image and even or odd horizontal lines of the right image; and

(b) performing an adaptive encoding process by the use of an MxN merged combined block of the first merged combined image and an MxN separate combined block corresponding to the MxN merged combined block.