Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T15/00—3D [Three Dimensional] image rendering
  • G06T15/10—Geometric effects
  • G06T15/20—Perspective computation
  • G06T15/205—Image-based rendering
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/128—Adjusting depth or disparity
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N2213/00—Details of stereoscopic systems
  • H04N2213/005—Aspects relating to the "3D+depth" image format

Definitions

• the present invention relates to a linear decorrelation method and apparatus that adjust a probability distribution of a layered depth image to improve coding efficiency in encoding and decoding the layered depth image.
• a multi-view video has been used in various applications to provide more realistic services, but a great amount of data is required so that an extremely wide bandwidth is needed to transmit the data. Therefore, to solve this problem, a layered depth image ("LDI") method requiring a relatively narrow bandwidth can be utilized.
• LDM layered depth image
• LDI represents a 3-D object with an array of pixels seen from a single camera position.
• Each LDI pixel is represented by its color, depth that is the distance of the pixel to the camera, and some other property information supporting LDI rendering.
• the LDI is composed of pixels similar to a typical 2-D image, but each pixel has color information as well as depth information and additional information that supports rendering. Therefore, any view image within a certain view angles can be easily rendered by using the LDI, which is constructed from a single view.
• the LDI contains color information on Y, Cb, Cr, and Alpha, depth information representing a distance between a camera and an object, and a splat table index used to support various pixel sizes upon rendering.
• Each LDI pixel contains 63-bit information in total to include all the information, so that one sheet of LDI includes data from several megabytes to several tens of megabytes.
• LDI is divided into multiple layers, each of which contains a mask indicating the existence of pixel in the layer.
• LDI is characterized in that the distribution of pixels becomes sparser towards the back layer. Such a phenomenon becomes more noticeable, as the number of LDI layers becomes more.
• IEEE TRANSACTIONS OF IMAGE PROCESSING, VOL., 12, NO.3, 2003/3) discloses a data aggregation method as a preprocessing process prior to compression.
• Data aggregation which uses a feature of LDI that the distribution of pixels becomes sparser towards the back layer, is performed to aggregates pixels distributed in each layer.
• Such simple data aggregation does not consider correlation of LDI data. [6] Therefore, there is a need to improve the coding efficiency of LDI data by removing redundant (duplicated) information from a highly correlated LDI data, converting them into de-correlated data, and then encoding the non-correlated data. Disclosure of Invention Technical Problem
• the object of the present invention is to propose a linear decorrelation process, which is a new preprocessing process to remove redundant depth information prior to performing data aggregation and a method for encoding multiview video using LDI.
• Other object of the present invention is to improve coding efficiency in encoding process using the LDI, by making a distribution of depth information of the LDI data highly skewed around a median, through the linear decorrelation.
• the present invention provides a method and an apparatus for encoding and decoding multi-view video using LDI.
• a method of encoding multiview video data using LDI includes: (i) generating the LDI including multiple layers by using color and depth information of each viewpoint image of the multi-view video; (ii) performing linear decorrelation in each layer of the LDI; (iii) performing data aggregation in each linearly-decorrelated layer of the LDI; and (iv) encoding the aggregated data in each layer of the LDI to generate an encoded LDI bit stream.
• the step (ii) may includes, for each of all pixels in each layer of the LDI, calculating a minimum distance between a line connecting two previous pixels and a depth value of a current pixel to replace the depth value of the current pixel with the minimum distance. Also, in the step (b), when a depth value of the current pixel does not exist, an average depth value of the two previous pixels may be used as the depth value of the current pixel.
• the information for compensating information loss occurred in the LDI generation step may be transmitted to a decoding apparatus, together with the encoded LDI bit stream, so that images close to the original ones may be reconstructed.
• a method of decoding a multi- view video comprising the steps of decoding an encoded LDI bit stream; decoding a bit stream of residual information between an original multi-view video and a multi-view video reconstructed from the encoded LDI bit stream; reconstructing the multi-view video based on the decoded LDI bit stream and residual information is provided.
• the image at the corresponding viewpoint may be reconstructed only.
• FIG. 1 shows a typical LDI structure.
• FIG. 2 is a schematic diagram of a multi-view video LDI-based encoding/decoding apparatus according to an embodiment of the present invention
• FIG. 3 is a diagram illustrating how a LDI is generated from multi-view video data
• FIG. 4 shows how to perform linear decorrelation on the LDI layer in which all the pixels have depth values, according to the present invention
• FIG. 5 shows how to perform linear decorrelation on the LDI layer in which some of the pixels do not have depth values, according to the present invention
• FIG. 6 is a flowchart showing the linear decorrelation process according to a preferred embodiment of the present invention. Mode for the Invention
• FIG. 1 shows a typical LDI structure.
• the LDI includes an array of pixels seen from a single LDI camera position, together with multiple layers based on any viewpoints.
• the rays intersect with an object at a plurality of points, which are ordered from the front to the back.
• the first intersection points constitute the first LDI layer; the second intersection points constitute the second layer, and so on.
• Each LDI layer is separated into individual components: luminance, color, transparency and depth. Further, component image of each layer is compressed separately. In order to increase a compression rate, data aggregation is performed to aggregate data on the same layer, so that data are more compactly distributed.
• FIG. 2 is a schematic diagram of a multi-view video LDI-based encoding/decoding apparatus according to an embodiment of the present invention.
• the apparatus 210 includes a LDI generation unit 201, a linear decorrelation unit 202, a data aggregation unit 203, an LDI encoding unit 204, an LDI decoding unit 205, a multi-view video generation unit 206 and residual information encoding unit 207.
• the LDI generation unit 201 generates a LDI, which is composed of multiple layers, by 3-D warping of multiview video images with depth information, which uses color and depth information of each image.
• a LDI which is composed of multiple layers, by 3-D warping of multiview video images with depth information, which uses color and depth information of each image.
• Fig. 3 while the images with depth information at different camera viewpoints C and C are warped into one with depth information at a common viewpoint C , when the warped pixels are placed in the same pixel location, their depth values are compared. If the difference between depth values is less than predefined threshold, they are merged. Otherwise, a new layer having the average depth value of the two pixels is created.
• the former case is shown as 'c' and 'd' in Fig. 3. Since algorithms for generating a LDI are well known to those skilled in the art, a detailed description thereof will be omitted in this spec ⁇ ification.
• the data aggregation unit 203 performs the LDI data aggregation in each LDI layer, in order to reduce distribution of depth values. Since data aggregation process is disclosed in the above-article "Compression of the layered depth image" (J. Duan and J. Li, IEEE TRANSACTIONS OF IMAGE PROCESSING, VOL., 12, NO.3, 2003/3), it will be omitted herein.
• the LDI encoding 204 encodes the data aggregated toward a certain direction in a space.
• the encoded LDI bit stream will be transmitted through a communication channel or a storage medium to a multi-view video decoding apparatus 220.
• the reconstructed images may have a residual with the original images. This is due to information loss during the LDI generation. Accordingly, it is required to separately transmit information for compensating such information loss to a multi-view video decoding apparatus 220, in order to reconstruct high-quality images close to the original ones.
• the multi-view video encoding apparatus 210 may additionally include LDI decoding unit 205, multi- view image generation unit 206 and residual information encoding unit 207.
• LDI decoding unit 205 receives the encoded LDI bit stream from the LDI encoding unit 204 and decodes it.
• the multi-view image generation unit 206 generates each of multi-view images from the decoded LDI data.
• the residual information encoding unit 207 calculates residual information between multi-view images generated by the multi- view image generation unit 206 and original multi-view images, encodes and transmits it to the multi-view video decoding apparatus 220.
• the multi-view video decoding apparatus 220 includes an LDI decoding unit 221, multi-view image generation unit 222 and residual information decoding unit 223.
• the LDI decoding unit 221 receives the encoded LDI bit stream from the multi-view video encoding apparatus 220 and decodes it.
• the residual information decoding unit 223 receives the encoded residual information bit stream and from the multi-view video encoding apparatus 220 and decodes it.
• the multi-view image generation unit 222 generates each of the multi-view images close to the original images, using the LDI data decoded by the LDI decoding unit 221 and the residual information decoded by the residual information decoding unit 223.
• a user can select which viewpoint will be reconstructed and, the multi-view image generation unit 222 can generate the image corresponding to the selected viewpoint, in response to the selection.
• FIG. 4 shows how to perform linear decorrelation on the LDI layer in which all the pixels have depth information, according to the present invention.
• the one- dimensional (1-D) depth value of a pixel may be considered as the two-dimensional (2-D) value point.
• the minimum distance between a line passing through previous two points, which represent the depth values of the previous two pixels, and the depth value of a current pixel is calculated; and then the depth value of the current pixel is replaced with the minimum distance.
• FIG. 5 shows how to perform linear decorrelation on the LDI layer in which some of the pixels do not have depth values, according to the present invention.
• the average depth value of the previous two points is inserted into a depth value of the pixel, which does not have a depth value.
• the minimum distance between a line passing through the previous two points and the depth value of the current pixel is calculated and then the depth value of the current pixel can be replaced with the minimum distance.
• the some of the previous two pixels do not have depth values.
• the depth value of a first pixel when the depth value of a first pixel does not exist, the depth value of the first pixel is filled with '0', and when the depth value of a second pixel does not exist, the depth value of the second pixel is filled with that of the first pixel.
• the depth value of the third pixel can be filled with the average depth value of the previous two pixels. Then, the minimum distance is calculated using this average value as a depth value of the current pixel. In other words, the depth values of all the pixels on each LDI layer are filled and then the minimum distance is calculated.
• the minimum distance, d, between a line passing through the previous two points, which represent the depth values of the two previous pixels (for example, A(x z ) and B(x , z )) and a current point, which represents the depth value of the current i i pixel (for example, C(x , z )) can be computed by
• a 1 - represents A(-z , x ). Since the depth value does not exist in the position of C, as described above, the average value of the previous two depth values is inserted into z . With this, the variance distribution of the depth values can be reduced.
• FIG. 6 is a flowchart of the linear decorrelation process according to an embodiment of the present invention.
• step 610 it is checked whether all the LDI pixels have depth values on the same LDI layer.
• step 620 it is determined if the pixel having no depth value is the first pixel in step 620. If it is the first pixel, the value thereof is filled with '0' in step 630.
• step 640 it is determined if the pixel having no depth value is the second pixel. If it is, the value of the second pixel is filled with the depth value of the first pixel in step 650.
• the depth value of the corresponding pixel is filled with the average depth value of the previous two points, which represent the depth values of the previous two pixels, in step 660.
• the steps 620 to 660 are performed to fill the depth values of the corresponding pixels.
• step 670 the minimum distance between a line passing through the previous two points and a depth value of a current pixel is calculated, and the depth value of the current pixel is replaced with the minimum distance.
• the present invention can be provided as one or more computer readable medium implemented on one or more products.
• the products may be a floppy disk, a hard disk, a CD ROM, a flash memory card, a PROM, a RAM, a ROM, or a magnetic tape.
• a computer readable program can be implemented in any programming language. Some examples of available languages include C, C++, or JAVA.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• Computing Systems (AREA)
• Geometry (AREA)
• Computer Graphics (AREA)
• General Physics & Mathematics (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)

Priority Applications (1)

Applications Claiming Priority (4)

Publications (1)

Family

ID=36148536

Family Applications (1)

Country Status (3)

Cited By (11)

Families Citing this family (3)

Family Cites Families (3)

• 2005
  • 2005-10-13 EP EP05809005A patent/EP1800493A4/en not_active Withdrawn
  • 2005-10-13 WO PCT/KR2005/003418 patent/WO2006041261A1/en active Application Filing
  • 2005-10-13 KR KR1020050096700A patent/KR100714068B1/ko not_active IP Right Cessation

Non-Patent Citations (6)

Also Published As

Similar Documents

Legal Events