KR100780840B1 - A temporal prediction apparatus and method for coding multi-view video based on layer-depth image - Google Patents

Abstract

A temporal prediction apparatus and method for coding multi-view video based on layered depth image are provided to predict a layered depth image sequence using temporal prediction which can acquire the largest coding gain, thereby improving overall multi-view video coding efficiency. A temporal prediction apparatus for coding multi-view video based on layered depth image includes an LDI(Layered Depth Image) frame generating unit(110), a partial image generating unit(120), an image jointing unit(130), a partial color image predicting unit(140), and a partial depth image predicting unit(150). The LDI frame generating unit generates a layered depth image sequence. The partial image generating unit generates a partial color image and a partial depth image. The image jointing unit joints the partial color image and the partial depth image as one image. The partial color image predicting unit predicts a partial color image. The partial depth image predicting unit predicts a partial depth image.

Description

Temporal Prediction Apparatus and Method for Hierarchical Depth Image Coding of Multi-view Video TECHNICAL VIEW APPARATUS AND METHOD FOR CODING MULTI-VIEW VIDEO BASED ON LAYER-DEPTH IMAGE

1 is a block diagram illustrating a temporal prediction apparatus for hierarchical depth image encoding of a multiview video according to an exemplary embodiment of the present invention.

2 is a diagram illustrating a visual prediction process of a hierarchical depth image for encoding a multiview video including depth information according to an embodiment of the present invention.

FIG. 3 is a diagram for describing a method of predicting by combining the color layer images into one of the temporal prediction methods of the layer depth images according to an embodiment of the present invention.

FIG. 4 is a diagram for describing a method of predicting by combining a depth layer image into one of the temporal prediction methods of the layer depth image according to an embodiment of the present invention.

5 to 8 are diagrams for describing a method of predicting a color layer image on a time axis without combining the color layer images into one of the temporal prediction methods of the layer depth image according to an exemplary embodiment of the present invention.

9 to 10 are diagrams for describing a method of predicting a time layer of a depth layer image without combining the depth layer image in one of the temporal prediction methods of the layer depth image according to an embodiment of the present invention.

<Description of the code | symbol about the principal part of drawing>

100/1 to 100 / n: camera 110: LDI frame generation unit

120: partial image generating unit 130: image combining unit

140: partial color image estimation unit 150: partial depth image estimation unit

The present invention relates to a multi-view video service system, and more particularly, to a temporal prediction apparatus and method for hierarchical depth video encoding of a multi-view video capable of expressing a multi-view video based on hierarchical depth video and improving encoding efficiency. will be.

Multi-view video is used in various applications to provide more realism, but requires a huge amount of data, so huge bandwidth is required to transmit it. By using the concept of hierarchical depth image using three-dimensional information rather than the extension of the conventional two-dimensional video coding method, it is possible to protect the multi-view video with a relatively low bandwidth.

Hierarchical depth image is a method of representing an object using an array of pixels visible from a single camera position, unlike a method of representing a three-dimensional model using a mesh. Each pixel represents a color and a distance from a pixel to a camera. It is represented by depth information and some other characteristic information that supports rendering. That is, the hierarchical depth image is composed of pixels similar to a general two-dimensional image, but each pixel has not only color information but also depth information and additional information used for rendering. Therefore, an image of any viewpoint can be easily generated within a limited field of view using the hierarchical depth image configured at one viewpoint. Specifically, the information constituting the LDI includes color information of Y, Cb, Cr, Alpha, depth information indicating the distance between the camera and the object, and a splat table index used to determine the size of the pixel during rendering. to be. Since one LDI pixel uses a total of 63 bits per pixel to contain all of this information, one LDI contains data ranging from a few MB to as many as tens of MBs.

Meanwhile, a conventional multiview video encoding technique uses H.264 / AVC based. Use the foundation. However, H.264 / AVC-based multi-view video encoding techniques are all extensions of two-dimensional video encoding technology and do not use three-dimensional information. In order to give the user or the viewer a more realistic depth effect and immersion, the three-dimensional information included in the multi-view video should be effectively used, but conventionally, there is a problem of using only two-dimensional information.

Therefore, there are many limitations when providing various services using multi-view video such as 3D TV or free view TV.

An object of the present invention is to solve the problems of the prior art, by generating a hierarchical depth image sequence by using the depth information that is three-dimensional information included in the multi-view video, and predicting it in the time axis, thereby improving the coding efficiency The present invention provides a temporal prediction apparatus and method for hierarchical depth image encoding of multiview video.

In order to achieve the above object, the present invention provides a temporal prediction apparatus for hierarchical depth image encoding of a hierarchical depth image based hierarchical depth image. Layered Death Image (LDI) frame generation unit for generating a hierarchical hierarchical depth image sequence configured by the time zone, and a partial color image for each time zone using a time zone color and a depth layer in the hierarchical depth image sequence generated by the LDI frame generator And a partial image generator which generates a partial depth image, an image combiner which combines the partial color image and the partial depth image generated by the partial image generator into one image, and one image from the image combiner. Partial color image on the time axis using I, B, B, P structure Receiving service prediction portion predicting unit and color image, a depth image-combining section into one image from the image combination unit to include I, part depth image predicting unit to predict the structure using the P portion of the depth images in the time axis.

In addition, the present invention is a temporal prediction apparatus for hierarchical depth image coding of hierarchical depth image based hierarchical depth image, and hierarchical depth for each time point composed of color and depth layer of each viewpoint for multiview image having depth information. A layered death image (LDI) frame generation unit for generating an image sequence, and a partial color image and a partial depth image for each time zone, respectively, using a time zone color and depth layer in the hierarchical depth image sequence generated by the LDI frame generator. A partial color image predictor configured to predict the partial color image on a time axis by applying an I, P, and P prediction structure to the partial color image for each time zone provided by the partial image generator; By applying I, P prediction structure to the partial color image for each time zone provided by the generation unit, the partial depth is zero in the time axis. And a partial depth image predictor for predicting an image.

In another aspect, the present invention is a temporal prediction method for hierarchical depth image encoding of a multiview video based on a hierarchical depth image. Generating a depth image sequence, generating a partial color image and a partial depth image for each time zone by using a time slot color and a depth layer in the generated hierarchical depth image sequence, and generating the partial color image Combining a single image and predicting the partial color image using the I, B, B, and P structures on the time axis; combining the generated partial depth images into one image, and then using the I, P structure Predicting the partial depth image.

In addition, the present invention is a temporal prediction method for hierarchical depth image encoding of a multi-view video based on a hierarchical depth image. The hierarchical depth for each multi-view video having depth information consists of color and depth hierarchies at each viewpoint. Generating an image sequence, generating a partial time-phase partial color image and a partial depth image by using a time-phase color and a depth layer in the generated hierarchical depth image sequence, and generating the partial time-phase partial color image Predicting a partial color image in each time axis by applying an I, P, and P prediction structure to it, and predicting a partial depth image in the time axis by applying an I, P prediction structure to the generated time-scale partial color image. Include.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a temporal prediction apparatus in a multi-view video service system based on a layer depth image, according to an exemplary embodiment.

Referring to FIG. 1, the temporal prediction apparatus includes n cameras 100/1 to 100 / n, a layered death image (LDI) frame generation unit 110, and a partial image generation unit ( 120, an image combiner 130, a partial color image predictor 140, and a partial depth image predictor 150.

The LDI frame generation unit 110 generates a hierarchical depth image sequence composed of color and depth hierarchies of each viewpoint for a multiview video having depth information obtained from n cameras 100/1 to 100 / n. Each hierarchical depth image is represented by a set of component images having a maximum of n hierarchies. In this case, since each hierarchical depth image includes a color and a depth layer, the hierarchical depth image is represented by a total of 2n image sets.

In addition, the LDI frame generator 110 predicts a hierarchical depth image according to a time change when generating a hierarchical depth image sequence using a multi-view video obtained from n cameras 100/1 to 100n. Create That is, as shown in FIG. 2, the hierarchical depth image sequence is hierarchical depth image (hierarchical) generated at each time T (T = 0,1,2, ..., M) according to the change of time of the right portion. Depth image 0, hierarchical depth image 1, ..., hierarchical depth image M) are arranged, and on the left part of the hierarchical depth image sequence, a multi-view video obtained from n cameras 100/1 to 100 / n Each video includes a color and depth hierarchy. That is, one hierarchical depth image is composed of at most n layers, and each layer is composed of a color layer part and a depth layer part.

The partial image generator 120 generates a partial color image and a partial depth image for each time zone by using a color and depth layer for each time in the hierarchical depth image sequence.

First, the partial image generator 120 extracts only a color layer for each time zone from the hierarchical depth image generated by the LDI frame generator 102 to generate a partial color image. As shown in FIG. The layer is extracted, and a partial color image including an image obtained by collecting horizontal and vertical data about the color layer for each time zone and an image filled with the empty pixel layer is generated.

The partial image generator 120 extracts only a depth layer for each time zone from the hierarchical depth images generated by the LDI frame generator 110 to generate a partial depth image. As shown in FIG. The layer is extracted, and a partial depth image including an image obtained by collecting horizontal and vertical data about the depth layer for each time zone and an image filled with the empty pixel layer is generated.

Here, the method of filling the empty pixel layer when generating the partial depth image and the color image fills the empty pixel layer by using the front layer information of the layer having the empty pixel or by using other interpolation methods.

In the partial image generator 120, horizontal and vertical data collections are arranged to the right with respect to an empty layer, that is, layers without a partial color image, and arranged with a pixel to the left.

Thereafter, the image combiner 130 combines the partial depth and the color image generated by the partial image generator 120 into one image, and the partial color image predictor 140 combines the partial color image combined into one image. A partial depth image combined with one image is output to the partial depth image predictor 150.

The partial color image predictor 140 predicts the partial color image on the time axis using I, B, B, and P structures. In the method of predicting a partial color image on the time axis using the I, B, B, and P structures, as shown in FIG. 3, after defining the partial color image that comes in at the first time (T = t) as I, T = t A partial color image input to +3 is predicted P based on I, and a partial color image of T = t + 1 and t + 2 is predicted to B based on I and P.

As described above, in the time-base partial color image prediction method using I, B, B, and P structures, the maximum number of B is two.

As illustrated in FIG. 4, the partial depth image predictor 150 predicts the partial depth image using the I and P structures of H.264 / AVC, that is, the partial depth image of the time (T = t). Assuming " I ", depth information next time (T = t + 1) is predicted as " P ". This depth information is important information for applications such as stereoscopic display, three-dimensional TV, free-view TV.

In the present invention, when the depth information is predicted by using the "I, P" structure as an example, if the accurate depth information is not necessary according to the application range of the system "I" used in the partial color image prediction unit 140 , B, B, P "structure can be used to predict the depth information.

Here, since the same scene is photographed with n cameras 100/1 to 100 / n, the number of generated hierarchies is almost constant. If the number of generated layers differs according to time zones, the number of layers at different times (t = 1, 2, 3…) is adjusted to match the number of generated layers in the first time (t = 0). That is, if the number of layers at different times is different from the number of layers at the first time, arbitrary layers are combined so that the number of layers at the corresponding time corresponds to the number of layers generated at the first time (t = 0). At this time, the resolution of the combined layers is set equal to the resolution of the image at the first time (t = 0).

Although an embodiment of the present invention has been described as a structure in which the generated partial color and depth images are combined into one image and then predicted in the time axis, as another embodiment of the present invention, each of them may be processed separately without combining into one image. . That is, the temporal prediction apparatus according to another embodiment of the present invention has the same configuration as the temporal prediction apparatus shown in FIG. 1 in which the image combiner 130 is omitted, and the partial image generator 120 generates the partial color image. Is output to the partial color image estimation unit 140 and the generated partial depth image is output to the partial depth image estimation unit 150.

In this case, the partial color image estimator 140 predicts the partial color image using I, P, and P structures, that is, three time zones as one bundle, and the partial color image prediction method may be classified into four cases.

First, as shown in FIG. 5, the number of layers generated when T = t is less than the number of layers created when T = t + 1, and the number of layers when T = t + 1 is T = t + 2. If there is more than the number of layers, if the image does not exist in the previous time, the current image is encoded with I and then the layer of each time zone (T = t + 1, t + 2) corresponding to the layer encoded with I. Is encoded as P.

In the second case, as shown in FIG. 6, the number of layers created when T = t is smaller than the number of layers created when T = t + 1, and the number of layers when T = t + 1. When = t + 2, it is less than the number of layers. If there is no image in the previous time, each time zone corresponding to the layer encoded with I after encoding the current image with I is encoded. The code of +2) is encoded by P.

In the third case, as shown in FIG. 7, the number of layers created when T = t is greater than the number of layers created when T = t + 1, but the number of layers when T = t + 1 is T = t. When t + 2, the number is less than the number of layers. If there is a previous video, the current video is encoded by P, otherwise, I is encoded.

In the fourth case, as shown in Fig. 8, the number of layers created when T = t is the largest, the number of layers when T = t + 1, and the number of layers when T = t + 2. In a few cases, I, P, P prediction is performed from partial color images at T = t when T = t + 2.

On the other hand, since the partial depth image information is important information for selecting a suitable pixel for multi-view reconstruction, the partial depth image predictor 150 predicts the partial depth image on the time axis using I, P, and P prediction structures.

That is, the partial depth image predictor 150 predicts the partial depth image of the time zone by applying the I and P prediction structures to the partial depth image provided from the partial image generator 120.

For example, as shown in FIG. 9, when the number of layers generated when T = t is larger than the number of layers generated when T = t + 1, the image is predicted using a direct I. P prediction structure. Otherwise, (when the number of layers is greater than the number of layers when T = t + 1), as shown in FIG. 10, encoding the current image, which is not predictable from the previous time, as “I”. do.

According to the present invention, by generating a hierarchical depth image sequence using depth information, which is three-dimensional information included in a multi-view video, and predicting the hierarchical depth image sequence, the coding efficiency may be improved.

The present invention is not limited to the above-described specific preferred embodiments, and various modifications can be made by any person having ordinary skill in the art without departing from the gist of the present invention claimed in the claims. Of course, such changes will fall within the scope of the claims.

As described above, the present invention can improve the overall multiview video encoding efficiency by predicting a sequence of hierarchical depth images using temporal prediction, which is a prediction method that can obtain the most coding gain in a multiview video.

Claims (16)

A temporal prediction apparatus for hierarchical depth image coding of multi-view video based on hierarchical depth image, A layered death image (LDI) frame generation unit for generating a time-phase hierarchical depth image sequence composed of color and depth hierarchies of each viewpoint for a multiview video having depth information; A partial image generator for generating a partial color image and a partial depth image for each time zone by using a color time zone and a color depth layer in the hierarchical depth image sequence generated by the LDI frame generator; An image combiner for combining the partial color image and the partial depth image generated by the partial image generator into one image, respectively; A partial color image predictor which receives a partial color image combined into one image from the image combiner and predicts the partial color image on the time axis using I, B, B, and P structures; A partial depth image predictor for receiving a partial depth image combined into one image from the image combiner and predicting the partial depth image on the time axis using I and P structures. A temporal prediction apparatus for hierarchical depth image encoding of a multiview video including a. The method of claim 1, The partial image generator, Temporal depth image coding for hierarchical depth image encoding of a multi-view video, comprising generating a partial color image and a depth image including an image obtained through horizontal and vertical data collection for each time and an image filled with empty pixels. Prediction device. The method of claim 2, The partial image generator, The apparatus for temporal prediction of hierarchical depth image coding of a multi-view video, characterized in that the empty pixel is filled using the front layer information of the layer having the empty pixel. The method of claim 1, The partial image generator, If the number of layers of the partial color image and the depth image for each time zone is different, the temporal prediction apparatus for hierarchical depth image coding of the multi-view video, characterized in that the number of layers of different time zones is adjusted according to the number of layers of the first time zone. The method of claim 4, wherein And a new hierarchical resolution of the other time zone is generated in accordance with the resolution of the initial time zone image. The method of claim 1, The partial depth image predictor, A hierarchical depth image encoding of a multi-view video is provided by receiving a partial depth image combined into one image from the image combiner and predicting the partial depth image for each time zone using I, B, B, and P structures. Temporal prediction device. A temporal prediction apparatus for hierarchical depth image coding of a hierarchical depth image based multi-view video service system, A layered death image (LDI) frame generation unit for generating a time-phase hierarchical depth image sequence composed of color and depth hierarchies of each viewpoint for a multiview video having depth information; A partial image generator for generating a partial color image and a partial depth image for each time zone by using a color time zone and a color depth layer in the hierarchical depth image sequence generated by the LDI frame generator; A partial color image predictor for predicting a partial color image on a time axis by applying I, P, and P prediction structures to the partial color image for each time zone provided by the partial image generator; Partial depth image predictor for predicting a partial depth image on a time axis by applying I and P prediction structures to partial color images for each time zone provided from the partial image generator A temporal prediction apparatus for hierarchical depth image encoding of a multiview video including a. The method of claim 7, wherein The partial depth image predictor, Temporal prediction for hierarchical depth image coding of multi-view video, wherein the partial depth image is predicted by applying I, P, and P prediction structures to the partial color image for each time zone provided by the partial image generator. Device. A temporal prediction method for hierarchical depth image coding of multi-view video based on hierarchical depth image, Generating a hierarchical hierarchical depth image sequence composed of color and depth hierarchies of each viewpoint for a multiview video having depth information; Generating a partial color image and a partial depth image for each time zone by using the color and depth layer for each time zone in the generated hierarchical depth image sequence; Combining the generated partial color images into one image and predicting the partial color images on the time axis using I, B, B, and P structures; Combining the generated partial depth images into one image, and predicting the partial depth images on the time axis using I and P structures. A temporal prediction method for hierarchical depth image encoding of a multiview video including a. The method of claim 9, Generating the partial color image and the partial depth image, respectively, Temporal depth image coding for hierarchical depth image encoding of a multi-view video, comprising generating a partial color image and a depth image including an image obtained through horizontal and vertical data collection for each time and an image filled with empty pixels. Forecast method. The method of claim 10, Generating the partial color image and the partial depth image, respectively, The method of claim 1, wherein the empty pixels are filled by using information on the front layer of the layer having the empty pixels. The method of claim 9, Generating the partial color image and the partial depth image, respectively, And when the number of layers of the partial color image and the depth image of each time zone is different, adjusting the number of layers of different time zones according to the number of layers of the first time zone. The method of claim 12, The resolution of the new layer in the other time zone is generated by matching the resolution of the first time zone image to the temporal prediction method for hierarchical depth image encoding of a multi-view video. The method of claim 12, Predicting the partial depth image, A method of temporal prediction for hierarchical depth image coding of a multiview video, comprising predicting the partial depth image on the time axis using I, B, B, and P structures of the partial depth image combined into one image. A temporal prediction apparatus for hierarchical depth image coding of multi-view video based on hierarchical depth image, Generating a hierarchical hierarchical depth image sequence composed of color and depth hierarchies of each viewpoint for a multiview video having depth information; Generating a partial color image and a partial depth image for each time zone by using the color and depth layer for each time zone in the generated hierarchical depth image sequence; Predicting a partial color image on each time axis by applying an I, P, and P prediction structure to the generated partial color image for each time zone; Predicting a partial depth image on a time axis by applying an I and P prediction structure to the generated partial color image for each time zone A temporal prediction method for hierarchical depth image encoding of a multiview video including a. The method of claim 15, Predicting the partial depth image, A time prediction method for hierarchical depth image encoding of a multi-view video, characterized by predicting partial depth images for each time zone by applying I, P, and P prediction structures to the partial color images for each time zone.

Images