KR20140124043A - A method of determining structure for separating prediction unit and an apparatus using it - Google Patents

Abstract

A method for determining a prediction block split structure according to an embodiment of the present invention comprises the steps of: receiving image data including depth information; generating object information based on the depth information included in the image data; and determining a prediction block split structure for the image data based on the object information.

Description

[0001] The present invention relates to a method and a device for determining a prediction block division structure using depth information,

The present invention relates to a method of efficiently encoding and decoding an image using a depth information image.

3D video provides users with a stereoscopic effect as if they are seeing and feeling in the real world through a 3D stereoscopic display device. As a result of this research, the Joint Collaborative Team on 3D Video Coding Extension Development (JCT-3V), a joint standardization group of ISO / IEC Moving Picture Experts Group (MPEG) and ITU-T VCEG (Video Coding Experts Group) Video standards are in progress. The 3D video standard includes standards for advanced data formats and related technologies that can support playback of autostereoscopic images as well as stereoscopic images using real images and their depth information maps.

In the present invention, a method of determining a partition structure of an efficient encoded prediction block using a depth information image is proposed.

According to another aspect of the present invention, there is provided a method of determining a divided structure, the method including: receiving image data including depth information; Generating object information based on depth information included in the image data; And determining a prediction block division structure for the image data based on the object information.

The present invention can improve the coding efficiency of a general image. There is an effect of increasing the coding efficiency and decreasing the complexity of the actual image by using the method of coding the actual image using the depth information.

1 is a diagram showing an example of a basic structure and a data format of a 3D video system.
2 is a view showing an example of an actual image and a depth information map image.
3 is a block diagram showing an example of the configuration of the image encoding apparatus.
4 is a block diagram illustrating an example of a configuration of a 3D video encoding / decoding apparatus.
5 is a diagram showing an embodiment of the configuration of a bit stream.
6 is a diagram showing an example of a method of dividing an image into a plurality of units for encoding.
7 is a diagram showing an example of a method of dividing an image into a plurality of prediction units.
8 is a diagram showing an example of a block division structure according to a coding result.
9 is a view showing an embodiment of a variable division structure according to object information.
10 is a view showing an embodiment of a longitudinal direction PU division structure.
11 is a diagram showing an embodiment of a horizontal direction PU division structure.
12 is a diagram for explaining an embodiment of a method for determining a p value in a PU division structure.
13 is a diagram showing an embodiment of a method for determining a PU division structure based on object information.
FIG. 14 is a diagram showing a PU and TU division structure according to another embodiment of the present invention.

The following merely illustrates the principles of the invention. Thus, those skilled in the art will be able to devise various apparatuses which, although not explicitly described or shown herein, embody the principles of the invention and are included in the concept and scope of the invention. Furthermore, all of the conditional terms and embodiments listed herein are, in principle, intended only for the purpose of enabling understanding of the concepts of the present invention, and are not intended to be limiting in any way to the specifically listed embodiments and conditions .

It is also to be understood that the detailed description, as well as the principles, aspects and embodiments of the invention, as well as specific embodiments thereof, are intended to cover structural and functional equivalents thereof. It is also to be understood that such equivalents include all elements contemplated to perform the same function irrespective of the currently known equivalents as well as the equivalents to be developed in the future, i.e., the structure.

Thus, for example, it should be understood that the block diagrams herein represent conceptual views of exemplary circuits embodying the principles of the invention. Similarly, all flowcharts, state transition diagrams, pseudo code, and the like are representative of various processes that may be substantially represented on a computer-readable medium and executed by a computer or processor, whether or not the computer or processor is explicitly shown .

The functions of the various elements shown in the figures, including the functional blocks depicted in the processor or similar concept, may be provided by use of dedicated hardware as well as hardware capable of executing software in connection with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which may be shared.

Also, the explicit use of terms such as processor, control, or similar concepts should not be interpreted exclusively as hardware capable of running software, and may be used without limitation as a digital signal processor (DSP) (ROM), random access memory (RAM), and non-volatile memory. Other hardware may also be included.

In the claims hereof, the elements represented as means for performing the functions described in the detailed description include all types of software including, for example, a combination of circuit elements performing the function or firmware / microcode etc. , And is coupled with appropriate circuitry to execute the software to perform the function. It is to be understood that the invention defined by the appended claims is not to be construed as encompassing any means capable of providing such functionality, as the functions provided by the various listed means are combined and combined with the manner in which the claims require .

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings, in which: There will be. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 shows an example of a basic structure and a data format of a 3D video system.

The basic three-dimensional video system considered in the three-dimensional video standard is as shown in FIG. 1. As shown in FIG. 1, the depth information image being used in the three-dimensional video standard is encoded together with a general image and transmitted to the terminal as a bit stream. On the transmitting side, the image content of N (N ≥ 2) viewpoints is acquired by using a stereo camera, a depth information camera, a multi-view camera, and a two-dimensional image into a three-dimensional image. The obtained image content may include video information of the N view point, its depth map information, camera-related additional information, and the like. The video content at time point N is compressed using the multi-view video encoding method, and the compressed bitstream is transmitted to the terminal through the network. The receiving side decodes the transmitted bit stream using the multi-view video decoding method, and restores the N view image. The reconstructed N-view image generates virtual view images at N or more viewpoints by a depth-image-based rendering (DIBR) process. The generated virtual viewpoint images are reproduced in accordance with various stereoscopic display devices to provide stereoscopic images to the user.

The depth information map used to generate the virtual viewpoint image is a representation of the distance between the camera and the actual object in the real world (depth information corresponding to each pixel at the same resolution as the real image) in a fixed number of bits. As an example of a depth information map, FIG. 2 shows a "balloons" image (FIG. 2 (a)) and its depth information map (FIG. 2 (b)) being used in the MPEG standard of 3D video coding standard Giving. The depth information map shown in FIG. 2 actually represents depth information on the screen in 8 bits per pixel.

As an example of a method of encoding an actual image and its depth information map, the Moving Picture Experts Group (MPEG) and the Video Coding Experts Group (VCEG) having the highest coding efficiency among the video coding standards developed so far jointly standardize Encoding can be performed using HEVC (High Efficiency Video Coding). An example of the coding structure of HEVC is shown in FIG. As shown in FIG. 3, the HEVC includes various new algorithms such as coding unit and structure, inter prediction, intra prediction, interpolation, filtering, and transform.

3 is a block diagram showing an example of the configuration of the image encoding apparatus, and shows the configuration of an encoding apparatus according to the HEVC codec.

When the actual image and its depth information map are encoded, they can be independently encoded / decoded. When the actual image and the depth information map are encoded, they can be encoded / decoded in dependence on each other as shown in FIG. In one embodiment, an actual image can be encoded / decoded using an already-encoded / decoded depth information map, and conversely, a depth information map can be encoded / decoded using an already encoded / decoded real image.

FIG. 4 is a block diagram illustrating an example of a configuration of a 3D video encoding / decoding apparatus.

In the 3D video codec, the encoded bit stream of the real image and its depth information image can be multiplexed into one bit stream.

FIG. 5 shows an embodiment of the configuration of a bitstream.

Referring to FIG. 5, an encoded bit stream of an actual image and its depth information image can be independently constructed and multiplexed into a single bit stream. In FIG. 5, the integrated header information may include information on parameters necessary for decoding the actual image and its depth information image and generating the virtual view image. The header information of the depth information image may include information on parameters necessary for decoding the depth information image and generating the virtual viewpoint image. Header information of an actual image may include information on parameters necessary for decoding a general image.

Since the actual image and its depth information image have a great correlation, it is possible to suggest an algorithm considering the depth information image which can be used in both the encoder / decoder in the 3D video codec.

In this paper, we propose a video coding method in 3D video codec using depth information image. The basic concept of the present invention is to utilize a 3D video codec to encode an actual image using a depth information image acquired by a depth information camera in order to maximize a coding efficiency.

In one embodiment, when the objects of a general image are classified and encoded using the depth information image, the coding efficiency for a general image can be greatly increased. In the block-based encoding codec, a plurality of objects may exist in the block. Different encoding methods are applied to the objects based on the depth information image. .

In the case of HEVC, the coding is performed by dividing an image into units of LCU (LCU: Largest Coding Unit), which is a basic unit of a coding unit (CU).

Here, the CU plays a role similar to a macro block (MB), which is a basic block in H.264 / AVC, which is a conventional video codec, but unlike an MB having a fixed size of 16x16, You can set the size to. In addition, the divided LCU for encoding can be divided into several CUs having a smaller size than the LCU for efficient encoding of the image.

FIG. 6 shows an example of a method of dividing an image into a plurality of units for encoding, and is an example of a method of dividing a 64x64 LCU into a plurality of CUs.

Referring to FIG. 6, the partition structure for the LCU is determined as a CU having various sizes. The distribution of the CUs is determined after determining whether or not the CUs are divided into four CUs, which are reduced in half by half from the LCU size. In addition, the divided CUs can be recursively divided into four CUs, which are reduced by half the length in the same manner, and the CUs are divided up to a predetermined depth.

FIG. 7 shows an example of a method of dividing an image into a plurality of prediction units.

Referring to FIG. 7, a CU structure is represented by 2Nx2N of a square size, and a prediction unit (PU) is shown as a prediction structure divided in the CU.

The PU is a structure determined for more efficient encoding when performing encoding in intra or inter mode in units of CU. This is to predict and encode each PU independently by dividing it into PU units in the CU when accurate prediction can not be performed in the encoding in the unit of CU.

In the process of determining the PU division structure, the PU division structure shown in FIG. 7 is predicted for the current block to be predicted, and the predicted value is determined as a PU division structure having the block closest to the current block.

However, this process does not consider object information for neighboring blocks.

This is because, in the case of a two-dimensional image, object information in an image must be extracted through a two-dimensional image analysis if there is no depth camera, and therefore, the existing two-dimensional video coding method is not equipped with an object-based coding method at all.

However, if the object information can be considered using the Depth camera, the PU division structure can be determined as follows for the 2D image coding.

FIG. 8 shows an example of a divided structure of blocks according to a result of encoding. As a result of encoding one of the HEVC standardization experiment images through the HEVC reference software HM 7.0, the divided structure for an arbitrary LCU is shown.

In FIG. 8, the blocks A, B, and C represented by the dotted lines have the divided structures 2NxN, nRx2N, and 2NxN, respectively, and the division structure is determined along the boundaries of the objects. Therefore, we can propose a method to use object information for PU partition structure determination.

Also, as shown in FIG. 7, the present partition structure has 2NxN as a symmetric structure, and nLx2N, nRx2N, 2NxnU, and 2NxnD as Nx2N asymmetric structures.

For this reason, it is required to add a partition structure having various sizes in order to express the size of a suitable block along the boundary.

PU partition structure using depth information

The proposed additional partitioning structure is an extended type nPx2N of Nx2N, nLx2N, nRx2N, which divides the block in the vertical direction, and an extended type 2NxnP of 2NxN, 2NxnU, 2NxnD, which divides the block in the horizontal direction.

FIG. 9 shows an embodiment of a variable division structure according to object information.

Referring to FIG. 9, the proposed vertical division structure nPx2N is a structure suitable for dividing a block in the vertical direction by object information, and the horizontal division structure 2NxnP divides a block in the horizontal direction by object information It is a suitable structure.

In the present invention, additional PU division structures nPx2N and 2NxnP considering object information are proposed. By proposing a PU partition structure suitable for the object information according to the object information, the coding complexity in the partition structure determination process is reduced and the more accurate PU partition structure is determined, thereby improving the coding efficiency.

This proposed method can obtain the object information from the depth information of the image coding, and as mentioned in the previous section, it can be said that it needs to be developed in the situation of changing into the coding environment where the depth information can be easily acquired have.

In the proposed HEVC, the PU partition structure is a vertically partitioned structure nPx2N and a horizontally partitioned structure 2NxnP.

FIG. 10 illustrates an embodiment of a longitudinal PU partitioning structure.

Vertical division structure nPx2N is a structure for the case where the object distribution in the block is divided in the vertical direction.

Here, when the object distribution is divided into longitudinal directions, it is assumed that an arbitrary one object distribution can be divided into a vertical line or a diagonal line assuming that the block is divided into four equal parts as shown in FIG.

Figure 11 shows an embodiment of a lateral PU partitioning structure.

Horizontal division structure 2NxnP is a structure for the case where the object distribution in the block is divided into the horizontal direction. Here, when the object distribution is divided into the horizontal direction, it is a case that an arbitrary one object distribution can be divided into a horizontal line or a diagonal line, assuming that the block is divided into four equal parts as shown in FIG.

12 is a view for explaining an embodiment of a method of determining a p value in a PU division structure.

Referring to FIG. 12, the determination of the p value in the longitudinal partitioning structure nPx2N is performed by starting from p = 1 or an arbitrary value, and increasing to 1 or an arbitrary unit from the end of the object distribution to find a value having the best predicted value.

Likewise, in the horizontal direction division structure 2NxnP, the p value also increases from 1 or an arbitrary value to 1 or an arbitrary unit from the end of the object distribution to find a value having the best prediction value.

The PU division structure determination method according to an exemplary embodiment of the present invention is a method of determining a PU division structure by applying a division structure added based on object information.

When the prediction block is composed of the same object, the best division structure is selected in the existing PU division structure. In the case where the prediction block is not the same object, the division is performed in the direction of 2Nx2N, NxN, and nPx2N, , 2Nx2N, NxN, 2NxnP are selected, and when it can not be divided into the vertical direction or the horizontal direction, it is selected from the existing PU division structure.

FIG. 13 shows an embodiment of a method for determining a PU division structure based on object information.

A method of determining a PU division structure according to an embodiment of the present invention will be described with reference to FIG.

(1) Read the object information for the prediction block.

(2) The process of (3) is performed when the object information of the prediction block is the same object. Otherwise, the process of (4) is performed.

(3) The division structure having the best predicted value is selected through the rate-distortion measurement while the divided structures 2Nx2N, NxN, 2NxN, Nx2N, nLx2N, nRx2N, 2NxnU and 2NxnD are candidates.

(4) When the prediction block is divided vertically according to object information, 2Nx2N, NxN, 2NxnP are candidates for 2Nx2N, NxN, and nPx2N as candidates, and 2Nx2N, The segmentation structure having the best predicted value is selected and terminated. If it can not be divided into the vertical direction or the horizontal direction, the process of (3) is performed and the process is terminated.

FIG. 14 is a diagram showing a PU and TU division structure according to another embodiment of the present invention.

Referring to FIG. 14, a transformation and quantization method according to the above-described additional PU division structure is performed in a rectangle type transform and quantization method that is not a conventional rectangular shape.

When the PU partition structure is determined as nPx2N or 2NxnP as shown in FIG. 14, the proposed method can be determined and transformed and quantized according to the structure of the TU.

If the partition structure for the PU is determined to be nPx2N or 2NxnP, the TU structure for the block is determined as the existing TU structure and the additional TU structure as shown in FIG. 14, and the transformation and quantization are performed.

All of the above methods can be applied differently depending on the block size or the CU depth. The variable (i.e., size or depth information) for determining the coverage can be set to use a predetermined value by the encoder or decoder, use a predetermined value according to the profile or level, If the bit stream is described, the decoder may use this value from the bit stream. If the application range is different according to the CU depth, as shown in the table below, the method A) applies only to a depth above a given depth, B) the method applied only to a given depth or less, C) There can be a way.

Table 1 shows an example of the range determination method applying the methods of the present invention when the given CU depth is 2. (O: applied to the depth, X: not applied to the depth.)

CU depth indicating coverage Method A Method B Method C 0 X O X One X O X 2 O O O 3 O X X 4 O X X

When the methods of the present invention are not applied to all the depths, they may be indicated by using an optional flag, or a value one greater than the maximum value of the CU depth may be expressed by signaling with a CU depth value indicating the application range have.

In addition, the above-described method can be applied to color difference blocks differently depending on the size of a luminance block, and can be applied to a luminance signal image and a chrominance image differently.

Luminance block size Color difference block size Apply brightness Color difference application Methods 4 (4x4, 4x2, 2x4) 2 (2x2) O or X O or X 1, 2, ... 4 (4x4, 4x2, 2x4) O or X O or X I, 1, 2, ... 8 (8x8, 8x4, 4x8, 2x8, etc.) O or X O or X Every 1, 2, .. 16 (16x16, 16x8, 4x16, 2x16, etc.) O or X O or X La 1, 2, .. 32 (32x32) O or X O or X Ma 1, 2, .. 8 (8x8, 8x4, 2x8, etc.) 2 (2x2) O or X O or X Bars 1, 2, .. 4 (4x4, 4x2, 2x4) O or X O or X Four, two, ... 8 (8x8, 8x4, 4x8, 2x8, etc.) O or X O or X Oh, 1, 2, .. 16 (16x16, 16x8, 4x16, 2x16, etc.) O or X O or X 1, 2, ... 32 (32x32) O or X O or X Car 1, 2, .. 16 (16x16, 8x16, 4x16, etc.) 2 (2x2) O or X O or X 1, 2, .. 4 (4x4, 4x2, 2x4) O or X O or X Wave 1, 2, .. 8 (8x8, 8x4, 4x8, 2x8, etc.) O or X O or X 1, 2, .. 16 (16x16, 16x8, 4x16, 2x16, etc.) O or X O or X Dogs 1, 2, ... 32 (32x32) O or X O or X My 1, 2, ..

Table 2 shows an example of a combination of methods.

Among the modified methods of Table 2, when the method of "Issue 1 " is the case where the size of the luminance block is 8 (8x8, 8x4, 2x8, etc.) and the size of the color difference block is 4 (4x4, 4x2, 2x4) The method of the present invention can be applied to a luminance signal and a color difference signal.

Among the above modified methods, the method "wave 2" is a case where the size of a luminance block is 16 (16 × 16, 8 × 16, 4 × 16, etc.) and the size of a color difference block is 4 (4 × 4, 4 × 2, 2 × 4) The method of the specification may be applied to the luminance signal and not to the color difference signal.

In another modified method, the method of the specification is applied only to the luminance signal and may not be applied to the color difference signal. Conversely, the method of the specification is applied only to the color difference signal, and may not be applied to the luminance signal.

The method according to the present invention may be implemented as a program for execution on a computer and stored in a computer-readable recording medium. Examples of the computer-readable recording medium include a ROM, a RAM, a CD- , A floppy disk, an optical data storage device, and the like, and may also be implemented in the form of a carrier wave (for example, transmission over the Internet).

The computer readable recording medium may be distributed over a networked computer system so that computer readable code can be stored and executed in a distributed manner. And, functional programs, codes and code segments for implementing the above method can be easily inferred by programmers of the technical field to which the present invention belongs.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It should be understood that various modifications may be made by those skilled in the art without departing from the spirit and scope of the present invention.

Claims (10)

Receiving image data including depth information;
Generating object information based on depth information included in the image data; And
And determining a prediction block division structure for the image data based on the object information. The method according to claim 1,
Wherein the step of determining the prediction block division structure includes adaptively determining a variable division structure based on the object information. The method according to claim 1,
Wherein the step of determining the prediction block division structure includes the step of determining any one of a longitudinal direction and a longitudinal direction based on the object information. The method according to claim 1,
Wherein the step of determining the predictive block division structure includes the step of selecting 2Nx2N, NxN, and nPx2N when the prediction block is not formed of the same object but is divided in the vertical direction. The method according to claim 1,
Wherein the step of determining the predictive block division structure includes the step of selecting 2Nx2N, NxN, 2NxnP when the prediction block is not formed of the same object but is divided into the horizontal direction. A receiving unit for receiving image data including depth information;
An object information generating unit for generating object information based on depth information included in the image data; And
And a structure determination unit for determining a prediction block division structure for the image data based on the object information. The method according to claim 6,
Wherein the structure determination unit adaptively determines a variable division structure based on the object information. The method according to claim 6,
Wherein the structure determination unit determines one of a longitudinal direction and a longitudinal direction based on the object information. The method according to claim 6,
Wherein the structure determining unit selects 2Nx2N, NxN, and nPx2N when the prediction block is not made up of the same object but divided in the vertical direction. The method according to claim 6,
Wherein the structure determining unit selects 2Nx2N, NxN, and 2NxnP when the prediction block is not formed of the same object but is divided in the horizontal direction.

Images