KR102129271B1 - Method and apparatus for encoding of video using depth image - Google Patents

Abstract

Disclosed is a video encoding method and apparatus using depth information. An image encoding method using depth information according to an embodiment may include extracting depth value distribution information of a current largest coding unit (LCU); Predicting split structure candidates of the LCU based on the depth value distribution information; And determining an optimal partitioning structure among candidate partitioning structures of the LCU based on at least one of coding efficiency and image quality information.

Description

METHOD AND APPARATUS FOR ENCODING OF VIDEO USING DEPTH IMAGE}

The present invention relates to video encoding using depth information, and to a method and apparatus for deriving object information using depth information and efficiently encoding an image.

Depth information images are widely used in 3D video encoding, and are equipped with new input devices such as Kinect cameras on Xbox gaming machines, Intel SENZ3D webcams, iSense 3D scanners on iPads, and Google Tango smartphones. Information cameras can be used in many different 3D and 2D applications.

On the other hand, with the popularization and dissemination of depth information cameras, 2D/3D application applications are being popularized through more various 2D/3D application services, and accordingly, depth information cameras may be included in the multimedia camera system to utilize various information. Do.

US Publication No. 20140085416 (Invention name: Method and apparatus of texture image compress in 3d video coding) Korean Patent Publication No. 2012-0137305 (Invention name: block division method and apparatus using such method) Korean Patent Publication No. 2014-0048784 (Name of invention: Method and apparatus for deriving motion information by sharing limited depth information value)

An object of the present invention is to provide an image encoding method and apparatus capable of efficient encoding without performance degradation by using depth information in 2D video encoding.

An image encoding method using depth information according to an embodiment may include extracting depth value distribution information of a current largest coding unit (LCU); Predicting split structure candidates of the LCU based on the depth value distribution information; And determining an optimal partitioning structure among candidate partitioning structures of the LCU based on at least one of coding efficiency and image quality information.

An image encoding method using depth information according to another embodiment includes: extracting depth value distribution information of a current largest coding unit (LCU); Predicting the object structure of the CU based on the depth value distribution information of the CU included in the LCU; And omitting some of the rate distortion cost calculation based on the object structure prediction of the CU and determining an optimal partition structure among the partition structure candidates of the LCU.

An image encoding method using depth information according to another embodiment may include extracting depth value distribution information of a current coding unit (CU) from a depth image; And checking whether the current CU is made of a single object based on the depth value distribution information, and predicting a split structure of the current CU according to whether the current CU is made of a single object.

An image encoding apparatus using depth information according to an embodiment includes: a depth value extraction unit for extracting depth value distribution information of a current largest coding unit (LCU) from a depth image; A split structure prediction unit predicting split structure candidates of the LCU based on the depth value distribution information; And an optimal partitioning structure determining unit that determines an optimal partitioning structure among candidates of partitioning structures of the LCU based on at least one of coding efficiency and image quality information.

An image encoding apparatus using depth information according to another embodiment includes a depth value extraction unit for extracting depth value distribution information of a current largest coding unit (LCU) from a depth image; An object structure prediction unit for predicting the object structure of the CU based on the depth value distribution information of the CU included in the LCU; And an optimal partitioning structure determining unit which omits some of the rate distortion cost calculation based on the object structure prediction of the CU and determines the optimal partitioning structure among the partitioning structure candidates of the LCU.

An image encoding apparatus using depth information according to another embodiment includes: a depth value extraction unit for extracting depth value distribution information of a current largest coding unit (LCU) from a depth image; And a split structure prediction unit that checks whether the current CU is made of a single object based on the depth value distribution information, and predicts a split structure of the current CU according to whether the current CU is made of a single object.

According to an embodiment of the present invention, it is possible to perform efficient encoding for a 2D image by encoding a 2D general image using a depth information image obtained from a depth information camera.

1 is an exemplary view of a general image and a depth information map of the general image.
2 is an exemplary view of a Kinect input device.
3 shows a webcam product.
4 shows an iSense 3D scanner device.
5 shows a Google Tango smartphone.
6 is a diagram for explaining an HEVC encoding apparatus.
FIG. 7 is a diagram for explaining an example of video encoding in which a HEVC encoder is applied in a smart phone.
8 is a diagram for describing an example of HEVC including a depth image in a smart phone.
9 shows examples of dividing an image into a plurality of units.
10 shows an example of determining a CU partitioning structure in units of LCUs.
11 is a diagram for explaining an example of dividing an image into a plurality of prediction units.
12 is a diagram showing an example of a general video.
13 is a diagram illustrating an example of a depth information map for the general image of FIG. 12.
14 shows an image encoding method according to an embodiment of the present invention.
15 illustrates a video encoding method according to another embodiment of the present invention.
16 shows a video encoding method according to another embodiment of the present invention.
17 is a flowchart illustrating a method for determining an optimal CU partitioning structure for each LCU according to an embodiment of the present invention.
18 is a flow chart for explaining the CU termination early termination process according to an embodiment of the present invention.
19 is a diagram illustrating examples of depth value distribution of a CU.
20 is a flowchart for explaining another example of the CU split early termination process according to an embodiment of the present invention.
21 is a view for explaining various examples of object configuration of the CU.
22 and 23 are diagrams for describing image encoding methods according to an embodiment of the present invention.
24 is a view for explaining a process of simplifying the partition structure determination according to an embodiment of the present invention.
25 is a view for explaining another example of the process of simplifying the partition structure determination according to an embodiment of the present invention.
26 is a view for explaining the configuration of an image encoding apparatus according to an embodiment of the present invention.

The following merely illustrates the principles of the present invention. Therefore, those skilled in the art may implement various principles included in the concept and scope of the present invention and implement the principles of the present invention, although not explicitly described or illustrated in the present specification. In addition, all conditional terms and examples listed herein are in principle intended only for the purpose of understanding the concept of the present invention, and are not limited to the examples and states specifically listed as such. Should be.

For example, it should be understood that the block diagrams herein represent conceptual views of exemplary circuits embodying the principles of the invention. Similarly, all flow diagrams, state transition diagrams, pseudo-codes, etc. are understood to represent various processes performed by a computer or processor, whether or not the computer or processor is clearly depicted, which may be substantially represented on a computer-readable medium. Should be.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is an exemplary view of a general image and a depth information map of the general image. 2 is an exemplary view of a Kinect input device.

Referring to FIG. 1, (A) of FIG. 1 is an image actually photographed through a camera, and (B) represents a depth image of an actual image, that is, a depth information image (or depth information map). Depth information refers to information representing a distance between a camera and an actual object.

This depth information image is mainly used to create a 3D virtual view image. In fact, research related to this is a joint standardization group of Moving Picture Experts Group (MPEG) of ISO/IEC and Video Coding Experts Group (VCEG) of ITU-T. Three-dimensional video standardization is currently underway in The Joint Collaborative Team on 3D Video Coding Extension Development (JCT-3V).

The 3D video standard includes standards for advanced data formats and related technologies that can support not only stereoscopic images but also autostereoscopic images using normal images and their depth information images.

In November 2010, Microsoft released a Kinect sensor as a new input device for the XBOX-360 gaming device, which recognizes human motion and connects it to a computer system. It is not made up of 3D depth sensors. In addition, Kinect can generate RGB images and up to 640x480 depth maps as an imaging device and provide them to connected computers. In 2014, Intel announced the 720p CREATIVE SENZ3D webcam with 320x240 depth sensor for laptops, Apple released iSense as a 3D scanner for iPad with RGB camera and depth sensor, and Google launched Tango Smart with depth sensor Phone announced.

3 shows a webcam product.

Referring to FIG. 3, a CREATIVE SENZ3D webcam is shown, FIG. 3A shows a SENZ3D webcam product, and (B) shows a SENZ3D webcam prototype.

FIG. 4 shows the iSense 3D scanner device, and FIG. 5 shows the Google Tango smartphone.

4A shows an iSense product, and (B) shows an example of a scanning process through iSense. 5A shows a Google Tango smartphone product, and (B) shows a Google Tango smartphone prototype.

The emergence of Kinect, iSense 3D scanners, imaging equipment such as the Intel SENZ3D webcam, and Google Tango smartphones have made it a popular opportunity to enjoy a wide range of expensive 2D and 3D games and various application applications such as video services. It is showing that information cameras or video devices with sensors are becoming popular.

In this way, the future video system is not only a service for 2D general video, but also a Depth camera combined with a general video camera, so that 2D and 3D applied video services are basically provided, or input assistance in a handheld system is provided. It is expected to develop in the form of a device.

A video system in which a general camera and a depth camera are basically combined is not only a depth information in a 3D video codec, but also a new method of using depth information in a 2D video codec.

In addition, in a camera system including a depth information camera, encoding of a general image may be performed using an existing video codec as it is. Here, as an example of the existing video codec, MPEG-1, MPEG-2, MPEG-4, H.261, H.262, H.263, H.264/AVC, MVC, SVC, HEVC, SHVC, 3D-AVC , 3D-HEVC, VC-1, VC-2, VC-3, etc., and various other codecs.

6 is a diagram for explaining an HEVC encoding apparatus.

As an example of a method of encoding a real image and its depth information map, jointly completed standardization by the Moving Picture Experts Group (MPEG) and Video Coding Experts Group (VCEG), which have the highest coding efficiency among the video coding standards developed to date. Coding may be performed using a High Efficiency Video Coding (HEVC). An example of the HEVC coding structure is illustrated in FIG. 6. As shown in FIG. 6, HEVC includes various new algorithms such as coding units and structures, inter prediction, intra prediction, interpolation, filtering, and transformation methods. 6 is a block diagram showing an example of a configuration of an image encoding apparatus, and shows a configuration of an encoding apparatus according to the HEVC codec.

In this case, a sample adaptive offset (SAO) may be provided between the filter unit of FIG. 6 and the reference image buffer. The SAO can add an appropriate offset value to the pixel value to compensate for the coding error.

HEVC performs inter-prediction coding, that is, inter-prediction coding, so the currently coded image needs to be decoded and stored in order to be used as a reference image. Therefore, the quantized coefficients are inversely quantized in an inverse quantization unit and inversely transformed in an inverse transformation unit. The inverse quantized and inverse transformed coefficients are added to the prediction block through the adder 175 and a reconstructed block is generated. The reconstructed block that has passed through the filter unit is stored in a reference image buffer.

FIG. 7 is a diagram for explaining an example of video encoding in which a HEVC encoder is applied in a smart phone. 8 is a diagram for describing an example of HEVC including a depth image in a smart phone.

Referring to FIG. 7, a general form in a smartphone to which a HEVC encoder is applied may be a form in which an image captured through a smartphone is encoded using a HEVC encoder and a service is provided with the encoded image.

However, if an image is taken through a smartphone including a depth camera, a normal image (texture) and a depth image (depth) are independently produced as shown in FIG. 8, and optimization is performed using correlation with the normal image using depth information. Through the HEVC encoder through, it is possible to obtain an improved encoded image through complexity reduction.

US Patent Publication No. 20140085416, which is a prior art patent document 1, discloses a configuration of merging a block by checking information about an object of a current block from a depth map, but how to code using depth information It has not been disclosed at all whether to divide and code the unit.

In addition, Korean Patent Application Publication No. 2012-0137305 (Patent Document 2) and Korean Patent Publication No. 2014-0048784 (Patent Document 3) do not disclose content using depth information or construct a structure for predicting the partitioning structure of a CU. It is not clearly presented.

9 shows examples of dividing an image into a plurality of units.

In the high-efficiency video coding method, encoding is performed by dividing an image into LCU (Large Coding Unit) units, which is a basic unit of a coding unit (CU).

Here, the CU plays a role similar to the basic block of the existing video codec H.264/AVC, the macro block (MB: Macro Block,'MB'), but unlike the MB having a fixed size of 16x16, the CU is variable You can set the size. Also, the LCU divided for encoding may be divided into several CUs having a smaller size than the LCU for efficient encoding of an image.

Referring to FIG. 9, an LCU having a size of 64x64 may be divided into a plurality of CUs in various ways.

FIG. 9A shows an example of dividing a 64x64 LCU having a split depth value of 0 into 32x32 CUs having a split depth of 1.

FIG. 9B shows an example in which one of the 32x32 sized CUs is divided into a split depth of 2, and (C) shows an example in which two of the 32x32 sized CUs are split in a split depth of 2.

9(D) shows an example including CUs divided by a split depth of 3.

Accordingly, candidates for split structure of the LCU or CU may exist in various ways.

The LCU splitting structure is split information of coding units. After generating various LCU partition structures as described above and storing them in LCU partition structure candidates, in the step of determining the optimal LCU partition structure, one of the LCU partition structure candidates is selected as the optimal LCU partition structure in LCU units. . By using this method, it is possible to perform efficient encoding in terms of encoding efficiency and image quality by performing encoding based on an LCU splitting structure that is adaptive to the characteristics of an image in units of LCU.

10 shows an example of determining a CU partitioning structure in units of LCUs.

In the HEVC encoder, a method of dividing a coding unit in an LCU may be differently determined according to characteristics of an image. That is, in order to determine the optimal LCU splitting structure, a plurality of splitting structures can be coded. As a method for determining the optimal LCU splitting structure, prediction modes of all coding units for each splitting depth (Intra, Inter mode, etc.) After encoding is performed, a method of determining a prediction mode of a corresponding coding unit may be used according to the amount of encoded bits and image quality.

For example, as illustrated in FIG. 9, encoding is performed in Intra mode and Inter mode with 64x64 coding units having a depth of 0, and then an optimal mode is stored, and 64x64 coding units are divided into four to encode each 32x32 size. As a unit, encoding can be recursively performed in Intra mode and Inter mode, respectively. In this case, the prediction units may be independently selected for each of the 4 coding units having a size of 32x32. Also, encoding is performed by dividing into 4 coding units having a size of 16x16 even within a coding unit having a size of 32x32. In this way, after coding is performed by dividing the corresponding coding unit in a recursive manner, a split structure of the most efficient coding unit in terms of bit amount and image quality is determined.

For example, when the bit amount and picture quality of the four encoded 32x32 sized coding units are more efficient than the case of 64x64 sized coding units, it is determined that the coding unit is divided into four 32x32 sized coding units. When encoding an image in an encoder (image encoder), the optimal coding unit distribution is found for the number of cases in which the LCU can be split, which acts as a factor to increase the computational complexity of the encoder.

For efficient compression performance, as the coding efficiency is determined for a large number of cases, a computational complexity increases.

11 is a diagram for explaining an example of dividing an image into a plurality of prediction units.

Referring to FIG. 11, the basic structure of a CU may be represented by 2Nx2N having a square size.

Prediction unit (PU: Prediction Unit, hereinafter referred to as'PU') is a prediction structure divided within a CU.

In FIG. 11, reference numeral 1110 denotes an example of Symmetric Motion Partitioning (SMP) as a symmetrical partitioning structure, and reference numeral 1120 denotes an example of 2NxN and Nx2N structures as symmetrical partitioning structures of up and down or left and right.

In addition, reference numeral 1130 denotes an example of 2NxnU, 2NxnD, nLx2N, and nRx2N as examples of Asymmetric Motion Partitioning (AMP) in an asymmetric partitioning structure. At this time, n is an integer, U, D, L, R may be an integer or rational number, respectively.

PU is a structure that is determined for more efficient encoding when encoding is performed in intra or inter mode on a CU basis. This is to predict and encode each PU independently by dividing it into PU units in the CU when accurate prediction cannot be performed when performing coding in a CU unit. In the process of determining the PU partition structure, the current block to be predicted may be predicted by all the PU partition structures shown in FIG. 11, and the predicted value may be determined as a PU partition structure having a block closest to the current block.

The split structure determination and the PU split structure determination of FIGS. 9 to 11 do not consider object information with respect to neighboring blocks. This is natural, because in the case of a 2D image, if there is no depth camera, the object information in the image must be extracted through 2D image analysis. Therefore, the method of using object information is not installed in the existing 2D video encoding method.

For the same reason, in the case of HEVC, no coding method using object information is installed in the split structure determination method. However, if object information can be considered using a depth camera, in a method for determining a structure of a partition, if a correlation can be determined according to object configuration information of a corresponding CU, and a structure can be determined by efficient prediction, encoding complexity is effectively reduced.

Accordingly, when depth information is used in a method of determining a split structure, coding complexity can be effectively reduced when an effective split structure is predicted for a corresponding region by determining object information or a configuration of an object region of the corresponding CU.

12 is a diagram showing an example of a general video. 13 is a diagram illustrating an example of a depth information map for the general image of FIG. 12.

As shown in FIGS. 12 and 13, the part corresponding to the object boundary area has a high probability of having a complex CU partitioning structure, and it is highly likely to have a relatively simple partitioning structure where it is determined to be an object interior or background area.

Therefore, when obtaining object information for a region to be currently coded using depth information, a high probability is determined that the current coding region is determined by a complex partitioning structure composed of a plurality of CUs or a simple partitioning structure composed of a few CUs. Can make predictions. Through this, it is possible to reduce the amount of computation by limiting the determination of the partition structure with low probability. The method proposed in the present invention can predict and encode a split structure having a high probability using depth information in determining a split structure.

In the 2D video codec according to the prior art, algorithms are designed without reflecting the use of depth information at all. However, since the actual image and its depth information image have a great correlation, the method of using depth information in 2D video encoding by developing an algorithm that considers the depth information, considering that depth information can be used to encode a 2D image You can consider

The basic principle of the present invention is to use depth information in a motion prediction method for efficient encoding in a 2D video codec and to use it to encode a real image using depth information obtained from a depth information camera.

For example, when the objects of the general image are classified and coded by using the depth information image, the coding complexity for the general image can be greatly reduced.

Here, the objects mean multiple objects, and may include a background image. In a block-based encoding codec, multiple objects may exist in a block, and different encoding methods may be applied to each object based on the depth information image. Can.

14 shows an image encoding method according to an embodiment of the present invention.

Referring to FIG. 14, in step 1410, the image encoding apparatus extracts depth value distribution information of a current largest coding unit (LCU) from a depth image.

At this time, the depth value distribution information may be a depth information map of the CU, for example, as shown in (A) or (B) of FIG. 19, and may be represented as a normalized value. For example, in FIG. 19B, the normalized depth value of M1 is 9, the normalized depth value of M2 is 7, and the normalized depth value of M3 may be 1.

In step 1420, the image encoding apparatus predicts the split structure candidates of the LCU based on the depth value distribution information.

At this time, step 1420 of predicting the split structure candidates of the LCU is a process of checking the depth value distribution information of the CU currently included in the LCU and checking whether the CU is made of a single object based on the depth value distribution information of the CU, and the When the CU is composed of a single object, a process of ending prediction of the partition structure candidate prediction of the CU may be included.

In step 1430, the video encoding apparatus determines an optimal split structure among the split structure candidates of the LCU based on at least one of coding efficiency and image quality information.

At this time, in step 1430 of determining the optimal split structure, the video encoding apparatus predicts the object structure of the CU based on the depth value distribution information, and omits some of the rate distortion cost calculation based on the object structure prediction of the CU The optimal partitioning structure can be determined.

15 illustrates a video encoding method according to another embodiment of the present invention.

Referring to FIG. 15, in step 1510, the image encoding apparatus extracts depth value distribution information of a current largest coding unit (LCU).

In operation 1520, the image encoding apparatus predicts the object structure of the CU based on the depth value distribution information of the CU included in the LCU.

At this time, the prediction of the object structure may estimate whether the object structure in the CU is a single object, a configuration divided up or down, or a structure divided left and right from the depth value distribution information of the CU. For example, when the depth value distribution of the CU of FIG. 21 is equal to (C), when the same depth value is distributed over a predetermined number on the left and right sides based on the center of the CU, the object structure is divided into left and right structures. Can be predicted as At this time, the preset number may be set according to the accuracy allowed by the system.

At this time, the center of the CU may be the center of the horizontal axis when determining the upper and lower object structures, and the center of the vertical axis when determining the left and right object structures.

As another example, an object structure may be determined based on a case in which the maximum and minimum values of the depth values of the left and right sides are smaller than a specific value based on the center of the horizontal axis of the CU.

In step 1530, the video encoding apparatus omits some of the rate distortion cost calculation based on the object structure prediction of the CU and determines an optimal segmentation structure among the segmentation candidates of the LCU.

Specific examples of the operation omitted from the rate distortion cost operation will be described through examples shown in FIG. 21.

16 shows a video encoding method according to another embodiment of the present invention.

Referring to FIG. 16, in step 1610, the image encoding apparatus extracts depth value distribution information of a current coding unit (CU) from a depth image.

In this case, step 1610 of predicting a split structure of the current CU includes determining that the current CU is not split when the current CU is composed of a single object.

Therefore, according to the embodiment illustrated in FIG. 16, when it is determined through a depth information that a CU has a single object, coding complexity may be reduced by determining a partition structure candidate as not to further divide the CU. have.

In step 1620, the image encoding apparatus determines whether the current CU is composed of a single object based on depth value distribution information, and predicts a split structure of the current CU according to whether the current CU is composed of a single object.

At this time, in step 1620 of predicting the split structure of the current CU, if the size of the current CU is greater than or equal to a preset value, and the current CU is predicted to be composed of a single object, and the encoding mode of the CU is a skip mode, the current CU is split. And determining not to.

At this time, in step 1620 of predicting the split structure of the current CU, the size of the current CU is smaller than a preset value, the current CU is predicted to be composed of a single object, and the size of the reference CU of the CU is greater than or equal to the preset value. When the reference CU is encoded in the skip mode, it may include determining that the CU is not split.

At this time, whether the current CU is composed of a single object may be determined by the depth value of the four corners of the CU, and various examples of determining whether a single object is based on the depth value distribution will be described in detail with reference to FIGS. 18 to 20. Shall be

17 is a flowchart illustrating a method for determining an optimal CU partitioning structure for each LCU according to an embodiment of the present invention.

17 is a flowchart illustrating in more detail the process of determining the LCU unit division structure shown in FIG. 14.

For example, step 1410 of FIG. 14 may include steps 1701 and 1703 of FIG. 17. Also, step 1420 of FIG. 14 may include steps 1705 to 1710, 1750 and 1760 of FIG. 17.

In step 1705, the image encoding apparatus starts a CU unit recursion process, and in step 1707, LCU information such as depth information or a coding mode of a reference CU of a current CU is extracted, a CU of a current split depth is encoded, and an optimal CU candidate is obtained. Can be saved. For example, the CU of the current division depth may be (A) of FIG. 9.

In step 1710, the image encoding apparatus may perform an early termination process of CU segmentation using depth information. At this time, the CU split early termination process may be a split structure candidate prediction process illustrated in FIGS. 14, 18, and 20.

It may be determined to divide the CU by performing the partitioning early termination process in step 1710, and in step 1750, the CU may be divided into four CUs having an increase in the division depth by 1 as shown in FIG. 9(B).

If, after the CU unit recursive process, the CU split structure candidate is determined as (A), (B), and (C) of FIG. 9, the video encoding apparatus considers coding efficiency for each split structure candidate in step 1730. Can determine the optimal CU partitioning structure.

When the optimal CU partitioning structure is determined, in step 1740, the CU unit recursion process may end.

18 is a flow chart for explaining the CU termination early termination process according to an embodiment of the present invention.

Referring to FIG. 18, step 1801 of extracting depth information of the current LCU may be the same step as step 1410 of FIG. 14 or step 1701 of FIG. 17.

In step 1810, a current coding unit (CU) is selected, and it is determined whether the size of the current CU is 32x32 or more, for example, whether the size of the current CU is 64x64 or 32x32.

If the size of the current CU is 32x32 or more, step 1811 is performed; otherwise, step 1820 is performed.

In step 1811, it is determined whether the current CU consists of a single object and the current CU encoding mode is a skip mode. If the current CU is composed of a single object, and the encoding mode of the CU is a skip mode, in step 1813, information indicating that the current CU is not split may be stored and execution of the early termination of the CU partitioning process may be terminated.

In step 1811, if it is determined that the current CU is not composed of a single object, or if the current CU encoding mode is not a skip mode, information indicating that the current CU is split may be stored and the execution of the early termination of the CU partitioning process may be terminated.

In operation 1820, the video encoding apparatus determines whether the current LCU and the current CU are composed of a single object. If it is determined that the current LCU or the current CU does not consist of a single object, in step 1830, information for dividing the current CU may be stored, and execution of the early termination process of partitioning the CU may be terminated.

If it is determined in step 1820 that the current LCU or the current CU is not composed of a single object, in step 1840, depth information of the reference CU is extracted and step 1841 is performed.

In step 1841, the video encoding apparatus determines whether the size of the reference CU is 32x32 or more (for example, when the size of the reference CU is 64x64 or 32x32) and whether the encoding mode of the reference CU is a skip mode.

If the size of the reference CU is greater than or equal to 32x32 in step 1841 and the encoding mode of the reference CU is skip mode, information indicating that the current CU is not split is stored in step 1843 and the execution of the early termination of the CU partitioning process can be ended.

For example, when the size of the current CU is a CU having a size smaller than 32x32, it is determined that the current CU is composed of a single object through depth information, and the reference LCU is also determined to be composed of a single object through depth information. If the size of the CU (located at L0) is 64x64 or 32x32 and is encoded in Skip mode, the current CU may not be further partitioned by predicting that the CU is also composed of a large size, that is, a probability of a simple partitioning structure is large. .

If the size of the reference CU is not greater than 32x32 or the encoding mode of the reference CU is not a skip mode, the operation 1830 is performed.

In this case, the skip mode may be an encoding mode in which a differential signal (or a differential image) from a prediction image or a reference image for an original image is not encoded or transmitted to a decoding stage.

At this time, whether the CU is made of a single object or not, if the difference between the maximum and minimum values included in the depth value distribution information of the CU is equal to or less than a preset value, it can be determined that the CU is made of a single object.

Accordingly, referring to steps 1810 to 1843 illustrated in FIG. 18, the step of predicting the split structure candidates of the LCU according to an embodiment is included in information about the size of the CU included in the current LCU and the depth value distribution information of the CU And determining whether to divide the CU by considering at least one of a difference between a maximum value and a minimum value, and whether the CU encoding mode is a skip mode.

19 is a diagram illustrating examples of distribution of depth values of a CU.

In an image encoding method according to an embodiment, as an example of a method of determining whether a CU or block is composed of the same single object, a depth value of four corner positions of a CU or block may be used.

Referring to FIG. 19, the apparatus for encoding an image may determine that the distribution of the CU depth value as shown in (A) is selected without changing the depth value.

On the other hand, the image encoding apparatus may determine that the CU depth value distribution as shown in FIG. 19B is not composed of a single object because the depth value change is large.

Referring to (B) of FIG. 19, it can be seen that the change of the depth value of the middle portion and the edge portion in the CU is very large. In this case, it can be seen that the difference between the maximum value and the minimum value among the depth values of the four edges is large. . Therefore, since such a CU has a low probability of being composed of a single object, CU partitioning can be performed.

The video encoding apparatus may determine that the CU is composed of a single object when the difference between the maximum and minimum values among the four corner depth values of the CU is equal to or less than a preset reference value.

In this case, for example, in FIG. 19A, the normalized depth value of M1 is 9, the normalized depth value of M2 and M3 is 7, and the normalized depth value of M4 may be 7.

In addition, in FIG. 19B, the normalized depth value of M1 is 9, the normalized depth value of M2 and M4 is 7, and the normalized depth value of M3 may be 1.

At this time, in FIG. 19A, the difference between the maximum and minimum values of the depth values of the four corners of the CU is 2, and in the case of FIG. 19B, the difference between the maximum and minimum values of the depth values of the four corners of the CU is 8.

Accordingly, when the preset reference value is 5, it can be determined that (A) of FIG. 19 consists of a single object, and (B) of FIG. 19 does not consist of a single object.

20 is a flowchart for explaining another example of the CU split early termination process according to an embodiment of the present invention.

Referring to FIG. 20, in step 2010, the video encoding apparatus determines whether the current CU is 32x32 or more. If the conditions of step 2010 are satisfied, step 2020 is performed, otherwise step 2050 is performed.

In step 2020, the image encoding apparatus determines whether the size of the CU is greater than or equal to a preset value, the difference between the maximum and minimum values of the four corner depth values of the CU is equal to or less than a preset reference value, and whether the CU is encoded in a skip mode.

At this time, the preset reference value may be 5.

If the condition of 2020 is satisfied, information indicating that the current CU is not partitioned may be stored, and the CU partition early termination process may be terminated.

Accordingly, in the step of predicting the split structure candidates of the LCU according to an embodiment, the size of the CU is greater than or equal to a long set value, and the difference between the maximum and minimum values among the four corner depth values of the CU is less than a preset reference value and the CU is skipped When it is coded in a mode, it may include determining that the CU is not split.

If the conditions of the 2020 step are not satisfied, information indicating that the CU is split may be stored and the early termination of the CU partitioning process may be terminated.

In step 2050, the image encoding apparatus determines whether the size of the CU is smaller than a long set value and that the four corner depth values of the LCU and the CU are all the same value.

If the condition of step 2050 is satisfied, step 2060 is performed, otherwise step 2080 is performed.

In steps 2060 and 2070, the image encoding apparatus extracts depth information of the reference CU, and determines whether the size of the reference CU is 32x32 or more and whether the encoding mode of the reference CU is a skip mode.

If the condition of step 2070 is satisfied, step 2090 is performed, otherwise step 2080 is performed.

Therefore, in the step of predicting the split structure candidates of the LCU according to an embodiment, the size of the CU is smaller than a long set value, and the four corner depth values of each of the LCU and the CU are the same value, and the reference CU of the CU When the size is greater than or equal to the preset value and the reference CU is encoded in the skip mode, it may include determining that the CU is not split.

21 is a view for explaining various examples of object configuration of the CU.

According to an embodiment of the present invention, the object structure of the CU is determined through depth information, and the rate distortion (RD-Cost, hereinafter referred to as RD-Cost) operation required for the prediction of the divided structure mode in determining the divided structure can be simplified. .

Using the depth value distribution information of the CU, since the object configuration information of the CU can be predicted, the determination of the mode prediction is simplified by not performing an Asymmetric Motion Partitioning (AMP) RD-Cost operation for a case that satisfies any condition. I can do it.

For example, in the case of a CU consisting of a single object configuration as shown in (A) of FIG. 21, the probability that the split structure mode of the CU is selected as one of the AMPs is low. Therefore, in this case, RD-Cost calculation may not be performed for 2NxnU, 2NxnD, nLx2N, and nRx2N.

Accordingly, in the optimal partitioning structure prediction step or the process of simplifying the partitioning structure determination according to an embodiment of the present invention, when the object structure of the CU is predicted to consist of a single object, asymmetric motion partition (AMP) for the CU Rate distortion calculation can be omitted.

In addition, when the composition of the CU is divided into upper and lower parts as shown in FIG. 21(B), since there is a high probability that the upper and lower parts of the CU are composed of different objects, RD-Cost operations for nLx2N and nRx2N are performed. It may not.

Therefore, in the optimal partitioning structure prediction step or the process of simplifying the partitioning structure determination according to an embodiment of the present invention, when the object structure of the CU is predicted to be a divided structure up and down, asymmetric motion partitioning (AMP) of the CU is performed. Among the rate distortion operations for Korea, operations related to left and right division may be omitted.

Also, as shown in (C) of FIG. 21, when the CU is divided into left and right, and is composed of objects, there is a high probability that the left and right sides of the CU are composed of different objects, so RD-Cost operations for 2NxnU and 2NxnD are performed It may not.

Therefore, in the optimal partitioning structure prediction step or the process of simplifying the partitioning structure determination according to an embodiment of the present invention, when the object structure of the CU is predicted to be divided into left and right structures, asymmetric motion partition (AMP) of the CU is used. Among the rate distortion operations for Korea, operations related to the top and bottom division may be omitted.

22 and 23 are diagrams for describing image encoding methods according to an embodiment of the present invention.

Referring to FIG. 22, it can be seen that steps 2201, 2203, and 2205 are the same as steps 1701, 1703, and 1705 of FIG. 17, respectively.

In addition, it can be seen that the remaining steps 2220 to 2270 are the same as steps 1709 to 1760 of FIG. 17 except for step 2210, which is a step of performing a simplified structure determination process using depth information.

In this case, step 2210, which is a step of simplifying the partition structure determination using depth information, may include the process illustrated in FIG. 24 or the process illustrated in FIG. 25.

Referring to FIG. 23, after performing a process of simplifying the partition structure determination using depth information in the process of determining the entire partition structure of the LCU unit, it is possible to perform an early termination process of CU partition using depth information in step 2320 through step 2320. Able to know.

At this time, in step 2310, the image encoding apparatus may perform steps 2320 to 2260 after storing only the operations that are omitted according to the object structure of the CU.

For example, if the rate distortion operation that is omitted according to the object structure of the CU is "the operation related to the left and right division among rate distortion operations for the asymmetric motion partition (AMP)", the image encoding apparatus in step 2350. The optimal CU partitioning structure may be determined among CU partitioning structure candidates in consideration of the omitted operation.

24 is a view for explaining a process of simplifying the partition structure determination according to an embodiment of the present invention.

Referring to FIG. 24, in operation 2510, the image encoding apparatus may determine that the current CU is composed of a single object. If the condition of step 2410 is satisfied, step 2420 may be performed, otherwise, step 2430 may be performed.

If the condition of step 2410 is satisfied, it is determined in step 2420 that the AMP RD cost operation is skipped, and the determination of all partition structures can be completed.

In step 2430, the image encoding apparatus may determine whether the current CU object configuration is divided up and down, and if the conditions are satisfied, it may be determined in step 2440 to skip the calculation of the rate distortion cost of nLx2N and nRx2N.

In step 2450, the image encoding apparatus may determine whether the CU object configuration is divided into left and right, and if satisfied, the 2NxnU and 2NxnD rate distortion cost calculation may be skipped in step 2460.

25 is a view for explaining another example of the process of simplifying the partition structure determination according to an embodiment of the present invention.

25 shows an example of predicting an object configuration of a CU through a distribution of depth values of the CU.

For example, in step 2510, if the four corner depth values of the CU are all the same, the image encoding apparatus determines that the CU is composed of a single object, and performs step 2520, which is the same as step 2420.

In addition, in step 2530, the image encoding apparatus does not have the same depth values of the four corners of the CU, but when the depth values of the upper two corners are the same and the depth values of the lower two corners are the same, the object configuration of the CU is divided up and down. It is determined that step 2540, which is the same as step 2440, is performed.

Also, in step 2550, the image encoding apparatus does not satisfy the conditions of steps 2510 and 2530, but when the depth values of the left corners of the CU are the same and the depth values of the two right corners are the same, it is determined that the object configuration of the CU is divided into left and right. Step 2560 may be performed.

26 is a view for explaining the configuration of an image encoding apparatus according to an embodiment of the present invention.

The image encoding apparatus illustrated in FIG. 26 may perform an image encoding method according to an embodiment of the present invention.

Referring to FIG. 26, the image encoding apparatus 2600 may include a depth value extraction unit 2610, a split structure prediction unit 2620, and an optimal split structure determination unit 2630. Also, the image encoding apparatus 2600 may further include an object structure prediction unit 2640.

The depth value extractor 2610 extracts depth value distribution information of a current largest coding unit (LCU) from a depth image.

The split structure prediction unit 2620 predicts split structure candidates of the LCU based on the depth value distribution information. In this case, prediction of candidates for the split structure of the LCU may be performed in step 1710 of an early termination process of a CU partition using depth information of FIG. 17.

In addition, the split structure prediction unit 2620 determines whether the current CU is made of a single object based on depth value distribution information, and predicts the split structure of the current CU according to whether the current CU is made of a single object. .

The optimal split structure determining unit 2630 determines an optimal split structure among split structure candidates of the LCU based on at least one of coding efficiency and image quality information.

The object structure prediction unit 2640 predicts the object structure of the CU based on the depth value distribution information of the CU included in the LCU. At this time, the optimal partition structure determining unit 2630 omits some of the rate distortion cost calculation based on the object structure prediction of the CU and determines the optimal partition structure among the partition structure candidates of the LCU.

[Table 1] shows the experimental results of applying the embodiment shown in Figure 25 to HEVC.

Through the experimental results, it can be seen that the coding complexity is reduced without significant deterioration of the image quality with the same quality in the subjective image quality.

[Table 1]

In embodiments of the present invention, a target range or an application range may be varied according to a block size or a division depth of a CU.

At this time, the variable (that is, size or depth information) that determines the coverage may be set such that the encoder and decoder use a predetermined value, or may use a predetermined value according to a profile or level, and the encoder may use a variable value If is written in the bitstream, the decoder may obtain and use this value from the bitstream. When the coverage is different according to the CU division depth, it is as illustrated in [Table 2]. Method A may be a method that applies only to a depth greater than or equal to a preset depth value, Method B may be a method that applies only to a predetermined depth value or less, and Method C may have a method that applies only to a preset depth value.

[Table 2]

[Table 2] shows an example of a method for determining an application range in which the methods of the present invention are applied when a given CU division depth is 2. (O: Applies to the depth, X: Does not apply to the depth.)

When the methods of the present invention are not applied to all depths, an arbitrary flag may be used to indicate the bitstream, and a value greater than the maximum value of the CU depth is signaled as a CU depth value indicating the coverage. It can also be expressed by

In addition, each of the above-described methods may be differently applied to the color difference block according to the size of the luminance block, and may be differently applied to the luminance signal image and the color difference image.

[Table 3] shows an example that is applied differently according to the size of the luminance block and the color difference block when each method is combined.

[Table 3]

Looking at Method 4 among the modified methods in [Table 3], the luminance block size is 8 (8x8, 8x4, 2x8, etc.), and the color difference block size is 4 (4x4, 4x2, 2x4). The method of constructing a merge list according to an embodiment of the present invention can be applied to a luminance signal and a color difference signal.

Looking at method wave 2 among the above modified methods, the present invention is when the size of the luminance block is 16 (16x16, 8x16, 4x16, etc.), and the size of the color difference block is 4 (4x4, 4x2, 2x4). The method of constructing a merge list according to an embodiment may be applied to a luminance signal and not to a color difference signal.

In another modified method, a merge list construction method according to an embodiment of the present invention is applied only to a luminance signal, and may not be applied to a color difference signal. Conversely, the merge list construction method according to an embodiment of the present invention is applied only to the color difference signal, and may not be applied to the luminance signal.

The device described above may be implemented with hardware components, software components, and/or combinations of hardware components and software components. For example, the devices and components described in the embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (micro signal processors), microcomputers, field programmable arrays (FPAs), It may be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. Further, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and/or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the signal wave being transmitted. The software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. Includes hardware devices specifically configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, proper results can be achieved even if replaced or substituted by equivalents.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

Extracting depth value distribution information of a current largest coding unit (LCU);
Determining whether a CU included in the current LCU consists of a single object based on the depth value distribution information and predicting split structure candidates of the LCU according to whether the CU consists of a single object; And
Predicting the object structure of the CU based on the depth value distribution information, and determining an optimal segmentation structure among candidates of the segmentation structure of the LCU based on the object structure prediction of the CU
Image encoding method using depth information comprising a. According to claim 1,
When the CU consists of a single object, ending prediction of the partition structure candidate of the CU.
Image encoding method using depth information comprising a. According to claim 2,
Whether the CU consists of a single object,
If the difference between the maximum and minimum values included in the depth value distribution information of the CU is equal to or less than a preset value, it is determined that the CU is composed of a single object.
Image coding method using depth information. According to claim 2,
Whether the CU consists of a single object,
If the difference between the maximum and minimum values of the four corner depth values of the CU is equal to or less than a preset reference value, it is determined that the CU is composed of a single object.
Image coding method using depth information. Extracting depth value distribution information of a current largest coding unit (LCU);
Predicting split structure candidates of the LCU based on the depth value distribution information; And
And predicting an object structure of a CU included in the LCU based on the depth value distribution information, and determining an optimal segmentation structure among candidates of the segmentation structures of the LCU based on the object structure prediction of the CU,
Predicting the split structure candidates of the LCU,
Determining whether to divide the CU by considering at least one of a size of a CU included in the LCU, a difference between a maximum value and a minimum value included in the depth value distribution information of the CU, and whether the encoding mode of the CU is a skip mode Course containing
Image coding method using depth information. Extracting depth value distribution information of a current largest coding unit (LCU);
Predicting split structure candidates of the LCU based on the depth value distribution information; And
And predicting an object structure of a CU included in the LCU based on the depth value distribution information, and determining an optimal segmentation structure among candidates of the segmentation structures of the LCU based on the object structure prediction of the CU,
Predicting the split structure candidates of the LCU,
And determining that the CU is not split when the size of the CU is greater than a preset value and the difference between the maximum and minimum values among the four corner depth values of the CU is less than or equal to a preset reference value and the CU is encoded in a skip mode. doing
Image coding method using depth information. Extracting depth value distribution information of a current largest coding unit (LCU);
Predicting split structure candidates of the LCU based on the depth value distribution information; And
And predicting an object structure of a CU included in the LCU based on the depth value distribution information, and determining an optimal segmentation structure among candidates of the segmentation structures of the LCU based on the object structure prediction of the CU,
Predicting the split structure candidates of the LCU,
The size of the CU is smaller than a preset value, all four corner depth values of the LCU and the CU are the same value, the size of the reference CU of the CU is greater than the preset value, and the reference CU is encoded in a skip mode. And determining that the CU is not divided.
Image coding method using depth information. According to claim 1,
Determining the optimal division structure,
Based on the object structure prediction of the CU, it is characterized in that a part of the rate distortion cost calculation is omitted and an optimal division structure is determined
Image coding method using depth information. delete The method of claim 8,
Determining the optimal division structure,
When it is predicted that the object structure of the CU is composed of a single object, rate distortion calculation for asymmetric motion partition (AMP) of the CU is omitted.
Image coding method using depth information. The method of claim 8,
Determining the optimal division structure,
When the object structure of the CU is predicted to be a structure divided up and down, the operation related to the left and right division among the rate distortion operations for the asymmetric motion partition (AMP) of the CU is omitted.
Image coding method using depth information. The method of claim 8,
Determining the optimal division structure,
When the object structure of the CU is predicted to be divided into left and right structures, among the rate distortion operations for the asymmetric motion partition (AMP) of the CU, the operation related to the upper and lower partitions is omitted.
Image coding method using depth information. delete delete delete delete delete A depth value extraction unit for extracting depth value distribution information of a current largest coding unit (LCU) from the depth image;
A split structure prediction unit that checks whether a CU included in the current LCU consists of a single object based on the depth value distribution information, and predicts split structure candidates of the LCU according to whether the CU consists of a single object; And
An optimal partitioning structure determination unit that predicts an object structure of the CU based on the depth value distribution information and determines an optimal partitioning structure among candidates of partitioning structures of the LCU based on the object structure prediction of the CU
Video encoding apparatus using depth information including. The method of claim 18,
The optimal division structure determining unit,
Based on the object structure prediction of the CU, omitting some of the rate distortion cost calculation and determining an optimal partitioning structure among the partitioning structure candidates of the LCU
Video encoding apparatus using depth information including. The method of claim 19,
The optimal division structure determining unit,
When it is predicted that the object structure of the CU is composed of a single object, rate distortion calculation for asymmetric motion partition (AMP) of the CU is omitted.
Image encoding device using depth information.

Images