KR102091860B1 - Method and apparatus for image encoding - Google Patents

Abstract

The present invention relates to an image encoding method, and more specifically, to obtain a depth map and a color image using depth image-based rendering; Classifying the depth map into a background depth texture and a foreground depth texture; And predicting a depth of the hole region after excluding the foreground depth texture. An encoding method capable of improving the quality of an image in a hole filling process is provided.

Description

Method and apparatus for image encoding}

The present invention relates to an image encoding method and apparatus.

3D video processing technology is a core technology in the field of next-generation information and communication services, and it is a cutting-edge technology in which demand and technology development competition is fierce along with development into the information industry society. This three-dimensional video processing technology is an essential element for providing high-quality video services in multimedia applications. Today, it is applied not only to the information and communication fields, but also to applications such as broadcasting, medical, education (or training), military, games, animation, and virtual reality. The field is becoming very diversified. In addition, three-dimensional video processing technology has also established itself as a core base technology for next-generation realistic three-dimensional multimedia information and communication that is commonly required in various fields, and research on this has been actively conducted in developed countries.

In general, three-dimensional video can be defined from two viewpoints as follows. First, the 3D video may be defined as a video that is configured to allow a user to feel a part of the image protruding from the screen by applying depth information to the image. Second, the three-dimensional video can be defined as a video that is configured to provide a variety of viewpoints to the user, from which the user can feel the reality (ie, three-dimensional feeling) in the image. These 3D videos can be classified into binocular, multi-eye, IP (Integral Photography), multi-view (Omni, Panorama), hologram, etc. according to the acquisition method, depth impression, display method, etc. have. In addition, there are largely two methods for expressing the three-dimensional video: image-based representation (construction) and mesh-based representation (mesh-based representation).

Recently, depth image-based rendering (DIBR) has been spotlighted as a method of expressing such a 3D video. Depth image-based rendering refers to a method of creating scenes at different viewpoints by using reference images having information such as depth or difference angle for each related pixel. The depth image-based rendering not only easily renders difficult and complex shapes of a 3D model, but also enables application of a signal processing method such as general image filtering, and is capable of producing high quality 3D video. Have In order to realize the above, the depth image based rendering is a depth image (or depth map) and a texture image acquired through a depth camera and a multi-view camera. (or Color Image)). In particular, depth images are used to express three-dimensional models more realistically. That is, it is used for the creation of a more three-dimensional 3D video.

As described above, DIBR is a method of generating an image of a different view using a color image of a given view and a depth map accordingly.

However, an empty area, that is, a hole, is generated in the image generated after DIBR.

1 shows a hole area generated during DIBR. Since the hole has a great influence on the subjective image quality of a stereoscopic image, a hole-filling technique is required. However, since the color information for the hole area does not exist in the original image before DIBR, filling the hole with high quality is not easy.

Conventionally, as a prior study considering the depth of the hole, there is a method of predicting the depth of the hole by applying an existing image painting technique.

FIG. 2 illustrates a prediction of a color image, a depth map, and a hole region depth generated after conventional DIBR.

Referring to FIG. 2, in DIBR, a color image and a corresponding depth map are also rendered, and a depth map as well as a color image at a generated time point (see FIG. 2 (a)) is obtained (see FIG. 2 (b)).

Then, the depth map is predicted (see FIG. 2 (c)) to fill the holes of the generated depth map. For depth prediction, an inpainting algorithm using Navier-Stokes and fluid dynamics equations can be applied.

However, in the case of using the conventional depth prediction method, the DIRB hole region depth illustrated in FIG. 3 is also predicted because the existing inpainting algorithm is directly applied to the hole prediction of the depth map even though the hole region is a background region. As a result, a problem in which the foreground and the background are mixed is predicted.

In addition, the inpainting algorithm using the Navier-Stokes and fluid dynamics equations applies inpainting on a pixel-by-pixel basis, so there is a problem that the surrounding texture information cannot be utilized.

Prediction of the depth of the inaccurate hole region does not improve the quality of the generated image by playing a secondary role in hole filling of the color image, but rather, it lowers the efficiency of the algorithm.

The present invention has been devised to solve the above problems, and in particular, to provide an image encoding method and apparatus for improving image quality by performing hole filling to prevent mixing of the foreground and background when predicting the depth of the DIRB hole region. There is a purpose.

In addition, an object of the present invention is to provide an image encoding method and apparatus for improving the efficiency of an algorithm by predicting a depth of a hole region by using surrounding texture information.

Consistent with the present invention devised to achieve the above object, an image encoding method comprising: acquiring a depth map and a color image using depth image-based rendering; Classifying the depth map into a background depth texture and a foreground depth texture; And predicting the depth of the hole region after excluding the foreground depth texture.

Here, it is preferable to use an inpainting algorithm for the depth prediction, and in particular, it is preferable to use Navier-Stokes and fluid dynamics equations.

The classifying step includes a difference value I between a first pixel located at a boundary in the first area and a second pixel located at a boundary in the second area abutting the first area, and within the third area abutting one side of the hole area. Calculate the difference value II between the third pixel located at the boundary and the fourth pixel located at the boundary in the fourth area contacting the other side of the hole area, and determine based on the difference III between the difference value I and the difference value II It is desirable to be.

Further, the classifying step is configured such that the first to fourth pixels are respectively corresponded in plural, and is based on a cumulative sum difference value I, a cumulative sum difference value II, and a cumulative sum difference value III of each difference value. It may be determined as.

Here, it is preferable that the first region is the background depth texture.

In the classifying, when the cumulative sum difference value III is large, the fourth region is determined as the foreground depth texture, and when the cumulative sum difference value III is small, the fourth region is determined as the background depth texture. It is desirable to do.

Meanwhile, it is preferable that the plurality of first to fourth pixels are selected from regions closest to the first to fourth regions.

According to the present invention, the depth of the hole region of the DIBR is predicted using only the background depth texture excluding the foreground depth texture when predicting, thereby providing more sophisticated depth assist information when filling the hole of the color image.

In addition, according to the present invention, it is possible to provide a more accurate and high-quality color image by utilizing surrounding texture information.

1 shows a hole area generated during DIBR.
FIG. 2 shows prediction of a color image generated after DIBR, a depth map, and a depth diagram of a hole region.
3 shows a general DIRB hole area depth prediction result.
4 is a flowchart illustrating an image encoding method according to an embodiment of the present invention.
5 illustrates a depth texture separation method used in an image encoding method according to an embodiment of the present invention.
6 illustrates a method for predicting a depth of a hole used in an image encoding method according to an embodiment of the present invention.
7 illustrates a result image of a method for predicting a depth of a hole used in an image encoding method according to an embodiment of the present invention.
8 is a block diagram illustrating an image encoding apparatus according to an embodiment of the present invention.

The present invention can be applied to various changes and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals are used for similar components.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from other components. For example, the first component may be referred to as a second component without departing from the scope of the present invention, and similarly, the second component may be referred to as a first component. The term and / or includes a combination of a plurality of related described items or any one of a plurality of related described items.

When an element is said to be "connected" or "connected" to another component, it is understood that other components may be directly connected to or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that no other component exists in the middle.

The terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "include" or "have" are intended to indicate the presence of features, numbers, steps, actions, components, parts or combinations thereof described herein, one or more other features. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Hereinafter, the same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted.

4 is a flowchart illustrating an image encoding method according to an embodiment of the present invention.

Referring to FIG. 4, an image encoding method according to an embodiment of the present invention includes: obtaining a depth map and a color image, dividing the depth map into a background and a foreground, and predicting a depth of a hole region It includes.

First, a depth map and a color image are obtained using depth image-based rendering (S10). The color image refers to a color image at a given viewpoint (eg, a left viewpoint image). The depth map displays information for generating an image of a different viewpoint, and includes a background depth texture located at the back of the screen and a foreground depth texture located at the front of the screen.

Thereafter, the generated depth map is classified into a background depth texture and a foreground depth texture (S20). The method of classifying them will be described later in detail.

Next, after excluding the foreground depth texture, the depth of the hole region is predicted (S30). As described above, when the depth of the hole region is predicted using only the background depth texture excluding the foreground depth texture, the hole can be filled with high quality.

5 illustrates a depth texture separation method used in an image encoding method according to an embodiment of the present invention. The video encoding method described below is performed by a video encoding device that performs video encoding, particularly in a processor within the video encoding device.

Referring to FIG. 5, an image encoding method according to an embodiment of the present invention includes obtaining a depth map and a color image using depth image-based rendering, and classifying the depth map into a background depth texture and a foreground depth texture It includes.

First, as illustrated in FIG. 2, when a color image of a different viewpoint is generated through the DIBR, the processor of the image encoding apparatus also generates a depth map through rendering (in this description, a left color image and a corresponding depth map are generated). It is assumed that a stereoscopic image of the right view is generated through).

The processor of the present embodiment classifies the depth texture around the hole into the foreground layer and the background texture before predicting the hole of the generated depth map. Referring to Figure 5 in detail the step of the classification as follows.

5 (a) is a case where the depth texture is a background, and (b) is a case where the depth texture is a foreground. The present invention uses a depth layer separation technique through pixel difference calculation at the edge of the hole region (4). The processor includes a first pixel p located at a boundary within a first area of the original image (object located at the left of the border, 1 in the left figure of FIG. 5 (a)) and a second area adjacent to or adjacent to the first area (classification) The difference I between the second pixel q located in 3) in the right figure of FIG. 5 (a) is calculated. In addition, the processor and the third pixel (p ') located on the boundary within the third region (the object located on the left side of the hole where the hole occurred, 1 in the right figure in FIG. , After calculating the difference value II between the fourth pixel (q ') in the fourth area (the right boundary of the hole as the object to be classified, 3 in the right figure in FIG. 5 (a)) that comes into contact with the other side, the difference value I, Using II, it is determined whether the object to be classified (3 in the case of FIG. 5 (a)) corresponds to the foreground or the background.

That is, the processor includes the first area of the object located on the left side of the hole area 4, the pixels p, p 'located on the third area, the second area of the object located on the right boundary, and the pixels q located on the third area, Each difference value between q 'x = (pq) and y = (p'-q') is calculated, and the depth texture of the object located on the right side of the hole is determined as the foreground or background using the calculation results of x and y. . In particular, when x-y is larger than a predetermined reference value, the right depth texture is determined as the foreground, and when the difference value is small, the background is determined. At this time, the first region is a background depth texture.

In this embodiment, the first region of the left figure of FIG. 5 (a) and the third region of the right figure of FIG. 5 (a) are regions corresponding to each other of the same object, and thus the reference for predicting the depth value of the right object is The pixel value may be different depending on the viewpoint.

In addition, the pixel located at the boundary is a pixel located within a predetermined distance from the boundary (for example, it may be specified by the number of pixels), and when computation is performed on all pixels of the corresponding object located at the boundary, a computational burden is imposed. Because of this, it may be defined as a pixel selected according to a predetermined predetermined interval, that is, some pixels of an object located in a boundary area.

In addition, the number of p, q, p ', and q' is preferably selected so that each region is selected from the closest band band (region), and the number is also the same corresponding to each other. The p, q, p ', and q' are configured to correspond to a plurality, respectively, and a cumulative sum, which is the sum of the differences, can be used. In this case, it can be said that the reliability is high because the texture is determined based on more pixels.

The step of classifying determines the third region as a foreground depth texture when the cumulative sum (xy) is greater than a preset threshold, and if the cumulative sum (xy) is less than a predetermined threshold, the second sum. 3 Determine the area as the background depth texture.

In relation to the selection of p, q, p ', and q', the processor in this embodiment may select q 'from a hole-occurring image according to a predetermined criterion according to a DIBR execution result. The processor may select q 'at a predetermined number or a predetermined interval at the boundary between the right hole region and the depth texture created by DIBR. Next, the processor determines the pixels that are symmetrically or facing each other at the position of q 'from the object located on the opposite side of the object containing q', and are positioned around the hole as p '. The processor may determine p 'and q', and then determine p and q through an inverse DIBR transformation. The method of selecting p, q, p ', and q' among the numerous pixels existing in the object is not limited to the above-described method, and may be selected by other methods by those skilled in the art.

In the case of FIG. 5 (b), the processor determines whether the fourth area (2 in FIG. 5 (b)) is the background or the foreground in the same manner as described for FIG. 5 (a). In the case of FIG. 5 (b), when viewed based on the region where the hole has occurred, the processor first includes pixels q 'located in the right boundary region of the hole, pixels p' located at the left side of the hole, and correspondingly The pixel values of the pixels (p) present in the left boundary area of the original image boundary and the pixels (q) present in the right boundary area are read, and x (pq) and y (p'-q ') are respectively calculated. do.

In FIG. 5 (b), p and p 'of the first and third regions 1 of the left object corresponding to the reference region are actually the same object, and their values maintain a constant level, but in the case of q and q' Since it is actually different, the value is not kept constant. That is, the value of the difference between x and y (p-p '-(q-q')) has a relatively large value when compared to FIG. 5 (a), and is a level exceeding a range of a predetermined reference value. In this case, 2 of FIG. 5 (b) may be classified as a foreground.

6 illustrates a method for predicting a depth of a hole used in an image encoding method according to an embodiment of the present invention.

The depth map prediction is performed using an inpainting algorithm after classifying the background depth texture and the foreground depth texture in the step of FIG. 5 described above.

When the classification of the depth texture around the hole is completed, a patch-based inpainting algorithm is applied to the hole area, and the depth texture of the foreground part is performed by excluding the inpainting algorithm (FIG. 6 (b)). FIG. 6 (a) shows the depth map prediction, and FIG. 6 (c) shows the result of applying the depth map prediction of the present invention.

Here, the inpainting algorithm can perform band in-painting and seamless cloning, which largely fills the inpainting area. Band in-painting defines a target band having a certain thickness along the border of the inpainting area, and has the same shape and size as the target band, centered on all pixels outside the inpainting area. The branch is to calculate the difference between the source band and the target band, and copy the value of the source band region with the smallest difference in value to the inpainting region. Seamless cloning removes the border between the inpainting area and the input image.

In order to predict the depth value of the hole region, in this embodiment, it is possible to use Navier-Stokes and fluid dynamics equations. The above equation is a method for predicting a depth value of a hole region based on image inpainting or depth hole-filling. In addition, as another method for predicting the depth value of the hole region, after classifying the depth texture around the hole, using only patch-based image inpainting algorithm using only the background depth texture information around the hole. It is desirable to predict depth (or inpainting).

7 illustrates a result image of a method for predicting a depth of a hole used in an image encoding method according to an embodiment of the present invention.

Referring to FIG. 7, it can be seen that a more sophisticated image is generated without a distorted image portion as in FIG. 3.

 As described above, according to the present invention, by accurately predicting the depth of the hole using only the background depth texture, more sophisticated depth assistance information can be utilized when filling the hole of the color image.

8 is a block diagram illustrating an image encoding apparatus according to an embodiment of the present invention.

Referring to FIG. 8, an image encoding apparatus according to an embodiment of the present invention includes an image processing unit 810, a depth map generation unit 820, and a 3D image generation unit 830.

The image processing unit 810 classifies a 2D image image received through a multi-view camera into each region and generates a color image. By using the generated color image as a reference image, information such as depth or difference angle for each pixel can be used to generate a scene at different viewpoints.

The depth map generating unit 820 generates a depth map using a depth image acquired through a depth camera. In order to fill the hole region that may be generated by the depth map generator 820, a more sophisticated depth map can be generated using the depth map prediction method described above. Meanwhile, the depth map generator 820 may store depth information in advance in a table form.

The 3D image generation unit 830 generates a right view using the depth map generated by the depth map generation unit 820, and uses the original image as a left view to generate a 3D image. Produces In this case, an existing general method is used to generate a 3D image using a depth map.

The depth of the hole predicted by the method proposed in the present invention can be used for various purposes, such as searching for textures similar to the hole area when filling the hole, and determining patch priority in the inpainting algorithm. have.

In addition, when the depth-of-hole prediction algorithm is applied to view-synthesis software in the field of 3DVC (3D video coding), which is undergoing international standardization for the next-generation high-definition 3D broadcasting, high-quality hole filling results are obtained. It is expected to be obtained.

Claims (14)

In the video encoding method,
Obtaining a depth map and a color image using depth image-based rendering (DIBR), and generating an original depth image and a predicted depth image;
Classifying the depth texture of the depth map into a background depth texture and a foreground depth texture; And
And excluding the foreground depth texture and predicting the depth of the hole region using an inpainting algorithm.
The classifying step selects a first region and a second region from the original depth image, selects a third region and a fourth region from the predicted depth image, and (i) according to the depth image-based rendering result. After selecting the fourth pixel located in the fourth area corresponding to the boundary between the hole area and the depth texture, (ii) after selecting the fourth pixel, located in the periphery of the hole area, the fourth pixel A pixel that is symmetric or facing the position of the fourth pixel in the third area located on the opposite side of the fourth area including is selected as a third pixel, and (iii) after selecting the third pixel, inverse depth A second pixel positioned at a boundary in the second region and a first pixel positioned at a boundary in the first region are determined through image-based rendering (inverse DIBR) transformation,
The classifying step includes the difference value I between the first pixel located at the boundary in the first area and the second pixel located at the boundary in the second area abutting the first area, and the one contacting one side of the hole area Calculate the difference value II between the third pixel located at the boundary in the third area and the fourth pixel located at the boundary in the fourth area that abuts the other side of the hole area, and based on the difference value I and the difference value II The video encoding method. delete The method according to claim 1,
The inpainting algorithm, Navier-Stokes equations, video encoding method. delete The method according to claim 1,
In the classifying, the first pixel to the fourth pixel are configured to correspond to a plurality of each, so that the cumulative sum difference value I, which is the sum of the difference values, and the cumulative sum difference value II, and the difference value I and difference values, respectively. The image coding method is determined based on the cumulative sum difference value III which is a difference of II. The method according to claim 5,
The first region is the background depth texture, the image encoding method. The method according to claim 6,
In the classifying, if the cumulative sum difference value III is greater than a predetermined reference value, the fourth region is determined as the foreground depth texture. The method according to claim 6,
In the classifying, if the cumulative sum difference value III is smaller than a predetermined reference value, the fourth region is determined as the background depth texture. delete The method according to claim 5,
The first region is the foreground depth texture, the image encoding method. The method according to claim 10,
In the classifying, when the cumulative sum difference value III is large, the third region is determined as the background depth texture, and when the cumulative sum difference value III is small, the third region is determined as the foreground depth texture. Video encoding method. A processor; And
It is connected to the processor and includes a memory for storing information for driving the processor,
The processor acquires a depth map and a color image using depth image-based rendering (DIBR), generates an original depth image and a predicted depth image, and sets the depth texture of the depth map to background depth texture After classifying as and foreground depth texture, after excluding the foreground depth texture, the depth map of the hole region is predicted using an inpainting algorithm.
The processor selects a first region and a second region from the original depth image, selects a third region and a fourth region from the predicted depth image, and (i) the hole according to the depth image-based rendering result. A fourth pixel located in the fourth area corresponding to a boundary between an area and the depth texture is selected, and (ii) after selecting the fourth pixel, located in the periphery of the hole area, and the fourth pixel included In the third area located on the opposite side of the fourth area, a pixel that is symmetric or facing the position of the fourth pixel is selected as a third pixel, and (iii) after the third pixel is selected, an inverse depth image is based on A second pixel positioned at a boundary in the second region and a first pixel positioned at a boundary in the first region are determined through inverse DIBR transformation,
The processor may include a difference value I between the first pixel located at the boundary in the first area and the second pixel located at the boundary in the second area contacting the first area, and the third contact with one side of the hole area. Calculate the difference value II between the third pixel located at the boundary in the area and the fourth pixel located at the boundary in the fourth area that abuts the other side of the hole area, and determine based on the difference I and the difference II Video encoding device. delete The method according to claim 12,
Classifying the background depth texture and the foreground depth texture is
The first pixel to the fourth pixel are configured to respectively correspond to a plurality, and the cumulative sum difference value I and the cumulative sum difference value II and the cumulative sum difference of the difference value I and the difference value II are summed values of the difference values. And configured to be determined based on the difference value III.

Images